• working on DNS.. links may break temporarily.

KCBS judge seating changes

GrillBillie_D

GrillBillie_D

Full Fledged Farker
Joined
Mar 8, 2012
Messages
342
Reaction score
299
Points
0
Location
Redding, CA
Just saw the board meeting quick notes:

Early 2017, it was determined that judges would be seated via the Shuffle. This was instituted nationwide. Mid-year, Mark Gibbs developed a method of seating of judges that was based on their scoring averages divided evenly among the number of tables to be utilized at a contest. This program was Beta tested across the country for several months and it was determined that it was very successful. It was rolled out to ALL reps in January of 2018 and mandated to be used for all Master Series contests.

I am in favor of this change but this is not the end of scoring issues. It may help a little at contests but there are still other issues. The fundamental issue of judge training differences still needs to be addressed.
 
GrillBillie_D said:
Just saw the board meeting quick notes:

Early 2017, it was determined that judges would be seated via the Shuffle. This was instituted nationwide. Mid-year, Mark Gibbs developed a method of seating of judges that was based on their scoring averages divided evenly among the number of tables to be utilized at a contest. This program was Beta tested across the country for several months and it was determined that it was very successful. It was rolled out to ALL reps in January of 2018 and mandated to be used for all Master Series contests.

I am in favor of this change but this is not the end of scoring issues. It may help a little at contests but there are still other issues. The fundamental issue of judge training differences still needs to be addressed.
Click to expand...
No offense to Mark but this is putting a band aid on an arterial wound. YOU CANNOT AVERAGE RANDOM EVENTS. It'll look good for a bit but the TOD's and TOA's will remain until, as stated above "The fundamental issue of judge training differences still needs to be addressed."
 
They used the new seating system at the KCBS masters event here in Melbourne earlier this month.

I'm a KCBS judge myself so I'm interested in the opinions of competitors, could you explain what the "fundamental issue of judge training differences still needs to be addressed" is/are? :confused:
 
Gary Tomato said:
They used the new seating system at the KCBS masters event here in Melbourne earlier this month.

I'm a KCBS judge myself so I'm interested in the opinions of competitors, could you explain what the "fundamental issue of judge training differences still needs to be addressed" is/are? :confused:
Click to expand...


Not the only one but a big issue a lot of people have including many of the judges is the way they were instructed. Some were instructed to start at a 9 and work down from there, others to start at a 7 and work up or down and yet others at a 5 and work up. That is one major issue
 
Fat Freddy said:
Not the only one but a big issue a lot of people have including many of the judges is the way they were instructed. Some were instructed to start at a 9 and work down from there, others to start at a 7 and work up or down and yet others at a 5 and work up. That is one major issue
Click to expand...

There is not supposed to be any predefined starting score number. Each entry is supposed to be scored as it is, on its own merit. Many years ago the standard was to start at 9; everything was considered to be a 9 until you saw it and then actually sampled it. This resulted in a LOT of 180 scores. At some point it was decided to define 6 as the starting point and score up or down based on each entry. I think the intent was to reduce the number of 180 scores.
 
Really I don't care where they seat me. My scoring patterns are not set by how I score past events and past entrys but are more caused by the entrys provided to my table on that one day. I react to the quality of each contests entry.
I also contend there will always be a TOA and TOD it just has to have a winner and a loser.
As far as
"The fundamental issue of judge training differences still needs to be addressed."

All CBJ training classes have for several years been taught from one power point presentation. You can't be more consistent than that. All that's left is to recertify judges trained before the programs beginning.
Mandatory recetification? I'm good with that. It can be done online in a hour...
I've been to 7 CBJ classes taught by 4 or 5 different instructors and I don't see a problem with current training procedures.
Ed
 
Fat Freddy said:
Not the only one but a big issue a lot of people have including many of the judges is the way they were instructed. Some were instructed to start at a 9 and work down from there, others to start at a 7 and work up or down and yet others at a 5 and work up. That is one major issue
Click to expand...

Agreed!

QN said:
There is not supposed to be any predefined starting score number. Each entry is supposed to be scored as it is, on its own merit. Many years ago the standard was to start at 9; everything was considered to be a 9 until you saw it and then actually sampled it. This resulted in a LOT of 180 scores. At some point it was decided to define 6 as the starting point and score up or down based on each entry. I think the intent was to reduce the number of 180 scores.
Click to expand...

While that may be true to day, it was not always true. When I sent to my first CBJ class we were taught to start at a 6 and score high for food that was above average, etc. My wife went a couple of years later and she was taught to start at a 9 and score lower as required. Both classes were taught by the same instructor, but the content had changed.

Without some sort or re-certification program to bring every one up to today's standards, there are judges who were taught differently and approach the table differently.

ModelMaker said:
Really I don't care where they seat me. My scoring patterns are not set by how I score past events and past entrys but are more caused by the entrys provided to my table on that one day. I react to the quality of each contests entry.
I also contend there will always be a TOA and TOD it just has to have a winner and a loser.
As far as
"The fundamental issue of judge training differences still needs to be addressed."

All CBJ training classes have for several years been taught from one power point presentation. You can't be more consistent than that. All that's left is to recertify judges trained before the programs beginning.
Mandatory recetification? I'm good with that. It can be done online in a hour...
I've been to 7 CBJ classes taught by 4 or 5 different instructors and I don't see a problem with current training procedures.
Ed
Click to expand...

It's been a while since we've run into you at a competition, Ed, but I don't think senior judges like you (and I mean that in terms of judging experience, not age :-D ) will change how they score based on seating. But, it may affect newer judges, and may help to even things out, but only time will tell.
 
I still think the problem is the fact KCBS will not define what is "average" bbq. The score card dictates that average bbq rates a 6 and then moves up or down from there. Is average everything you have sampled in your life or only those meats eaten at a judging table? I think a good start would be to define average as competition average. This would even out people's life experience to some point.
 
rob g said:
I still think the problem is the fact KCBS will not define what is "average" bbq. The score card dictates that average bbq rates a 6 and then moves up or down from there. Is average everything you have sampled in your life or only those meats eaten at a judging table? I think a good start would be to define average as competition average. This would even out people's life experience to some point.
Click to expand...

Competition Average????
So every new certified judge starts out with absolutely no idea what "competition average" is?
Do you want to use the meat offered in the class to set the new judges average for all future contests?
After one contest do they know what competition average is then?

I've always held that 10 judges have 10 ideas of what average is, kind of what the word average means.
Do you think there is such a swing in the word average that contests are won or lost because of it? A new judge that is from small town Iowa's exposure to BBQ is going to be dramatically different than a new judge from downtown Kansas City.
Average moves constantly.
Only experience will set what a judge determines average to be.
Ed
 
Ron_L said:
It's been a while since we've run into you at a competition, Ed, but I don't think senior judges like you (and I mean that in terms of judging experience, not age :-D ) will change how they score based on seating. But, it may affect newer judges, and may help to even things out, but only time will tell.
Click to expand...

If the goal is to even things out, we are all in big trouble!

On a side note, I wonder how they determined it was a success? More scores bunched to the middle of the pack?
 
ModelMaker said:
Competition Average????
So every new certified judge starts out with absolutely no idea what "competition average" is?
Do you want to use the meat offered in the class to set the new judges average for all future contests?
After one contest do they know what competition average is then?

I've always held that 10 judges have 10 ideas of what average is, kind of what the word average means.
Do you think there is such a swing in the word average that contests are won or lost because of it? A new judge that is from small town Iowa's exposure to BBQ is going to be dramatically different than a new judge from downtown Kansas City.
Average moves constantly.
Only experience will set what a judge determines average to be.
Ed
Click to expand...

You are correct that only experience will determine what average is. As new cbj's get a few contests under their belt then their idea of average will settle out if they are directed to base their opinions only on what they have eaten at a contest table. KCBS still needs to try to define average so judges have something to base their opinion on and try to develop a standard that can be used at every contest.

While we are told not to compare sample 1 vs sample 2 judging by its very nature is comparative. We are comparing the piece of meat we have in front of us to our individual definition of average or hopefully a defined standard.

Until somebody can put a piece of meat in front of you and say this should be scored a 6/7/8 based on published KCBS standards you will not get consistency across the board. Standards need to be defined and that is easier said than done.
 
I would say you definitely can get more consistency in terms of appearance scoring as ALL of the judges are looking at the same box but yet you’ll get scores ranging from 6-9. That shouldn’t happen. Yes, that lone 6 is thrown out, but the remaining 7 kills you when the others gave it 8-9. It’s the same dang box, the scores should be done within 1 point! Now for taste and tenderness, you’re going to get some natural variability as palates differ and what is the right amount of heat for a spicehead is too much for someone with acid reflux. You’re also going to get different tenderness scores because the meat may differ even if cooked to the same temperature. That’s going to happen no matter what and is part luck and skill of the cook.

As far as people’s palates go, the only way you can even come close is for KCBS to finally determine what “average barbecue” is. Is it your neighbor’s backyard bbq chicken that’s a little too charred and covered in SBR or is it the local bbq joint’s consistently decent stuff? Everyone has different life experiences but I would think the natural tendency is for more experienced judges to go with what they’ve tasted in previous comps for comparison’s sake (unless instructed otherwise). As others also referenced, ALL judges either need to start at 9 and go down or start at 6 (average) and go up or down... Of course, then you gotta come up with a universally understood definition of what “average barbecue” is for that 6 starting point to work. Good luck with that...

Reminds me of an old PJ O’Rourke joke about the social sciences which he says boils down to 3 rules:
1. People do lots of things.
2. We don’t know why.
3. Test on Friday.
 
I think a starting point is a good thing. But starting at 9 will inflate scores as stated above. If anything a starting point of average ~x (6?) is acceptable. To define average is a little tough because flavor is so subjective. Somebody's average is somebody's perfect. How common are scores of 5 or below? But at the end of the day, aren't all boxes subject to the good/bad table dilemma? If so, then outcome will always vary a bit... IMO.
 
I went through the class on 2000 or 2001. Started at 9. Have never had any new training or certification.... That is my problem with KCBS judging and until everyone is on the same page, these changes are just a band-aid
 
Scottie said:
I went through the class on 2000 or 2001. Started at 9. Have never had any new training or certification.... That is my problem with KCBS judging and until everyone is on the same page, these changes are just a band-aid
Click to expand...

Jeez Scottie, where you been, I haven't been yelled at in a long time...
Have you ever considered sitting in on a class that would be free to you?
Have you taken the 50 question quiz on the KCBS website to see if your current on rules and practices?
I know you want to be the best judge you can be.
Ed
 
D&D BBQ said:
I would say you definitely can get more consistency in terms of appearance scoring as ALL of the judges are looking at the same box but yet you’ll get scores ranging from 6-9. That shouldn’t happen. Yes, that lone 6 is thrown out, but the remaining 7 kills you when the others gave it 8-9. It’s the same dang box, the scores should be done within 1 point! Now for taste and tenderness, you’re going to get some natural variability as palates differ and what is the right amount of heat for a spicehead is too much for someone with acid reflux. You’re also going to get different tenderness scores because the meat may differ even if cooked to the same temperature. That’s going to happen no matter what and is part luck and skill of the cook.

As far as people’s palates go, the only way you can even come close is for KCBS to finally determine what “average barbecue” is. Is it your neighbor’s backyard bbq chicken that’s a little too charred and covered in SBR or is it the local bbq joint’s consistently decent stuff? Everyone has different life experiences but I would think the natural tendency is for more experienced judges to go with what they’ve tasted in previous comps for comparison’s sake (unless instructed otherwise). As others also referenced, ALL judges either need to start at 9 and go down or start at 6 (average) and go up or down... Of course, then you gotta come up with a universally understood definition of what “average barbecue” is for that 6 starting point to work. Good luck with that...

Reminds me of an old PJ O’Rourke joke about the social sciences which he says boils down to 3 rules:
1. People do lots of things.
2. We don’t know why.
3. Test on Friday.
Click to expand...

What if that 7 for appearance is accurate and the 8-9 judges routinely hand out those scores and rarely, if ever, score below that? What if the problem is too many 8s and 9s?

I haven't seen the data or had the opportunity to do any analysis. To my knowledge, nobody else has either. When Reps tell me that we are living in a world where 80% of scores are 8-9 I have a problem believing that 20% of judges are screwing it up for everyone.

I don't believe that over the course of a season 80% of cooks are within 1 point per criterion of Clark Crew BBQ, or any other top 5 ToY team.
 
Jorge said:
What if that 7 for appearance is accurate and the 8-9 judges routinely hand out those scores and rarely, if ever, score below that? What if the problem is too many 8s and 9s?

I haven't seen the data or had the opportunity to do any analysis. To my knowledge, nobody else has either. When Reps tell me that we are living in a world where 80% of scores are 8-9 I have a problem believing that 20% of judges are screwing it up for everyone.

I don't believe that over the course of a season 80% of cooks are within 1 point per criterion of Clark Crew BBQ, or any other top 5 ToY team.
Click to expand...

Due to some of the homogonization of flavors, I could totally see that being the case in taste. Tenderness, imo, is where people like Travis, Brad, and Tim kick our ass.
 
Jorge said:
What if that 7 for appearance is accurate and the 8-9 judges routinely hand out those scores and rarely, if ever, score below that? What if the problem is too many 8s and 9s?
Click to expand...

Again, that’s a problem with standards and how they are applied. “Beauty is in the eye of the beholder” coupled with grade inflation because there’s no OBJECTIVE, quantifiable standard that everyone can point to and definitely say, “That’s a 6”. Yes, it could be it’s a problem of grade inflation partly because judges don’t like giving out less than 8 as that requires a comment card OR the 6 and 7 given out are from 2 crotchety grumps because “9 means excellent, so very few entries are worth that”, even if they are on a table at the Royal or Jack with all the tables’ entries from top teams....And it’s going to be that way as the “sport” is based off of people’s subjective opinions and not off an easily quantifiable standard.

If competition bbq was a game show, it wouldn’t be Jeapordy as there’s no “right answer”, it would be more like Family Feud where the winning answer is not the “correct answer” but rather “what’s the most popular answer?”... And the top teams have pretty much figured out what the most popular answers are going to be.

BTW, I think beer comps have this down a little better as rookie judges are sat with master judges and the table can’t differ very much from the master judge. More consistency but at the cost of conformity.
 
Jorge said:
What if that 7 for appearance is accurate and the 8-9 judges routinely hand out those scores and rarely, if ever, score below that? What if the problem is too many 8s and 9s?

I haven't seen the data or had the opportunity to do any analysis. To my knowledge, nobody else has either. When Reps tell me that we are living in a world where 80% of scores are 8-9 I have a problem believing that 20% of judges are screwing it up for everyone.

I don't believe that over the course of a season 80% of cooks are within 1 point per criterion of Clark Crew BBQ, or any other top 5 ToY team.
Click to expand...

More and more I’m liking the idea of the pull from the crowd judging. Wish I was closer to IBCA to give it a try.
 
I used to assist with judging classes. They still havent brought everyone to the same page.

ModelMaker said:
Jeez Scottie, where you been, I haven't been yelled at in a long time...
Have you ever considered sitting in on a class that would be free to you?
Have you taken the 50 question quiz on the KCBS website to see if your current on rules and practices?
I know you want to be the best judge you can be.
Ed
Click to expand...
 
You must log in or register to reply here.

Similar threads

O
Long brisket hold skeptic
2 3 4
Replies
74
Views
4K
Itsdone
I
N
KCBS Announces Rod Gray as new CEO
Replies
11
Views
126K
Smokin D@wg@77
S
thirdeye
Any KCBS Judges Having Sign-Up Issues in Judge Portal?
Replies
3
Views
457
thirdeye
thirdeye
P
Weber Kettle 22 - What was I thinking?!?
Replies
1
Views
2K
Happy Hapgood
Happy Hapgood
ModelMaker
For you judge types
Replies
16
Views
1K
thirdeye
thirdeye
Back
Top