Judge accountability

[Atlasman]

Is there such a thing???.......or are they just let slide because of the excuse for incompetence (low score dismissed)


Example:

Our last chicken scores

988--888--999--999--857--988


I'm sorry but this should require an explanation. Further cause for concern is this judges AVG score he gave out all day for chicken was 27........3-5 points lower than every other judge at his/her table. So this person was at a table that EVERY other judge thought got good chicken all day yet they graded every entry they tasted poorly.

I wonder if these judges even comprehend a little bit what the consequences of their actions can be. To see teams get denied a chance at the Jack and have contests and championships effected by such silliness is a little disheartening when you factor in the amount of hard work, time and money that is invested by cooks and teams trying to have a successful season.

[Gowan]

Assuming this was a KCBS contest....

1) Comment cards are currently suspended, so this judge could not provide an explanation to the team. Hopefully his Table Captain asked him for one when he turned in his slip.

2) Disparate scores are most common in chicken, where you are typically cooking separate pieces of meat. It is entirely possible for one thigh of six to deserve significantly lower scores than the other five.

3) Reps work to distribute boxes equally to all tables to minimize the damage of a low-scoring judge or table. This is why statistically a 24 team contest is the most fair, as each team's box should hit each table exactly one time. With more than four tables, luck becomes a big factor for teams that manage to miss the cold table.

If one judge was lowballing for the whole category, we would hope the Table Captain/Rep would have pointed this out and nudged them back to the centerline. I have seen this happen, usually with new judges and they usually take the correction well and are scoring back in range with the rest of the table.

[rolfejr]

I feel your pain- Ribs in Dover

989 -- 988 -- 856--765--878--989

Please fix the comment cards !!!!

[Teamfour]

Doesn't KCBS throw out the lowest score?

[DawgPhan]

Is there such a thing???.......or are they just let slide because of the excuse for incompetence (low score dismissed)


Example:

Our last chicken scores

988--888--999--999--857--988


I'm sorry but this should require an explanation. Further cause for concern is this judges AVG score he gave out all day for chicken was 27........3-5 points lower than every other judge at his/her table. So this person was at a table that EVERY other judge thought got good chicken all day yet they graded every entry they tasted poorly.

I wonder if these judges even comprehend a little bit what the consequences of their actions can be. To see teams get denied a chance at the Jack and have contests and championships effected by such silliness is a little disheartening when you factor in the amount of hard work, time and money that is invested by cooks and teams trying to have a successful season.


lulz...you should have saved this for the silly season.

You got a bad score. It happens...it actually got dropped so it didnt impact your score. You probably didnt walk, but that score wasnt the reason.

Also it is entirely possible that piece of chicken earned that score.

[TooSaucedToPork]

MBN is tracking individual judge scores, and thoroughly reviewing its judges...and then booting or retraining the problem ones. Wish KCBS would jump on the Bandwagon.

And before a Judge flames me, I am a KCBS, MIM, and MBN CBJ. The vast majority of judges are good, just, and fair...but you know as well as I there are odd ducks that score on wacky things like "I dont like muffin pan chicken so I never give it a 9" or "I score down for not have pulled pork in a box" - both statements were made by CBJ's at contests.

When I screw up as a cook, a judge gives me a bad score...when a judge screws up or is a bad judge, he or she ruins a team's chances of winning and costs us cooks a lot of money...HUGE difference

So yeah, judge accountability is an issue.

[QN]

I see a lot of judges just using the numbers 9-8-7 for every entry they score at KCBS events. I have even heard judges say that they just can't give anything lower than 7 even if it should be 5 (below average). If you have someone trying to use the entire scale using all the numbers from 2 to 9 and other judges at the same table are using only 9-8-7 then the judge trying to score correctly looks like a low scoring judge. There was even an article in the Bullsheet recently about this.
From the article in the September Bullsheet;
Another habit I see in judging is the 7,8,9 judges. The numbers available
to us as judges are 2 thru 9.
Granted 2,3,4 are seldom seen at one of my contests but they do happen
and I for one have no problem writing them on my card. If a sample
is that bad usually I’m not the only one at the table with that score. But I
have also heard a judge say “I never give anything less than a 7 because
I feel bad for the cook”, That’s nice and all but what about the cook? A 7
means above average, are you doing the cook a disservice by telling him
his entry is above average at this contest and the next contest judges give
him a realistic score of 5 and now he is extremely confused? Wouldn’t it
be more fair to truthfully give him the score that really reflects his entry so
that he knows where he stands and can begin the process of improving?

[Lake Dogs]

Atlasman and to an extent Rolfe, I think the main points to come away with here have been listed above, being:

> Disparate scores are most common in chicken, where you are typically cooking separate pieces of meat. It is entirely possible for one thigh of six to deserve significantly lower scores than the other five.

Ribs being second most common because of different ends, different racks, etc...


Also, scores varying 2 numbers are normal and to be accepted. Frankly, it's very subjective (other than tenderness). Even then, tenderness does have a tendency to affect taste scores. I think that's what we're seeing on Rolfe's scores above.

The 5's and 6's to the 9's displayed in Rolfe's example is noteworthy IMHO, but also much more easily explained in ribs or chicken. The variance is pretty huge, but I've seen it in my own Q a time or two. One time we put forth ribs from 2 different racks, only to find out that one of the racks (had to be the un-sampled one, didnt it, da**ed laws of nature) was skunky. We were slammed in taste on those 2 ribs; rightfully so...

I'm not saying yours were skunky. However, it appears as though you had 2 or 3 ribs there that probably were tough as nails (if I had to guess).

[Pole D]

I can understand differences in taste and tenderness. Especially in ribs and chicken. However, appearance should be pretty consistent. How can someone come up with a 7 when the rest are 9's and 8's?

[nkay]

I actually competed in an event earlier this summer and at the cook's meeting we were told that all judges had erasers and that they could change our scores based on what other entries scored. When asked why they didn't use a set standard for each category instead of comparative judging, which gives us no mark to strive for, they did not offer an answer except to explain it again. Seems to me that any event using comparative scoring hurts its cooks simply because the cooks not longer have an idea of what T T and A should look like. That's like saying if you have more tackles and bigger players than the other team, you should always win the game.

[Pole D]

^ Was it Beer, Bourbon, BBQ? That's the way NCBS runs things.

[TooSaucedToPork]

I actually competed in an event earlier this summer and at the cook's meeting we were told that all judges had erasers and that they could change our scores based on what other entries scored. When asked why they didn't use a set standard for each category instead of comparative judging, which gives us no mark to strive for, they did not offer an answer except to explain it again. Seems to me that any event using comparative scoring hurts its cooks simply because the cooks not longer have an idea of what T T and A should look like. That's like saying if you have more tackles and bigger players than the other team, you should always win the game.

What sanctioning body?

Example...Each box in Blind Judging in KCBS is judged on its own merits. In MBN entries are judged agianst each other...best on table wins.

[QN]

I can understand differences in taste and tenderness. Especially in ribs and chicken. However, appearance should be pretty consistent. How can someone come up with a 7 when the rest are 9's and 8's?
As Lakedogs stated above, "scores varying 2 numbers are normal and to be accepted". 7 is defined as above average, so getting a 7 along with 8's and 9's is not inconsistent and the lowest score is dropped.

[nkay]

And that is why I am a KCBS member....

[Lake Dogs]

I can understand differences in taste and tenderness. Especially in ribs and chicken. However, appearance should be pretty consistent. How can someone come up with a 7 when the rest are 9's and 8's?

In general I agree with you, BUT . . . darned if the devil isn't in the actual details...

First, I hear multiple times from many people that appearance score is the most objective (and least subjective) of them. I happen to disagree 100%. Frankly, what really turns your taste buds on (in appearance) may not set my lights afire and visa versa. Also, imagery comes in to it for some, and I'll author a point that if a piece of something comes across your table at a restaurant and it looks completely like a turd, something tells me that you're not thinking: "DAMN, I JUST MUST EAT THAT TURD". As it happens, plenty MM's that aren't displayed well look like turds. Pulled pork can be made to look like cat food if you aren't careful. I've seen chicken that looked like potatoes, and many others that didn't look like food at all. It's tough to say "Damn, I want to eat those taters" when we're trying to judge how appetizing the chicken entry...

So, honestly, I think a 2 point variance (in a KCBS system) is normal. However, 3 or 4 point variance, unless something is very off, should be questioned.

[AlabamaAvenue]

Is it fair to say that geography/where comp is held has something to do with the outcome judging?

[Balls Casten]

Assuming this was a KCBS contest....


If one judge was lowballing for the whole category, we would hope the Table Captain/Rep would have pointed this out and nudged them back to the centerline. I have seen this happen, usually with new judges and they usually take the correction well and are scoring back in range with the rest of the table.

I hope they are doing this after the contest and not between turn-ins.
Someone correct me but I dont think a judge should be influanced in the middle of a contest. ????

[Fat Freddy]

I started out as a judge and always looked at things through a judges eyes, then when I started cooking i looked at things through a competition cooks eyes. As time has progressed I have tried to see things with OPEN eyes.

Atlasman I will only address your chicken score and this is only a possible explanation.

Appearance: four 9s and two 8s= pretty consistent
Taste:two 9s, three 8s and a 5= maybe one judge got a bite with way more rub or pepper or spicier than expected or whatever but it still may have been unintentionally different.
Tenderness:three 8s two 9s and a 7= you want to have more 9s than 8s but not bad. Maybe the one judge got a rubbery skin, maybe skin came off completely.

You cant tell for sure but by having more 8s than 9s in taste and tenderness you can tell there may be minor issues. All in all not bad but I hope I pointed out there could be very legitimate reasons for your score.

And if you dont like any of those explanations, then use the explanation a buddy of mine uses every time he gets a score he doesnt like. " Why is that person a judge, he didnt come here wanting BBQ he was wanting Chinese food"

[ITBFQ]

I am a KCBS CBJ and I do backyard competitions. Judging is not the easiest job in the world. I enjoy it, and I am as fair as I can be, just like I expect the judges to be when I turn in ribs or chicken or whatever in a competition. I re-read the red KCBS judge's manual the night before I judge at a competition to make sure I am familiar with the standards KCBS has set. If I judge meat that really tastes "average", then that is the score they are going to get. If a box looks excellent, then that is the score it is going to get for appearance. Low scores happen once in a while... this is a competition - someone has to win, and someone has to lose. As long as human beings are judging, it isn't going to be perfect, no matter how hard we judges try. And the lowest score is dropped... so unless Atlasman was tied with someone and it cost him GC or money or a call, is this really that big of a deal?

[TooSaucedToPork]

I am a KCBS CBJ and I do backyard competitions. Judging is not the easiest job in the world. I enjoy it, and I am as fair as I can be, just like I expect the judges to be when I turn in ribs or chicken or whatever in a competition. I re-read the red KCBS judge's manual the night before I judge at a competition to make sure I am familiar with the standards KCBS has set. If I judge meat that really tastes "average", then that is the score they are going to get. If a box looks excellent, then that is the score it is going to get for appearance. Low scores happen once in a while... this is a competition - someone has to win, and someone has to lose. As long as human beings are judging, it isn't going to be perfect, no matter how hard we judges try. And the lowest score is dropped... so unless Atlasman was tied with someone and it cost him GC or money or a call, is this really that big of a deal?

Yup it is.

YOU ARE A GOOD JUDGE. You do things right and true. I appreciate that more than you know. We cooks need EVERY Judge to be like you

BUT, you know there are judges out there that do things wrong, don't follow the rules, and get away with it because there is absolutely no accountability system in place, and that needs to be corrected. MBN is currently doing that, why can't KCBS?

Good judges would skate through with flying colors. They would catch the judges that need to be retrained on rules, the ones confused on what should be or shouldn't be in a box, and ESPECIALLY the few that don't care what the rules say...they do what they want.

[ModelMaker]

I'd like to throw this out to the KCBS judges. When you see a judge doing something that you know is sideways to the rules or procedures why not just bring it out in table discussion right then and there? You don't have to yell or be accusitory just bring it up and have the discussion right now.
If every wrong was righted in such a manner all the big problems would soon turn into little ones maybe.
Ed

[Jorge]

BUT, you know there are judges out there that do things wrong, don't follow the rules, and get away with it because there is absolutely no accountability system in place, and that needs to be corrected. MBN is currently doing that, why can't KCBS?



The ability to track judges is there in the new software. I'd hope that KCBS would take a look at the data, and make sure that it is a meaningful sample, and develop a productive program. There are plenty of theories based on anecdotal evidence. A quick knee jerk solution, using limited data is likely to do more harm than good.

I absolutely agree that there is some benefit to be had. I'd hope that the long term benefits of improving consistency is the priority, rather than doing "something/anything" simply to be responsive.

[TooSaucedToPork]

The ability to track judges is there in the new software. I'd hope that KCBS would take a look at the data, and make sure that it is a meaningful sample, and develop a productive program. There are plenty of theories based on anecdotal evidence. A quick knee jerk solution, using limited data is likely to do more harm than good.

I absolutely agree that there is some benefit to be had. I'd hope that the long term benefits of improving consistency is the priority, rather than doing "something/anything" simply to be responsive.

Amen

I agree a study needs to be launched, and it needs to be at least year or two long to track inconsistant and problem judges. Knee jerk only leads to Drama and Problems. Unbiased, Scientific evidence is the only way to go, and it will take time, but it will work.

That settles blind scores.

Let me give you an example of what I have a problem with and what, I'm told, MBN is currently investigating and resolving. Many of us cooks/judges have had the pleasure of sitting next to a judge that makes outlandish statements such as "I always score down if I see Muffin Pan Chicken" or "I don't give a 9 unless there is pulled pork in a pork box" etc. Now say I hear this, I report it, where is it logged? What besides a rep saying "Oh, Please don't do that anymore" and making a note that will probably get lost in the shuffle, is going to happen. They will continue giving an 8 on my chicken based on a personal preference and not the rules. Another team gets a 9 from that judge because they used the loaf method. People downplay it, but that is a big problem. That is the difference between a 1st and a 2nd. Could even be the difference between a GC and an RGC.

These are the type of problems I have with A FEW Judges. The reps need a national database with files they can add to. This prevents bad judges. This identifies judges for retraining...

I know I am preaching to the choir, but as a judge, you judge unbiasedly period. If you can't do that, you shouldn't be allowed to judge a contest...

[Gowan]

I hope they are doing this after the contest and not between turn-ins.
Someone correct me but I dont think a judge should be influanced in the middle of a contest. ????

You would prefer to see a novice judge lowball for the entire contest? I guess the question is do you choose to potentially punish everyone by not taking action vs. "keeping it even" by allowing the newbie to continue uncorrected.

Really, all you can do is advise the judge that they are scoring differently than the rest of the table and ASK them to explain why they scored as they did. It's hard to question somebody about scores they gave for chicken two hours later - I know I myself would be hard pressed to remember the entries and my impressions accurately at that point.

[Gowan]

Amen

These are the type of problems I have with A FEW Judges. The reps need a national database with files they can add to. This prevents bad judges. This identifies judges for retraining...



Access to such information is the big question for me as a judge, cook AND organizer.

It is the job of the organizer to select the judges that serve at their event(s). Assuming there is data available about the performance of judges, should this information be available to organizers? Would there be a potential for abuse if an organizer tried to pack an event with high-scoring judges in order to appeal to more teams?

Very interesting questions.

[speedrcer1]

I'd like to throw this out to the KCBS judges. When you see a judge doing something that you know is sideways to the rules or procedures why not just bring it out in table discussion right then and there? You don't have to yell or be accusitory just bring it up and have the discussion right now.
If every wrong was righted in such a manner all the big problems would soon turn into little ones maybe.
Ed
Your technically not supposed to see the other judges scores, but sometimes you do by accident. I brought the same thing up earlier this year (here) with a judge next to me was way off the mark.
Ideally, the reps should say something, but at that point they are busy entering in scores.

[rolfejr]

To be clear, I'm not cry'in. You get judged and that's the end.
It would, however, be interesting to see the judges rationale (comment card)
for scoring a 5 in light of the other scores.
Our turn in was the same rack, sliced and sauced, and reassembled in order so the low marks were glaring.

That's yesterday, but I would like to do better the next time.

[Lake Dogs]

To be clear, I'm not cry'in. You get judged and that's the end.
It would, however, be interesting to see the judges rationale (comment card)
for scoring a 5 in light of the other scores.
Our turn in was the same rack, sliced and sauced, and reassembled in order so the low marks were glaring.

That's yesterday, but I would like to do better the next time.


100% understand, and this is where that comment card will let you know, if nothing else, what the **** that person was thinking...

[QN]

I'd like to throw this out to the KCBS judges. When you see a judge doing something that you know is sideways to the rules or procedures why not just bring it out in table discussion right then and there? You don't have to yell or be accusitory just bring it up and have the discussion right now.
If every wrong was righted in such a manner all the big problems would soon turn into little ones maybe.
Ed

I was at a contest recently and had what I thought was an excellent rib and I scored it as such; 9-9-9. After the cards were turned in and we were discussing the entries I commented on how good that particular entry was and that I scored it all 9's. The judge next to me said she thought it was good, but did not score it 9 because it was not the best she ever had. I pretty much went off on her and told her that the best she ever had did not have anything to do with judging a KCBS contest. I think I pissed her off, but then she backpedaled and said she just did not think it was a 9 rib entry. Just an example of things you see and hear at different contests.

[BMerrill]

Accountablity of judges should be deeply rooted at the event level. A good table captian will quickly see an "out of line scoring judge" which can be either a low or high scoring judge as compared to the other judges on the table. The between turn-in discussions will often times give clues, including the reasons the "out of line" judge's scores differ from the rest of the table. This discussion should be encouraged as it helps young judges learn what is good 'Q vs. bad 'Q. The rep's should also get involved as soon as possible by having a side bar discussion with the judge(s).

Long term tracking of judges scores may not be very reliable because from time to time a judge may have to DQ an entry which would adveresly affect his average assuming the DQ'd score is included.

[TooSaucedToPork]

Accountablity of judges should be deeply rooted at the event level. A good table captian will quickly see an "out of line scoring judge" which can be either a low or high scoring judge as compared to the other judges on the table. The between turn-in discussions will often times give clues, including the reasons the "out of line" judge's scores differ from the rest of the table. This discussion should be encouraged as it helps young judges learn what is good 'Q vs. bad 'Q. The rep's should also get involved as soon as possible by having a side bar discussion with the judge(s).

Long term tracking of judges scores may not be very reliable because from time to time a judge may have to DQ an entry which would adveresly affect his average assuming the DQ'd score is included.

DQ's are logged by reps. Those could easily be thrown out

[Balls Casten]

You would prefer to see a novice judge lowball for the entire contest? I guess the question is do you choose to potentially punish everyone by not taking action vs. "keeping it even" by allowing the newbie to continue uncorrected.

Really, all you can do is advise the judge that they are scoring differently than the rest of the table and ASK them to explain why they scored as they did. It's hard to question somebody about scores they gave for chicken two hours later - I know I myself would be hard pressed to remember the entries and my impressions accurately at that point.

I would think yes, all contestants should be judged by the same judges or the same pool of judges.
Talble captains can and do questions scores that are out of line for an explanation. But I would hope as quoted above they do not "nudged them back to the centerline" in the middle of a contest.

[TooSaucedToPork]

Access to such information is the big question for me as a judge, cook AND organizer.

It is the job of the organizer to select the judges that serve at their event(s). Assuming there is data available about the performance of judges, should this information be available to organizers? Would there be a potential for abuse if an organizer tried to pack an event with high-scoring judges in order to appeal to more teams?

Very interesting questions.

Yup...think if you could load each table with 1 high scorer, 4 average scorers, and 1 low scorer...

Even playing field...

just sayin

As far as abuse...the organizer would be exposed and booted for doin it. Other organizers or reps would never let that happen.

[BMerrill]

Access to such information is the big question for me as a judge, cook AND organizer.

It is the job of the organizer to select the judges that serve at their event(s). Assuming there is data available about the performance of judges, should this information be available to organizers? Would there be a potential for abuse if an organizer tried to pack an event with high-scoring judges in order to appeal to more teams?

Very interesting questions.

What I look at when considering a contest:

1) Location of the event/travel distance, typically limit to 6 hour drive one way.
2) Date /Time of the year.
3) How well is it organized. The number of teams is a good indicator.
4) Entry fees.
5) Number of teams, last year, current year, limit. (see 2 above)
6) Prize money.
7) Requirements beside the 4 meats.

[rhazzon]

One of the problems I've found is that there is nothing in place to educate judges beyond the KCBS rules. There are subtle nuances to judging that take time to learn. One of the things I'd love to see judges do is discuss their scores after the cards have been turned in. I certainly try to make it a point, especially when sitting next to a brand-new judge.

KCBS SCORE has the ability and does, in fact, track judges scores. Whether or not KCBS will review that data and use it to critique and make better judges is still to be seen. I do hope they do something with it.

Let's all keep in mind that this sport has EXPLODED in the past couple of years. Most of the old judges are BBQ'ers at heart and started with a pit in the backyard before ever judging a piece of meat. They understand the amount of work that is required to produce good BBQ. We now have to contend with the newfound growth in BBQ and the new breed of judges that saw Myron on TV and took a 4 hour judging class. They lack the passion for the sport. They make no effort to become better judges. But I don't think they'll be around for the long haul. Judging is an expensive hobby (contrary to what some teams may believe). Gas, tolls, hotel and meals adds up to around $200-250 a contest - far less that what the cooks lay out, but with no chance of any return. I think as the bubble bursts and things settle down a bit, everything will return to normal.

I take my hat off to all of the teams that work as hard as they do to produce some of the best BBQ in the world. I wish each and every one of you the best of luck. I can't speak for others, but can promise to be as fair (and sometimes forgiving :wink:) as possible when judging your entries.

[Ron_L]

One of the problems I've found is that there is nothing in place to educate judges beyond the KCBS rules.

The rules aren't the only thing... KCBS put up a continuing education program for judges a while back. It is designed to be a study guide for the Master Judge test, but I think it would be great review for all judges.

http://www.kcbs.us/judges-education.php

But, I wonder how many even know that it exists.

[Ford]

The rules aren't the only thing... KCBS put up a continuing education program for judges a while back. It is designed to be a study guide for the Master Judge test, but I think it would be great review for all judges.

http://www.kcbs.us/judges-education.php

But, I wonder how many even know that it exists.

I'm a CBC but don't judge. But I took the online test for fun. 1 wrong. Thought the question was poorly worded. Should have new test every year and CBJ must get 80% or take the course again.

[Ron_L]

I'm a CBC but don't judge. But I took the online test for fun. 1 wrong. Thought the question was poorly worded. Should have new test every year and CBJ must get 80% or take the course again.

I agree on all points, Ford. I have heard that the wording is like that on the Master test, too.

[Ford]

Ok, enough of this s#%t. I'm talking kcbs here. The organizer is responsible for selecting judges. They can use cbj's or drunks off the street. The kcbs has no control of this. The cook team can look at CBJ % for the previous year and if it was 100% then maybe all cbj's will show and it will be 100% again.

There are many reasons why 1 sample can be different in taste and tenderness than the other 5 sampled. So the different score, higher or lower, is justified. Trying to tell a judge they were low balling based on one score is BS. If kcbs can now track judges scores and look at the last 20 contests a judge did then maybe they can see a trend. Of course a lot of "local" judges may do less than 5 contests so it would be 6 years until you have minimum data sample. Not much help.

How many who complain about scores do practice runs at home, where you the cook tries every piece you put in the box after it sits for 20 minutes. I bet not many. I have and I see differences in pieces. More sweet, more heat, one less tender, some mushy, some tough.

Finally, until we have well trained and paid judges controlled by the kcbs, this is a volunteer job. Experienced Cooks understand this. You pay your entry and take your chances.

[TooSaucedToPork]

Ok, enough of this s#%t. I'm talking kcbs here. The organizer is responsible for selecting judges. They can use cbj's or drunks off the street. The kcbs has no control of this. The cook team can look at CBJ % for the previous year and if it was 100% then maybe all cbj's will show and it will be 100% again.

There are many reasons why 1 sample can be different in taste and tenderness than the other 5 sampled. So the different score, higher or lower, is justified. Trying to tell a judge they were low balling based on one score is BS. If kcbs can now track judges scores and look at the last 20 contests a judge did then maybe they can see a trend. Of course a lot of "local" judges may do less than 5 contests so it would be 6 years until you have minimum data sample. Not much help.

How many who complain about scores do practice runs at home, where you the cook tries every piece you put in the box after it sits for 20 minutes. I bet not many. I have and I see differences in pieces. More sweet, more heat, one less tender, some mushy, some tough.

Finally, until we have well trained and paid judges controlled by the kcbs, this is a volunteer job. Experienced Cooks understand this. You pay your entry and take your chances.

Let me state I'm not Judge bashing. Without CBJ's we would be cooking mushy, lifeless, fall off the bone ribs for celebrity judges. ;-)

I don't believe all bad scores are because of bad/untrained judges...the truth is some are, but us cooks do have a big hand in bad scores as well. AND...ya, most of us teams/cooks practice a lot. We do let our food sit, and then try it, we experiment. That is the only way to taste what a judge tastes. Plus many of us cooks are CBJ's.

What I am saying is clean house of the bad "volunteers". Identify and Retrain or if they refuse, boot em from judging until they agree to a new class. I know they exist as I have sat next to them. This isn't a proposal for a judge witchhunt, and all judges are not condemned...just the very few "volunteers" who give judges a bad name.

[DawgPhan]

What I look at when considering a contest:

1) Location of the event/travel distance, typically limit to 6 hour drive one way.
2) Date /Time of the year.
3) How well is it organized. The number of teams is a good indicator.
4) Entry fees.
5) Number of teams, last year, current year, limit. (see 2 above)
6) Prize money.
7) Requirements beside the 4 meats.


None of those items would be at the top of my list when I consider a contest. The organizer and the reps have the greatest impact to my experience. If the organizer and the reps are solid I know I am going to get a fair shake in the tent and a good time in the field.

More teams should stop checking google maps and start checking reps before they send in their check and there would be a lot less moaning and groaning.

[Atlasman]

2) Disparate scores are most common in chicken, where you are typically cooking separate pieces of meat. It is entirely possible for one thigh of six to deserve significantly lower scores than the other five.

Someone gives me a 7 in a sea of 8's and 9's and I can see that.......6 would raise an eyebrow but 5??.....no way, especially on further review from a judge who low balled every entry he got.



3) Reps work to distribute boxes equally to all tables to minimize the damage of a low-scoring judge or table.

Silly IMO.......that's admitting you have crappy judges and just trying to limit there damage. Why not just hold them accountable to not be crappy??

[Atlasman]

You got a bad score. It happens..


I've heard many teams voice this outlook.............why are so many people willing to just accept bad judging as if it HAS to be part of the process??


I work way too hard and spend entirely too much time and money to put something special in those boxes to have it dismissed due to a lack of seriousness or some personal axe to grind.

[Gowan]

Someone gives me a 7 in a sea of 8's and 9's and I can see that.......6 would raise an eyebrow but 5??.....no way, especially on further review from a judge who low balled every entry he got.

It can and does happen, even to the best of cooks. I've seen a piece of chicken that was raw in the middle from a cook with dozens of GCs when all the other judges were giving 8s and 9s to thighs from the same box. Unless you personally bit the piece of chicken in question, you simply cannot be certain whether or not the judge was justified in the score you received. Could have been an undeserved low score.... or not.



Silly IMO.......that's admitting you have crappy judges and just trying to limit there damage. Why not just hold them accountable to not be crappy??

No, that is being realistic about human nature and doing what you can to minimize complaints like yours. That's why the table rotation procedure is KCBS policy.

[BigBellyBBQ]

Yup...think if you could load each table with 1 high scorer, 4 average scorers, and 1 low scorer...

Even playing field...

just sayin

As far as abuse...the organizer would be exposed and booted for doin it. Other organizers or reps would never let that happen.
This is a good point, the new score sheets really bring truth to the good table / bad table theory from the past. Look at the scores and most of the catagory winners come from the same judging table and the last place tier came from a tight table.
If the judges averAGE were figured in it might level the playing field. Maybe for the bigger events change to a second round where all the table winners would go to a final round and all entries go to 1 final table. I would have to look at the numbers again however I was 1st at the table and 11 in the tally..
Even if they put all the table winners in a hat and drew final order would be better than what we have now!
My oppinion on the judge that scores a 5 when everyone else is 9 or 8, is when a table captain has to ask why?

[TooSaucedToPork]

it might level the playing field. Maybe for the bigger events change to a second round where all the table winners would go to a final round and all entries go to 1 final table. I would have to look at the numbers again however I was 1st at the table and 11 in the tally..


MBN does this. It would not work for Kcbs for many reasons...cooking capacity of most teams, cost, etc.

[DawgPhan]

I've heard many teams voice this outlook.............why are so many people willing to just accept bad judging as if it HAS to be part of the process??


You got a bad score. You may or may not have gotten bad judging. There is a difference. You can't confuse correlation and causation. Right now all you know is that you got a low score(that was dropped) and 5 solid but not great scores.

The lowest score did not impact your final standings since that score was dropped. It appears to me that the system worked as designed in your case. You should thank the reps, organizer and KCBS to making that happen. I am sure they would appreciate it.

[Lake Dogs]

One of the problems I've found is that there is nothing in place to educate judges beyond the KCBS rules. There are subtle nuances to judging that take time to learn. One of the things I'd love to see judges do is discuss their scores after the cards have been turned in. I certainly try to make it a point, especially when sitting next to a brand-new judge.

KCBS SCORE has the ability and does, in fact, track judges scores. Whether or not KCBS will review that data and use it to critique and make better judges is still to be seen. I do hope they do something with it.

Let's all keep in mind that this sport has EXPLODED in the past couple of years. Most of the old judges are BBQ'ers at heart and started with a pit in the backyard before ever judging a piece of meat. They understand the amount of work that is required to produce good BBQ. We now have to contend with the newfound growth in BBQ and the new breed of judges that saw Myron on TV and took a 4 hour judging class. They lack the passion for the sport. They make no effort to become better judges. But I don't think they'll be around for the long haul. Judging is an expensive hobby (contrary to what some teams may believe). Gas, tolls, hotel and meals adds up to around $200-250 a contest - far less that what the cooks lay out, but with no chance of any return. I think as the bubble bursts and things settle down a bit, everything will return to normal.

I take my hat off to all of the teams that work as hard as they do to produce some of the best BBQ in the world. I wish each and every one of you the best of luck. I can't speak for others, but can promise to be as fair (and sometimes forgiving :wink:) as possible when judging your entries.

A little side to the topic, but to this point you made:
> One of the things I'd love to see judges do is discuss their scores after the cards have been turned in. I certainly try to make it a point, especially when sitting next to a brand-new judge.

On the one hand this is extremely helpful to new judges, BUT it's funny how you say something and 10 people hear 10 different things... For example, there is nothing in the rule that says all X number of pieces of chicken need to be the same size. Yet, how many judges take off for this? Mind you, if it's less appetizing to you personally because they're not all perfectly sized and symetrical, that's your opinion of appearance and it should be scored likewise. However, some judges end up coming away with some notion that if they're not all the same size there are points to be taken away regardless of how appetizing it is or isn't... This is just one example; there are thousands. What's not stressed in these little afterwards meetings are the fundamentals and basics, like "simply, is/was it appetizing in appearance to you." Instead they'll say "it didnt have XXXXXX in the box, so I took off for that, or "there was no skin on the chicken, so I took off for that", etc. It's amazing how many judges dont know that there is no rule requiring skin on chicken.

[BaggerBill]

As a new cook/competitor I have yet to be convinced that "I got a low score from one judge and I should accept it". If I'm getting 9's & 8's from all judges and 5's from one, then I'm going to look at it closer. If I see that judge was low balling everyone all day long then I'm going to view this judge as someone who sees them self as God's gift to BBQ and I want accountability (or in the very least a comment card). Dropping the low score doesn't make this any better either, it's something we say to try to justify it. LOw ball judges should be held accountable and hopefully comment cars in some shape or form will be brought back.

[gettinbasted]

As a new cook/competitor I have yet to be convinced that "I got a low score from one judge and I should accept it". If I'm getting 9's & 8's from all judges and 5's from one, then I'm going to look at it closer. If I see that judge was low balling everyone all day long then I'm going to view this judge as someone who sees them self as God's gift to BBQ and I want accountability (or in the very least a comment card). Dropping the low score doesn't make this any better either, it's something we say to try to justify it. LOw ball judges should be held accountable and hopefully comment cars in some shape or form will be brought back.

What about high ball judges? They can cost you a contest just as much as a low judge. We will all end up with straight 8s and a tie, but our feelings won't be hurt!

[mobow]

From a lot of experience in the judges tent I can tell you that there are a significant number of 999 judges out there and while a cook may like to get them they make the 978 judge look bad and give the cook a false read on the quality of his turn in. Keith

[DawgPhan]

As a new cook/competitor I have yet to be convinced that "I got a low score from one judge and I should accept it". If I'm getting 9's & 8's from all judges and 5's from one, then I'm going to look at it closer. If I see that judge was low balling everyone all day long then I'm going to view this judge as someone who sees them self as God's gift to BBQ and I want accountability (or in the very least a comment card). Dropping the low score doesn't make this any better either, it's something we say to try to justify it. LOw ball judges should be held accountable and hopefully comment cars in some shape or form will be brought back.


Getting past blaming the judges makes you a better cook.

[TooSaucedToPork]

Getting past blaming the judges makes you a better cook.

And admitting there are bad judges out there makes you strive to be a better judge.:icon_smile_tongue:

[ITBFQ]

Let's all keep in mind that this sport has EXPLODED in the past couple of years. Most of the old judges are BBQ'ers at heart and started with a pit in the backyard before ever judging a piece of meat. They understand the amount of work that is required to produce good BBQ. We now have to contend with the newfound growth in BBQ and the new breed of judges that saw Myron on TV and took a 4 hour judging class. They lack the passion for the sport. They make no effort to become better judges. But I don't think they'll be around for the long haul. Judging is an expensive hobby (contrary to what some teams may believe). Gas, tolls, hotel and meals adds up to around $200-250 a contest - far less that what the cooks lay out, but with no chance of any return. I think as the bubble bursts and things settle down a bit, everything will return to normal.

Listen, I am a fairly new CBJ... you are saying that I don't have passion for the sport? I don't give a *&%! about those guys on TV - sure, they make good TV, but they aren't the reason I decided to drive 4 hours one way to take a CBJ class. I have done everything I can to be the best possible judge I can be because I understand that is what teams who are there to seriously compete deserve. When I judge, I usually give up an entire day, I'm away from my family, things that need to get done, etc. so I can judge BBQ, and I take that pretty seriously. I am pretty sure that 95% of judges, whether new or seasoned, are the same. We all know that there are judges out there that do not follow what they are taught in the class... they need to be "retaught." But don't say that just because a judge is new they lack passion.

[BaggerBill]

Getting past blaming the judges makes you a better cook.

Every time I light the fire I do what I can to become a better cook. I have my food judged at comps so I know what direction to move towards in order to become a better cook. When five judges score it high and one judges it low and that same judge has been low balling all day, do I change to satisfy Judge Lowball or keep doing what I'm doing to please the majority.

Nobody is blaming the judge"S", just wanting accountability from the lowball judge. At $400-1000 per comp it's not asking a lot no matter how you try to justify it.

[DawgPhan]

Every time I light the fire I do what I can to become a better cook. I have my food judged at comps so I know what direction to move towards in order to become a better cook. When five judges score it high and one judges it low and that same judge has been low balling all day, do I change to satisfy Judge Lowball or keep doing what I'm doing to please the majority.

Nobody is blaming the judge"S", just wanting accountability from the lowball judge. At $400-1000 per comp it's not asking a lot no matter how you try to justify it.


Accountability for what exactly?

That you have jumped to the conclusion that his score was inaccurate? Bad scores does not equal bad judging. You can't make that conclusion from the data provided on the score sheet.

[cpw]

Accountability for what exactly?

That you have jumped to the conclusion that his score was inaccurate? Bad scores does not equal bad judging. You can't make that conclusion from the data provided on the score sheet.

I'm not arguing with you, but how do you account for an entire table's worth of top tier teams getting the lowest scores on multiple categories at the recent Sam's regional in Hendersonville? There's no way that that many teams scored that badly all on the same table unless there's an issue with the judging.

[rhazzon]

Listen, I am a fairly new CBJ... you are saying that I don't have passion for the sport?

You've misunderstood my statement. I was not referring to new judges, but the new *BREED* of judges that are not at the table for the same reason as you and I. BBQ is the new fad and lots of people are jumping on the bandwagon for the cool factor, not for the love of the sport.

As you judge more and more contests and spend time getting to know the other CBJ's, you'll see what I mean. There are those with passion and those that show up to be fed.

[BaggerBill]

Accountability for what exactly?

That you have jumped to the conclusion that his score was inaccurate? Bad scores does not equal bad judging. You can't make that conclusion from the data provided on the score sheet.

Accountable for scoring low while the rest of the table scored within reason of each other. It's that simple. Please explain how bad scoring doesn't equal bad judging in your opinion? If 5 of 6 judges score 8's & 9's while the remaining judge scores 5, either judge lowball is bad or he/she is the only good judge and the rest suck.

In that case that is exactly what they should be accountable for.

If all judges score food as bad then great, thats a perfect example of bad scores not equalling bad judging.

[CaptTable]

We DO NOT address a judge who seems to be scoring too low or too high ( I've had to speak to a judge who gave ALL 9's ALL day) until AFTER all judging is complete. That way EVERYONE is on an even playing field throughout the contest.

Phillip

[TooSaucedToPork]

Accountable for scoring low while the rest of the table scored within reason of each other. It's that simple. Please explain how bad scoring doesn't equal bad judging in your opinion? If 5 of 6 judges score 8's & 9's while the remaining judge scores 5, either judge lowball is bad or he/she is the only good judge and the rest suck.

In that case that is exactly what they should be accountable for.

If all judges score food as bad then great, thats a perfect example of bad scores not equalling bad judging.

Not entirely true brother.

Dawg is right ESPECIALLY with Ribs and Chicken. You as a cook can turn in a single bad piece of chicken or ribs taste and tenderness wise. BUT if you see the appearance score is low as well...that is a good indication that judge is a lowballer.

Example:

988, 989, 999, 988, 877, 899
this is probably a bad piece of meat.

Now if you get this
988, 989, 999, 988, 777, 899

We need accountability and tracking for this type of situtation. That should warrant a review and questioning of the judge.

[TooSaucedToPork]

I'll give you an example. What would you score this in appearance.

https://scontent-b-ord.xx.fbcdn.net/hphotos-prn1/544556_696889743672996_1622239005_n.jpg

It got 778,777,989,989,899,899 11th place first on the table

the next day...same recipe, same exact look.

888,898,999,999,999,999 1st place, first on table


Shoulda garnished the first one with hundred dollar bills. hahaha

DON'T ACTUALLY POST A SCORE...IT IS AN EXAMPLE.
Read Below

[RobKC]

Those meatballs :wink: get 8 from me on appearance because of the gloppy sauce.

[BaggerBill]

I'll give you an example. What would you score this in appearance.

https://scontent-b-ord.xx.fbcdn.net/hphotos-prn1/544556_696889743672996_1622239005_n.jpg

It got 778,777,989,989,899,899 11th place first on the table

the next day...same recipe, same exact look.

888,898,999,999,999,999 1st place, first on table


Shoulda garnished the first one with hundred dollar bills. hahaha

I'm not sure why the 7's, I think it's a good looking box and the next days scores really do contradict it. BUT.....if I was given the scores you posted I probably wouldn't have had (much of) a problem since you had 2 each of 9, 8 & 7's. The table was split evenly. Had it been 4 nines and 2 sevens and I would've raised an eyebrow, 5 nines and 1 seven I probably would've muttered a "WTF?" but thats just my opinion.

If I scored the box it would not have been less than an 8, somebody mentioned the sauce. To me, its to hard to tell if the sauce is thick or just appears that way from the lighting in the picture.

[bbq.tom]

We DO NOT address a judge who seems to be scoring too low or too high ( I've had to speak to a judge who gave ALL 9's ALL day) until AFTER all judging is complete. That way EVERYONE is on an even playing field throughout the contest.

Phillip

I must disagree, as I was a TC this past weekend and pointed out to the REPs that a judge was giving 5/6 when the other five judges were 8/9 and the REP spoke to the low judge immediately (after Chicken). Didn't seem to help too much as he was still lower than the rest of the table the rest of the contest, just not quite as low.

The comment cards ARE coming back soon! They will be "mark the box" with a place for written comments. Just trying to get them printed and distributed right now. Should be better than the old system.

I still contend that it would be best for judges to HAVE to comment on EVERY score that they give. This way every score is justified by the judge and hopefully thought about better. Also, the team would know exactly what the judges scored on.

[DawgPhan]

Not entirely true brother.

Dawg is right ESPECIALLY with Ribs and Chicken. You as a cook can turn in a single bad piece of chicken or ribs taste and tenderness wise. BUT if you see the appearance score is low as well...that is a good indication that judge is a lowballer.

Example:

988, 989, 999, 988, 877, 899
this is probably a bad piece of meat.

Now if you get this
988, 989, 999, 988, 777, 899

We need accountability and tracking for this type of situtation. That should warrant a review and questioning of the judge.


Exactly right...the low AP score is the one to look for.

Pork can have pockets of fat or injection as well that makes one judge score low. Brisket is really the only meat where each slice should be about the same.

[TooSaucedToPork]

Thats what I'm saying...perception is everything. One judge sees gloppy sauce, another sees a perfect chicken.

It was balanced. Then it was through the roof.

I didn't post it to ACTUALLY post a rating...I posted it to show balance. I looked at the new score sheet. And by their cumulitative scores:

The first table had 2 lowballers, 2 highballers, and 2 average judges...a balanced table
The second had 3 highballers and 3 average judges...not balanced.

Balance needs to be brought to the force. Did we win because of a table imbalance...I don't want to accept that, now is that the truth...good possibility it is. Should the highballers be retrained...yes. So should the lowballers. Highballers hurt teams just as much as lowballers.

Judges need to be held accountable, or at least use the new software to balance tables according to average score. Will mistakes be made, yes they will, but it is a step in the direction of true fairness in competition.

and by the way it was a Very thin sauce, just a shiny glaze. :-D

[Ford]

Accountable for scoring low while the rest of the table scored within reason of each other. It's that simple. Please explain how bad scoring doesn't equal bad judging in your opinion? If 5 of 6 judges score 8's & 9's while the remaining judge scores 5, either judge lowball is bad or he/she is the only good judge and the rest suck.

In that case that is exactly what they should be accountable for.

If all judges score food as bad then great, thats a perfect example of bad scores not equalling bad judging.
Or they got a bad piece of meat. It happens as much as we don't want to believe. An injection where on seasoning is dominant, a spot of too much sauce. It goes on and on with reasons for a bad sample. People who don't realize this either don't cook much or don't ample everything when practicing.

[Ford]

Thats what I'm saying...perception is everything. One judge sees gloppy sauce, another sees a perfect chicken.

It was balanced. Then it was through the roof.

I didn't post it to ACTUALLY post a rating...I posted it to show balance. I looked at the new score sheet. And by their cumulitative scores:

The first table had 2 lowballers, 2 highballers, and 2 average judges...a balanced table
The second had 3 highballers and 3 average judges...not balanced.

Balance needs to be brought to the force. Did we win because of a table imbalance...I don't want to accept that, now is that the truth...good possibility it is. Should the highballers be retrained...yes. So should the lowballers. Highballers hurt teams just as much as lowballers.

Judges need to be held accountable, or at least use the new software to balance tables according to average score. Will mistakes be made, yes they will, but it is a step in the direction of true fairness in competition.

and by the way it was a Very thin sauce, just a shiny glaze. :-D

How does this address celebrity judges?

[TooSaucedToPork]

How does this address celebrity judges?

Celebrity Judges are worthless...They only way to ensure they don't screw thing up would be to give them a chart that details descriptions of 1-9

or let them judge "best hamburger"...or Kids Q

My opinion, They have no place judging a KCBS contests 4 main categories, unless they are a certified judge.

[BaggerBill]

Or they got a bad piece of meat. It happens as much as we don't want to believe. An injection where on seasoning is dominant, a spot of too much sauce. It goes on and on with reasons for a bad sample. People who don't realize this either don't cook much or don't ample everything when practicing.

Yeah, I get it, a bad piece of meat...but if the same judge on the same table is consistently scoring low (like I said time and time again), then the bad piece of meat theory is out the window and the judge should be held accountable for lowball scoring....unless the judge is VERY unlucky and got bad meat all day long.

[Ford]

Yeah, I get it, a bad piece of meat...but if the same judge on the same table is consistently scoring low (like I said time and time again), then the bad piece of meat theory is out the window and the judge should be held accountable for lowball scoring....unless the judge is VERY unlucky and got bad meat all day long.
I agree. Remember I also cook FBA and have seen judges scores for years. Info tells me if maybe it's just a lowball judge or maybe they really didn't like something. Still I'm not making changes based on one judge at one contest. As far as doing anything about a low scoring judge I go back to my earlier post. You'd need at least 20 contests before being able to make a case to take action against judges.

[Ford]

Celebrity Judges are worthless...They only way to ensure they don't screw thing up would be to give them a chart that details descriptions of 1-9

or let them judge "best hamburger"...or Kids Q

My opinion, They have no place judging a KCBS contests 4 main categories, unless they are a certified judge.

Are you a CBJ?

[TooSaucedToPork]

Are you a CBJ?


Yup

KCBS, MBN, MIM

[Ford]

I have a solution. Pay judges $100 per contest plus a room for the night. Cook team fees go up $175 each. Require cbj's or no contest. Kcbs then hires somebody to monitor judges. Of course it's up to kcbs to find judges for contests. Also some contests will drop kcbs sanctioning to keep sponsors happy. Believe it or not sponsors want some perks like judging.

OR cooks can accept the fact judges are human and have different opinions on what's good BBQ and what's average. Sometimes you hit the good table, sometime the bad. That's life. IT'S JUST BBQ.

[Ford]

Yup

KCBS, MBN, MIM

So you know the CBJ course is a joke and there's no follow up. It's just a way to expand membership. I've done a lot of contests up north with under 50% cbj's and it's been more consistent than having a contest the day after a CBJ class where half the cbj's are brand new.

[TooSaucedToPork]

So you know the CBJ course is a joke and there's no follow up. It's just a way to expand membership. I've done a lot of contests up north with under 50% cbj's and it's been more consistent than having a contest the day after a CBJ class where half the cbj's are brand new.

And that is what we are complaining about...NO ACCOUNTABILITY

I have a solution. Pay judges $100 per contest plus a room for the night. Cook team fees go up $175 each. Require cbj's or no contest. Kcbs then hires somebody to monitor judges. Of course it's up to kcbs to find judges for contests. Also some contests will drop kcbs sanctioning to keep sponsors happy. Believe it or not sponsors want some perks like judging.

Or we can set you up in a chair next to the cook area...like many of us sleep.
Reps already monitor Judges...thats part of their duties.
If you drop sanction...you will lose teams. plain and simple.
Sponsors...So let them judge the ancilliaries...

That's life. IT'S JUST BBQ.

To some of us, BBQ IS OUR LIFE.

OR cooks can accept the fact judges are human and have different opinions on what's good BBQ and what's average

This^^^ is a major problem. Judges that put personal preference above rules. That is why accountability is needed. There are rules that define good bbq...some judges choose to ignore those rules...they need to be held accountable by being tracked, retrained, and if uncooperative...Booted.


Tell ya what...

Judging issues will be one of the downfalls of KCBS. People draw lines in the sand on this issue...Its like the Roe vs Wade of BBQ.

I heard a quote, "If it ain't broke...BREAK IT". Too many organizations sit back with the "If it ain't broke...don't fix it" attitude. That is how you get made obsolete. VHS wholluped Beta, Apple wholluped Microsoft, and WalMart wholluped everyone.

If you don't change and adapt, ahead of the game, you die or play a really chitty game of catchup. By letting a problem fester, and accountability is a HUGE problem in KCBS, the organization is setting itself up for future problems.

In the next 5 to 10 years I forsee a group of BBQers getting together to create a new organization, focused on the cooks, and accepting of already certified KCBS judges. They will break the current system, and KCBS will struggle to keep up. It has happened before...it will happen again. I hope KCBS decides to be proactive now and fix the problem before a fracture occurs in the organization.

[Rookie'48]

In the next 5 to 10 years I forsee a group of BBQers getting together to create a new organization, focused on the cooks, and accepting of already certified KCBS judges. They will break the current system, and KCBS will struggle to keep up. It has happened before...it will happen again. I hope KCBS decides to be proactive now and fix the problem before a fracture occurs in the organization.

So, what do you suggest that KCBS do to identify and correct these problems?

[TooSaucedToPork]

So, what do you suggest that KCBS do to identify and correct these problems?

1. Yearly tests for judges...85% or you fail
A password protected, trackable, interactive, online refresher course should be developed to be given to those who fail. If they fail the test at the end twice, they will need to retake their CBJ certification in a face to face class

2. Tracking of low scoring AND High scoring judges.
As above only instructing judges on the exact rules and and mechanics of judging...stressing what each number 1-9 means and how it should be applied.

Example: Apperarance
This could be in the form of pictures of turn in boxes with bubbles 2-9 they would rate a box, and then give a detailed reason for their rating. This would identify judges that need training in what to deduct points for and what not to deduct points for.

3. Identification of problem judges
Database should be started on judges with reported problems. Corrective Actions should be added to files when incidents occur.

4. Better training of new judges and current judges
Business card sized plastic cheat sheets with BBQ taste definitions on one side, and explanations of the 1-9 scale on the other side.

5. More interactions with teams.
Many judges don't ever set a single foot in a teams cook area. Setting a rule that after, lets say 5 contests, there is a tour led by a volunteer of team cook areas at a contest. Teams interact with judges (we don't bite...hard) and judges with teams. Meaningful conversation happens, the walls are broken down, understanding is reached on both sides.

6. Conversations about entries at tables
Table Captains could be trained, or given a script, to initiate meaningful conversation at tables between entries. This allows peer to peer education.

7. Use the new software to observe regional and national judging trends to identify where possible introduction of training materials could be of help.

Thats a few off the top of my head.

Many Organizations require their members to have a certain amount of training throughout the fiscal year...why not KCBS.

[Rookie'48]

With your permission I'll copy your post to the CBJ Committee for discussion. The Committee meets on the last Tuesday of each month by phone.

[TooSaucedToPork]

You have my permission sir.

neil

[Hawg Father of Seoul]

and WalMart wholluped everyone.



I guarantee they know when and why they sell every item on every shelf. Statistical analysis is what puts them ahead.

HOWEVER, the high and low ballers exist in the population and IF you want a score representative of the population.... tread lightly.

Most people can not accurately describe the results of a statistical analysis, let alone perform the computations. I fully expect a criterion problem and regression to the mean, but what do I know.

[gettinbasted]

I guarantee they know when and why they sell every item on every shelf. Statistical analysis is what puts them ahead.

HOWEVER, the high and low ballers exist in the population and IF you want a score representative of the population.... tread lightly.

Most people can not accurately describe the results of a statistical analysis, let alone perform the computations. I fully expect a criterion problem and regression to the mean, but what do I know.

Straight 8s... Statistically regress me to Truebud and I will be happy.

[TooSaucedToPork]

I guarantee they know when and why they sell every item on every shelf. Statistical analysis is what puts them ahead.

HOWEVER, the high and low ballers exist in the population and IF you want a score representative of the population.... tread lightly.

Most people can not accurately describe the results of a statistical analysis, let alone perform the computations. I fully expect a criterion problem and regression to the mean, but what do I know.

Correct, High and low ballers do exist in the population. So make it simple...for now.

You have a history of each Judge's scores at contests being tracked by the new software. Stick 4 average, 1 high and 1 low at each table...balance the population. That is a vast improvement and prevents "TABLE 5!!!"

:-)

[BigBellyBBQ]

Neil, you have some great points, keep it going!! Pete

[Atlasman]

I have a solution. Pay judges $100 per contest plus a room for the night. Cook team fees go up $175 each. Require cbj's or no contest. Kcbs then hires somebody to monitor judges. Of course it's up to kcbs to find judges for contests. Also some contests will drop kcbs sanctioning to keep sponsors happy. Believe it or not sponsors want some perks like judging.

OR cooks can accept the fact judges are human and have different opinions on what's good BBQ and what's average. Sometimes you hit the good table, sometime the bad. That's life. IT'S JUST BBQ.


Not directing this at you personally as I have seen/heard this perspective from many people so I'm sure it's not uncommon.

I just think that is the worst possible outlook and one that not only accepts subpar judging but actually invites and perpetuates it as well.

I'm sorry but the contests are NOT about the judges, or the reps for that matter...........why should they be paid and put up in hotels when we have cooks sleeping in the rain (if they sleep) and putting up hundreds of dollars of their own money and many hours of their time??? These judges get to eat all day from food coddled over a fire for more hours then most people work in a day. They get to eat some of the best BBQ the region has to offer in return for following a simple set of rules to record how good it was. Sounds like a pretty good deal to me.

The contests are about the cooks and the cooks alone. We practice for countless hours at home and spend thousands of dollars perfecting our actions along the way. We pay hundreds to a thousand dollars for each show we attend and then spend all day and night fussing over 4 meats on a strict timeline to deliver something that is not only visually stunning but also makes your taste buds dance.

I am EXHAUSTED at 2:00pm on Saturday. Mentally and physically drained. I'm willing to bet that all a judge would say at that point in time is how full they are.

I am not bashing judges here either so please if anyone is a good judge don't take offense as we love you and what you do............it's just some in your ranks don't realize the consequences of their actions or have a personal axe to grind and just don't care............this is very upsetting when these people are allowed to have DIRECT influence on whether or not I get to go to The Jack or where I end up in a single contest or cumulative championships.

[Atlasman]

For all the people saying I have no way of knowing if my piece of chicken deserved the 5 it got I will say two things.

That's the point!!! I don't know and no one else but the judge does so like I said he should have to explain himself. HE is the outlier.......the burden of proof should fall on him. I have 5 people that sampled out of the same box that say different. Again, be accountable for your actions.........if you are gonna go against EVERY judge at your table you should have to explain why.

Getting lost in the mix here is the avg scores he gave all day. Bravo to KCBS for putting this in now. He was 3-5 PTs lower then the rest of the table ALL DAY. So not only did he miraculously get the only bad piece in my box but every other entry as well.


I doubt that very much.

[TooSaucedToPork]

Not directing this at you personally as I have seen/heard this perspective from many people so I'm sure it's not uncommon.

I just think that is the worst possible outlook and one that not only accepts subpar judging but actually invites and perpetuates it as well.

I'm sorry but the contests are NOT about the judges, or the reps for that matter...........why should they be paid and put up in hotels when we have cooks sleeping in the rain (if they sleep) and putting up hundreds of dollars of their own money and many hours of their time??? These judges get to eat all day from food coddled over a fire for more hours then most people work in a day. They get to eat some of the best BBQ the region has to offer in return for following a simple set of rules to record how good it was. Sounds like a pretty good deal to me.

The contests are about the cooks and the cooks alone. We practice for countless hours at home and spend thousands of dollars perfecting our actions along the way. We pay hundreds to a thousand dollars for each show we attend and then spend all day and night fussing over 4 meats on a strict timeline to deliver something that is not only visually stunning but also makes your taste buds dance.

I am EXHAUSTED at 2:00pm on Saturday. Mentally and physically drained. I'm willing to bet that all a judge would say at that point in time is how full they are.

I am not bashing judges here either so please if anyone is a good judge don't take offense as we love you and what you do............it's just some in your ranks don't realize the consequences of their actions or have a personal axe to grind and just don't care............this is very upsetting when these people are allowed to have DIRECT influence on whether or not I get to go to The Jack or where I end up in a single contest or cumulative championships.

You gotta remember its about everyone...it's the circle of bbq life. The cools make the food. Certified Judges make judging the food fair. The reps bust as much butt as us cooks making the process smooth. Now don't forget the crowds that pay vendors, souvenirs, and admission, which pays for generators, land use, ice, etc. But then there are good organizers, that have just as many hairpulling, sleepless nights as is cooks.

But ya, without cooks no BBQ...but without everyone above...no contests

[BigBellyBBQ]

the bottom line is we all would like to win however just as long as we are treated fair we can handle it..We all know when we have a good cook or bad and when things are turned in when we see a 5 or 6 on appearence and the rest are all 9 ..what did only 1 piece out of 8 fall on the ground? These are the judges that need to be addressed..Like Neil wrote the person that is handing out 999 is also detrimental to our goals. No group should score the same but if the group of judges are averaged out then scoring system should work better. last week I had two entries fall on the table of death and with so many judges it should have been on a different table..but accountabilty is the key..

[DawgPhan]

For all the people saying I have no way of knowing if my piece of chicken deserved the 5 it got I will say two things.

That's the point!!! I don't know and no one else but the judge does so like I said he should have to explain himself. HE is the outlier.......the burden of proof should fall on him. I have 5 people that sampled out of the same box that say different. Again, be accountable for your actions.........if you are gonna go against EVERY judge at your table you should have to explain why.

Getting lost in the mix here is the avg scores he gave all day. Bravo to KCBS for putting this in now. He was 3-5 PTs lower then the rest of the table ALL DAY. So not only did he miraculously get the only bad piece in my box but every other entry as well.


I doubt that very much.

You are getting fired up over conclusions that you can't support with fact. You can't know that a judge was a low scoring judge based on this KCBS scoresheet. You just can't. The data isnt there.

That is probably the biggest drawback to this new scoresheet is that now everyone wants to play amateur analyst after a contest.

[Atlasman]

You gotta remember its about everyone...it's the circle of bbq life. The cools make the food. Certified Judges make judging the food fair. The reps bust as much butt as us cooks making the process smooth. Now don't forget the crowds that pay vendors, souvenirs, and admission, which pays for generators, land use, ice, etc. But then there are good organizers, that have just as many hairpulling, sleepless nights as is cooks.

But ya, without cooks no BBQ...but without everyone above...no contests


Yea.......certainly didn't mean to diminish anyone's roll in the contest setting as I'm sure everyone has tasks which are not easy or fun.


I still stand by assessment of the cooks in the food chain though.

[Atlasman]

You are getting fired up over conclusions that you can't support with fact. You can't know that a judge was a low scoring judge based on this KCBS scoresheet. You just can't. The data isnt there.

That is probably the biggest drawback to this new scoresheet is that now everyone wants to play amateur analyst after a contest.



I can't tell that he scored our entry 8-12 pts lower than the other 5 judges at the table and that he was consistently the lowest score at his table all day by 3-5 pts??


What data am I missing??

[DawgPhan]

I can't tell that he scored our entry 8-12 pts lower than the other 5 judges at the table and that he was consistently the lowest score at his table all day by 3-5 pts??


What data am I missing??


The fact that his scores were lower than other judges does not mean that his scores were wrong. You want him held accountable because you think that his scores are errors. His scores could be correct. There is not enough data in the scoresheet to support a call either way.

You dont know what scores went into the average score. You dont know what other tables he sat at. How he performed in comparison to his peers on other tables that your box didnt land on. The new scoresheet just provides a sliver of information about the judging, but is interpreted as comprehensive and it isnt.

Everyone should be concerned with judges that dont follow the guidelines and procedures of the sanctioning body. Judges that give accurate scores, high or low, should be encouraged.

Back to making sure you know the reps and organizers of any contest you attend.

[DawgPhan]

To put it another way, you have a little circumstantial evidence, no weapon, no motive, and no eye witness and you want a conviction. You can't make that call.

[BaggerBill]

You are getting fired up over conclusions that you can't support with fact. You can't know that a judge was a low scoring judge based on this KCBS scoresheet. You just can't. The data isnt there.

That is probably the biggest drawback to this new scoresheet is that now everyone wants to play amateur analyst after a contest.

Amateure analyst? Care to expand on this or are you jumping to conclusions on peoples ability (and training) to interpret numbers and statistics? You most certainly can tell if a judge is consistently scoring low or high just as you can tell what table is the table of death by looking at a score sheet.

[musicmanryann]

You dont know what scores went into the average score. You dont know what other tables he sat at. How he performed in comparison to his peers on other tables that your box didnt land on.

All of this information is on the new scoresheets, so yes, we know this information. At least all that except your comment about other tables he sat at. I didn't know judges were playing musical chairs in between turn-ins. My understanding is that Judge 2 at table 3 is the same judge throughout the entire contest. I guess I could be wrong though.

[BaggerBill]

All of this information is on the new scoresheets, so yes, we know this information. At least all that except your comment about other tables he sat at. I didn't know judges were playing musical chairs in between turn-ins. My understanding is that Judge 2 at table 3 is the same judge throughout the entire contest. I guess I could be wrong though.

Exactly! My understanding as well. Please correct me if I'm wrong on this.

[ModelMaker]

So you know the CBJ course is a joke and there's no follow up. It's just a way to expand membership. I've done a lot of contests up north with under 50% cbj's and it's been more consistent than having a contest the day after a CBJ class where half the cbj's are brand new.


The KCBS CBJ course does not make a person a credible judge. It simply teaches a novice the rules and procedures set down by an organization that provides same in a contest setting.
This person qualified by KCBS will not become a judge until they have experienced the process several,many,never times. A good judge will learn and adjust from table talk, comparing techniques of peer judging on BBQ Critic, following the lifestyle on a BBQ website, spending time with the BBQ community at the contest setting, on and on and on.
Taking a KCBS CBJ course is only a first step, the rest is up to the individual.
There.
Ed

[musicmanryann]

I'm not necessarily in favor of holding judges directly accountable for their judging patterns, unless it is an extreme circumstance, i.e. a judge being the dropped score for every entry at a contest or not knowing how to write any other number than "9" for 3 contests straight.

Giving judges a slap on the wrist every time they aren't within a certain degree of the table average has potential to alienate a lot of people, which I don't think is good for BBQ. I do think it would be helpful if judges were given information regarding their scoring patterns for their own information and use. Just like cooks get a scoresheet at the end of the contest, judges could as well. This by itself could be a powerful teaching tool. Right now, the only feedback they get is the occasional chat across the table in between categories.

[DawgPhan]

Amateure analyst? Care to expand on this or are you jumping to conclusions on peoples ability (and training) to interpret numbers and statistics? You most certainly can tell if a judge is consistently scoring low or high just as you can tell what table is the table of death by looking at a score sheet.


just making an observation based on the endless analysis that your guys think you are doing on these scoresheets.

You can't make the call about a judge consistently doing anything from the scoresheet. Thinking that you can is amateur.


For example.

A judge sits down as judge 6 at a rib table....a great looking box of ribs gets opened and passed around...judge 5 reaches in and grabs one rib, but the ribs arent cut cleanly and now judge 6 gets no ribs to judge. He gives the 1's for everything and that draws since average score down. Now some cook gets the scoresheet see that judge didnt score his brisket well, sees that his average score is lower than everyone else and declares him a bad judge that needs to be held accountable.

The judge may or may not be a bad judge, but you can't make the call from the scoresheet.

If you have good organizers and good reps, bad judges dont show up that often.

[musicmanryann]

A judge sits down as judge 6 at a rib table....a great looking box of ribs gets opened and passed around...judge 5 reaches in and grabs one rib, but the ribs arent cut cleanly and now judge 6 gets no ribs to judge. He gives the 1's for everything and that draws since average score down. Now some cook gets the scoresheet see that judge didnt score his brisket well, sees that his average score is lower than everyone else and declares him a bad judge that needs to be held accountable.


Again, the score the judge gave out for every entry is on the scoresheet so it is very easy to see if they gave out 1's on an entry and that is what brought their score down. There is more information on the scoresheet than just judges' average score that I am not sure you are aware of.

[Icekub]

Yea.......certainly didn't mean to diminish anyone's roll in the contest setting as I'm sure everyone has tasks which are not easy or fun.


I still stand by assessment of the cooks in the food chain though.


It's really more of a symbiotic relationship than a food chain because without any 1 or the participants (reps/organizers, cooks, judges) there is no competition. If there is a top of the food chain so to speak, I would have to give it to the rep/organizers. Granted, this is coming from a guy who has only cooked, but they work tirelessly on endless details and deal w/ way bigger financial #'s than either of the 2 other groups. They handle all the groups and also intially begin the whole process. I'm not sure which sanctioning body you compete in, but I know that KCBS's new scorecard is a step in the right direction. They just need to use the judge data already coming in to right some wrongs and in the future, the judging will be more consistent and improve. In the meantime, every cook is facing the same type of judging system (albeit imperfect) and that means everyone is being treated fairly.

[BaggerBill]

just making an observation based on the endless analysis that your guys think you are doing on these scoresheets.

You can't make the call about a judge consistently doing anything from the scoresheet. Thinking that you can is amateur.


For example.

A judge sits down as judge 6 at a rib table....a great looking box of ribs gets opened and passed around...judge 5 reaches in and grabs one rib, but the ribs arent cut cleanly and now judge 6 gets no ribs to judge. He gives the 1's for everything and that draws since average score down. Now some cook gets the scoresheet see that judge didnt score his brisket well, sees that his average score is lower than everyone else and declares him a bad judge that needs to be held accountable.

The judge may or may not be a bad judge, but you can't make the call from the scoresheet.

If you have good organizers and good reps, bad judges dont show up that often.

Are you kidding me? You are going to sit there and actually give that as your example, a rib that isn't cleanly cut? And use the word "amateur" towards others about this topic?

If I look at my score sheet and I see that five judges gave my ribs 8's & 9's while judge 3 gave them 5's I'm a bit curious. I look at the score sheet and see that my ribs were judged at table 2. All I have to do is look at what other ribs were scored at table 2 and see if judge 3 scored all ribs low. If he/she didn't score the others low, then great, my bad I gave the judge a crappy rib and lets move on to pork. But if I see judge 3 was the low number on all of boxes of ribs at that table on that day then I'm calling judge 3 a lowballer.

But wait! There's more! My curiosity has got the best of me now so I look at how judge 3 on table 2 scored chicken, pork and brisket. If judge 3 was the low score on that table time and time again for those categories then it's pretty much and indicator that judge 3 on table 2 lowballs the scores....and none of that involves a rib that wasn't cleanly cut.

All of that can be determine from one contest, not 20. Of course maybe the judge had a cold that day and his/her taste buds weren't working properly...

[Diva]

Interjecting....The last 2-3 contests we've cooked they've had more tables than they needed. One contest gave 4 boxes to each table. Another contest 4-5 boxes per table. I think that skews the results more than anything else. Thoughts??

[big matt]

just making an observation based on the endless analysis that your guys think you are doing on these scoresheets.

You can't make the call about a judge consistently doing anything from the scoresheet. Thinking that you can is amateur.


For example.

A judge sits down as judge 6 at a rib table....a great looking box of ribs gets opened and passed around...judge 5 reaches in and grabs one rib, but the ribs arent cut cleanly and now judge 6 gets no ribs to judge. He gives the 1's for everything and that draws since average score down. Now some cook gets the scoresheet see that judge didnt score his brisket well, sees that his average score is lower than everyone else and declares him a bad judge that needs to be held accountable.

The judge may or may not be a bad judge, but you can't make the call from the scoresheet.

If you have good organizers and good reps, bad judges dont show up that often.Just curious have you cooked a KCBS comp under the new scoring system?..there is now a ton of info on the sheet and it's very easy to figure out.

[Jorge]

Interjecting....The last 2-3 contests we've cooked they've had more tables than they needed. One contest gave 4 boxes to each table. Another contest 4-5 boxes per table. I think that skews the results more than anything else. Thoughts??

Statistically, I'd expect more deviation.

[TooSaucedToPork]

Like I said in a previous post. Lines are drawn in sand. BBQ Roe vs Wade haha

[musicmanryann]

Interjecting....The last 2-3 contests we've cooked they've had more tables than they needed. One contest gave 4 boxes to each table. Another contest 4-5 boxes per table. I think that skews the results more than anything else. Thoughts??

My first question would be why this is happening? To give more people an opportunity to be fed?? Is this SOP when extra judges show up?

I would also agree with the great Jorge.

[Diva]

My first question would be why this is happening? To give more people an opportunity to be fed?? Is this SOP when extra judges show up?

I would also agree with the great Jorge.

I don't know why this is happening. I want it to stop. I submitted critique sheets about it. I'll send a letter to the Board.

I know it's the organizers responsibility to get judges, but, I also believe it's the Contest Representatives responsibility to have the correct number of tables for the amount of cookers entered.

[Atlasman]

The fact that his scores were lower than other judges does not mean that his scores were wrong. You want him held accountable because you think that his scores are errors. His scores could be correct. There is not enough data in the scoresheet to support a call either way.

When did I say he was wrong???........I said he should have to explain such an outlier score. Especially when he does it all day.


Accountability.........plain and simple. I am held accountable for even the slightest errors in the boxes I submit. Why should a judge not be held to the same standard??

[Atlasman]

just making an observation based on the endless analysis that your guys think you are doing on these scoresheets.

You can't make the call about a judge consistently doing anything from the scoresheet. Thinking that you can is amateur.


For example.

A judge sits down as judge 6 at a rib table....a great looking box of ribs gets opened and passed around...judge 5 reaches in and grabs one rib, but the ribs arent cut cleanly and now judge 6 gets no ribs to judge. He gives the 1's for everything and that draws since average score down. Now some cook gets the scoresheet see that judge didnt score his brisket well, sees that his average score is lower than everyone else and declares him a bad judge that needs to be held accountable.

The judge may or may not be a bad judge, but you can't make the call from the scoresheet.

If you have good organizers and good reps, bad judges dont show up that often.



LOL..........have you even seen or cooked under the new system?? All that info IS easily available.

Since when do judges shuffle from table to table between each entry??

[Gowan]

Interjecting....The last 2-3 contests we've cooked they've had more tables than they needed. One contest gave 4 boxes to each table. Another contest 4-5 boxes per table. I think that skews the results more than anything else. Thoughts??

It is always tricky for organizers to match the number of judges to the number of cooks evenly because the two populations are about six months apart in their timing. Typically, judges plan their schedules as far in advance as possible, while cooks wait just as long as they can before sending in their applications.

As a result, the organizer is often faced with a situation where there may be one or more tables that receive more or less than six samples. I believe the current KCBS policy is to permit 4-8 boxes on a table for a category. In my experience, when a table is shorted on entries they tend to score higher overall, and lower when there are extra entries.

My theory is that the judge with four chicken boxes to judge is grateful they got a seat, whereas the judge staring down eight brisket samples is so full they are forcing it down.

[TooSaucedToPork]

One of y'all out there got a relative that is a statistics major? If so, this stuff has thesis written all over it...

[ModelMaker]

I don't know why this is happening. I want it to stop. I submitted critique sheets about it. I'll send a letter to the Board.

I know it's the organizers responsibility to get judges, but, I also believe it's the Contest Representatives responsibility to have the correct number of tables for the amount of cookers entered.

Steph Darlin,
I've thunk on this for a bit and don't believe that numerically this practice would be any different. As long as each table has 6 judges, your 4 box chicken entry tray has the same statiscal advantage as a tray of 6 entrys.
I had a contest last weekend that had extra judges and a 7th table
was set up to allow them a chance to judge. Not a single judge at my table even mentioned there were only 5 entrys per tray.
Ed

[Jorge]

Access to such information is the big question for me as a judge, cook AND organizer.

It is the job of the organizer to select the judges that serve at their event(s). Assuming there is data available about the performance of judges, should this information be available to organizers? Would there be a potential for abuse if an organizer tried to pack an event with high-scoring judges in order to appeal to more teams?

Very interesting questions.

I thought about this for a while. My answer would be, no. I'd go further and suggest that contest Reps probably shouldn't have access to the database either. I don't think the odds of wrongdoing are high, but that data could be used pretty easily make things less than fair.

The other issue I have, and is more important to me, is that I don't want to see judges excluded based on scoring. That makes the system punitive. The system I'd like to see put in place would offer guidance and education opportunities to get a judges scores in line, whether they are higher or lower than the norm on a consistent basis. If there is a reason for them to be excluded it needs to come from KCBS, and not the organizer.

The history of 'lists' is pretty mixed, and unfortunately a lot of it isn't good! I agree there is potential for good, but I think it's more than offset by the risks.

[TooSaucedToPork]

Correct. Whether 4 or 8 entries per table it matters not as long as you have 6 judges per table.

The average score of the judges will be the same. As everyone probably knows you add all the judges scores and divide by the number of entries judged.

Mathematically and Statistically...there are no differences that I have found.

[Jorge]

Steph Darlin,
I've thunk on this for a bit and don't believe that numerically this practice would be any different. As long as each table has 6 judges, your 4 box chicken entry tray has the same statiscal advantage as a tray of 6 entrys.
I had a contest last weekend that had extra judges and a 7th table
was set up to allow them a chance to judge. Not a single judge at my table even mentioned there were only 5 entrys per tray.
Ed

Statistically you are wrong. You can't add two tables, 12 judges, and not have a statistical impact on results unless everyone is giving identical scores. Whether you are mitigating or increasing risk/reward depends on the people sitting at the tables. It is statistically optimal to have the fewest tables possible, and not force so much food on the judges that they can't fairly evaluate brisket box #whatever.

[Jorge]

Correct. Whether 4 or 8 entries per table it matters not as long as you have 6 judges per table.

The average score of the judges will be the same. As everyone probably knows you add all the judges scores and divide by the number of entries judged.

Mathematically and Statistically...there are no differences that I have found.

I'll disagree. If we devolve into discussions of variance and deviation we'll have people either dozing off or looking for our location to give us Drain-o enemas.

The short version is that in a perfect system, every judge would taste every entry. It's not possible. The further we get from that, the more we introduce human error, bias, and a load of other factors.

[Q-Dat]

The more I learn about certified judging the more comfortable I am with taking my chances with the general public in IBCA contests. That certainly isn't perfect either, but it sure keeps the BBQ snob holier than thou types to a minimum.

[musicmanryann]

Statistically you are wrong. You can't add two tables, 12 judges, and not have a statistical impact on results unless everyone is giving identical scores. Whether you are mitigating or increasing risk/reward depends on the people sitting at the tables. It is statistically optimal to have the fewest tables possible, and not force so much food on the judges that they can't fairly evaluate brisket box #whatever.

Precisamente el jefe Jorge.

[billygbob]

Clearly the only solution for KCBS is for every cook team to provide a judge. That eliminates all the stooopid judges as every judge is an expert selected by an expert cook team. Reps and KCBScore would need modified to ensure every judge never gets their team's entry. The organizer provides one "celebrity" judges to make up for "rounding up" to full tables and ensure there are indeed some 999 scores. Now the cooks can just complain about how stooopid the other expert cooks are at picking judges. Problem solved; the cooks can just fight amoungst themselves. BBQ theater.

[Derussi]

I'm not necessarily in favor of holding judges directly accountable for their judging patterns, unless it is an extreme circumstance, i.e. a judge being the dropped score for every entry at a contest or not knowing how to write any other number than "9" for 3 contests straight.

Giving judges a slap on the wrist every time they aren't within a certain degree of the table average has potential to alienate a lot of people, which I don't think is good for BBQ. I do think it would be helpful if judges were given information regarding their scoring patterns for their own information and use. Just like cooks get a scoresheet at the end of the contest, judges could as well. This by itself could be a powerful teaching tool. Right now, the only feedback they get is the occasional chat across the table in between categories.

I would love to see how my scores related to others at the table. Rarely does another judge divulge their actual numeric scores. The usual discussion is "Which one did you like best / least? And if it scored low, why was it low. The Table Captain sees all of the scores and may comment that entry x had the highest scores. If a box scores low, they usually ask why it was scored so low.

Also, once a judge is seated and assigned, they cannot move between tables.

[Slamdunkpro]

We DO NOT address a judge who seems to be scoring too low or too high ( I've had to speak to a judge who gave ALL 9's ALL day) until AFTER all judging is complete. That way EVERYONE is on an even playing field throughout the contest.

Phillip

I must disagree, as I was a TC this past weekend and pointed out to the REPs that a judge was giving 5/6 when the other five judges were 8/9 and the REP spoke to the low judge immediately (after Chicken).
You don't want this happening. I table captained a contest last year where our table handed out a number of 6's and a few 5's during chicken (from the samples I got to try these marks were deserved). The rep came over after collecting and reviewing the score cards and gave the whole table the "Cooks work so hard, spend all this money, bla bla bla a 5 or a 6 kills their chances bla bla bla" lecture. The result? Our table became "Super table" - nothing but 9's and 8's the rest of the contest (and there were some substandard entries). This probably resulted in teams getting walks that shouldn't have.

[ModelMaker]

I'll disagree. If we devolve into discussions of variance and deviation we'll have people either dozing off or looking for our location to give us Drain-o enemas.

The short version is that in a perfect system, every judge would taste every entry. It's not possible. The further we get from that, the more we introduce human error, bias, and a load of other factors.


I guess my thoughts involved the 4 or 5 box trays not being mentally intrusive as the 7 or 8 box tray. However, numerically there should be no difference. 3 scores-6 judges. Math is math.
Ed

[TooSaucedToPork]

I thought about this for a while. My answer would be, no. I'd go further and suggest that contest Reps probably shouldn't have access to the database either. I don't think the odds of wrongdoing are high, but that data could be used pretty easily make things less than fair.

The other issue I have, and is more important to me, is that I don't want to see judges excluded based on scoring. That makes the system punitive. The system I'd like to see put in place would offer guidance and education opportunities to get a judges scores in line, whether they are higher or lower than the norm on a consistent basis. If there is a reason for them to be excluded it needs to come from KCBS, and not the organizer.

The history of 'lists' is pretty mixed, and unfortunately a lot of it isn't good! I agree there is potential for good, but I think it's more than offset by the risks.

If access will not be granted, could the current scoring program KCBS has be amended to assign a letter A, L, or H to a judges profile? Average, Low, High...

This list would ONLY be given to reps in order to balance tables .

[cpw]

If access will not be granted, could the current scoring program KCBS has be amended to assign a letter A, L, or H to a judges profile? Average, Low, High...

This list would ONLY be given to reps in order to balance tables .

I listened to The BBQ Central Show from the other week, he had Tim Grant from True Bud on there. Tim's suggestion about judging was basically the same as yours, but be more precise about it. You wouldn't have to re-train, retire, or kick out any judges because of their scores, you simply place the judges per table so that all of the tables have the same average judge score, based on the individual judge's average score. I don't think it would matter too much if the judge had only judged 2 contests or 30, their average is still their average. It just gets more 'average' over time.

[bbq.tom]

A major problem that I see is that not all judges are on the same sheet when it comes to their judging. I.e., some judges use the 7-8-9 method only and indicate a 7=below average or worse, 8=average, 9=above average or better. This is MAINLY because of the REPs and Organizers drumming it into their heads that the cooks spend so much time and money that we should give them credit. Also, so that they don't stand out as "low judges".
Other judges use the full scoring system 2-9. When the 7-8-9 judge gives a 7, it is the same in their mind as the 2-9 judge giving a 5. Therefore, while the 2-9 judge is actually scoring correctly and can justify their score, the 7-8-9 judge can also justify their score, but it definitely reflects differently on the KDBScore. Getting the 7-8-9 judges retrained to use the entire 2-9 range I see as the biggest problem.

Now, back to the OP original desire to hold judges accountable - once the "new" KCBS comment cards are printed and distributed, I truly believe that there will be MANY more comment cards turned in as the new cards make it MUCH easier for the judges to comment, The "mark the box" cards take away the requirement for the judge to come up with their own words to tell the cooks why they liked or disliked the entry, with a space for written comments as well.

I would still like to see KCBS go to the system where a judge MUST provide a comment on every entry. It works for other BBQ sanctioning organizatoins!

[mobow]

I can't tell that he scored our entry 8-12 pts lower than the other 5 judges at the table and that he was consistently the lowest score at his table all day by 3-5 pts??


What data am I missing??


I hear what you are saying and maybe even agree with you. But, you can not go by his average score as listed on that entry. You need to find his table and see what he scored at the individual tables. He may be the low average but he may have been the high score on three tables by a little and the low score on one table by a lot and his average would show up as the lowest. Keith

[Jorge]

If access will not be granted, could the current scoring program KCBS has be amended to assign a letter A, L, or H to a judges profile? Average, Low, High...

This list would ONLY be given to reps in order to balance tables .

If the software is going to be modified the secure solution is to have the software seat judges. It's a pretty simple sort routine, and would be faster than having the Reps do it themselves. Tables are balanced, data is kept secure, Reps have additional time to take care of other issues.

[TooSaucedToPork]

If the software is going to be modified the secure solution is to have the software seat judges. It's a pretty simple sort routine, and would be faster than having the Reps do it themselves. Tables are balanced, data is kept secure, Reps have additional time to take care of other issues.

Hey, I'm all for trying that...software seats by average score, it takes a little work off the reps, cooks have balance.

I see it as a win/win.

until it goofs hahaha...but I'd like to see how it plays out and works.

[Jorge]

until it goofs hahaha...but I'd like to see how it plays out and works.

That's no different than any other project, but I understand your point and don't disagree.

[Jorge]

I listened to The BBQ Central Show from the other week, he had Tim Grant from True Bud on there. Tim's suggestion about judging was basically the same as yours, but be more precise about it. You wouldn't have to re-train, retire, or kick out any judges because of their scores, you simply place the judges per table so that all of the tables have the same average judge score, based on the individual judge's average score. I don't think it would matter too much if the judge had only judged 2 contests or 30, their average is still their average. It just gets more 'average' over time.

I don't know of anyone that has the data to support that. There is anecdotal evidence that new judges tend to score lower, and then tend to fall in line with the norm over time. I tend to believe it but haven't seen objective data to prove it. If more average means a new judge scoring higher, and more in line with more experienced judges then I think you are probably right.

I think what we'll find out over time is that there are different groups of judges. There will be a group of experienced judges that will be pretty consistent. I think there will be a subset within that group referred to in this thread as 7-8-9 judges. I'd be shocked if we didn't have a smaller group that appears to be more erratic in the scores they give than the norm. The list of reasons is long!

This is just one of the reasons I'd like to see KCBS take some time and study a large sample before committing to a solution that could have unintended consequences.

[Slamdunkpro]

How about this? Change the scoring system to 1(dq) 8, 9 by 1/4 points (8.25, 8.5, 8.75, 9). Then everyone can get the 8's and 9's they know they deserve.:wacko::tongue::caked:

[bbq.tom]

I think what we'll find out over time is that there are different groups of judges. There will be a group of experienced judges that will be pretty consistent. I think there will be a subset within that group referred to in this thread as 7-8-9 judges. I'd be shocked if we didn't have a smaller group that appears to be more erratic in the scores they give than the norm. The list of reasons is long!


I agree that "new" judges score a bit lower than the "norm".

My experiences as a TC and judge lean toward the identification of a subset of "cook judges" as they usually score a bit lower than the "norm".

I'm sure that other subsets will manifest themselves as well.

[bignburlyman]

I agree that "new" judges score a bit lower than the "norm".

My experiences as a TC and judge lean toward the identification of a subset of "cook judges" as they usually score a bit lower than the "norm".

I'm sure that other subsets will manifest themselves as well.

At a invitational brisket championship I help with a "highly" respected, long time bbq cook who has published cookbooks, was the HARSHEST scoring judge in the room. He was 3 points under everyone else. The teams had all won a brisket category at a contest so there were few rookie cook teams involved. And this was even at a time when scoring started at 9.

[bignburlyman]

I agree that "new" judges score a bit lower than the "norm".

My experiences as a TC and judge lean toward the identification of a subset of "cook judges" as they usually score a bit lower than the "norm".

I'm sure that other subsets will manifest themselves as well.


Also, I remember a thread (can't find it tho), a few years ago where a cook from a team posted about his first judging experience and he was brutal in the scoring. If I remember right he had nothing good to say about any of the samples he judged.

[Jorge]

At a invitational brisket championship I help with a "highly" respected, long time bbq cook who has published cookbooks, was the HARSHEST scoring judge in the room. He was 3 points under everyone else. The teams had all won a brisket category at a contest so there were few rookie cook teams involved. And this was even at a time when scoring started at 9.

:mrgreen: Wonder who that could be?:mrgreen:

I have good memories of rendering on the tarmac in Great Bend and need to get back there when things resume!

[big matt]

Also, I remember a thread (can't find it tho), a few years ago where a cook from a team posted about his first judging experience and he was brutal in the scoring. If I remember right he had nothing good to say about any of the samples he judged.

That's happened out here as well..after a comp this year a father son team judged at one of our biggest comps..I've never seen this team walk even once in a few years so they aren't good at ALL..when they were walking by after turn ins were done I asked how the food was..I was expecting an answer like"our eyes were opened on what we need to do"..or "we learned something today"..instead I got "some was as good as ours the rest was crap"..WTF??..I also heard these two clowns had a talking to because they were dishing out 4's and 5's all day..this is exactly the kind of person(s) that should never even be in a tent..EVER!

[DUBBAGA]

That's happened out here as well..after a comp this year a father son team judged at one of our biggest comps..I've never seen this team walk even once in a few years so they aren't good at ALL..when they were walking by after turn ins were done I asked how the food was..I was expecting an answer like"our eyes were opened on what we need to do"..or "we learned something today"..instead I got "some was as good as ours the rest was crap"

Who are they... please tell me... I got to know!!!!!
http://images.zaazu.com/img/please-please-boy-sad-smiley-emoticon-000383-large.gif

[bignburlyman]

:mrgreen: Wonder who that could be?:mrgreen:

I have good memories of rendering on the tarmac in Great Bend and need to get back there when things resume!

According to one of Kent's sons the 2014 Winter Q is a go! I have not talked to Kent so I can't confirm that, we will see.

And in February the chance of you rendering out on the tarmac is remote. If we would have had the contest in 2013 there would have been 18 inches of snow to contend with, lol.