Question "Split" from the "Judges Tent"
 

I split this off.
Great basic thread and I am learning a lot!

But, here is a more detailed and specific question for our Judges and Experienced Competetors.

Please bear with me- this is complicated.

Background-
I was a Ladies Gymnastic Judge (USGF) in the 80's for 3 years.
Sounds like a similar system for training and certifying Judges.
Classes, video, study, pracitce, and a "killer" written test.
Start at the lower levels and, with "experience, a track record, and more testing", move up in the classes and authority.
The exact details are not important- the concept is. There are rules, training, and evaluation before you become a "Judge".

We (the USGF and FIA) had a final scoring system based on the judges being "In Range".

Basically, all judges must be "in range" with each other and the "head judge". The range depended on the score. For example, for "head judge" score of 9.5 to 10- we had to be within .1 or a "conference" was called.
This acceptable range went up to .5 points at a 7 score from the head judge (or something like that). The numbers are not important.

If a conference was required, the judges discussed (in private but face to face) just what they had seen to generate the score.
Normally a matter of "viewing angle" or rules interpretation.
I will save you the details.

The bottom line was--if the Judges observed or scored things outside of an acceptable range--It had to be resolved before the score for the girl was posted.

Lots of subjectivity--seems just like judging good Q

But, I learned that Judges almost always (but they are not perfect) tried to do the right thing, within the rules, to render a valid and accurate score!
Human evaluation is subjective by nature and can never be reduced to totally objective terms.
If so- hire a computer!

I have seen remarks (in some BBQ rules) about throwing out the lowest scores to resolve ties, etc. I saw some comments in the origonal thread about spices, texture, presentation, etc. that raised my curiosity!
Other than that, I have no practical experience. Help me out here.

So- finally- here is/are the questions for the guys who have been there!

From meet to meet, are the judges normally in agreement on scores?
How about within a meet if they are way off- is anything done?
If a judge is "way low" or "way high", is the score resolved?
If a judge seems to "favor" or "dis-favor"some item, i.e. presentaion or spice leve, is action taken on the spot to equalize or normalise the scoring.

I am not advocating adoptance of rule changes in any way shape or form!
I am just curious about how the current system really works.

DF and I am entering that world as comtestants and appreciate any feedback on how the system works.

Thanks,
TIM

Just wondering :oops:

Never judged official Tim, but I am certified.

So I'll try to answer from a "novice" perspectiive.

the answer: "no"

Judging in KCBS is an individual thing, with no interaction with others. Its not allowed. You have 6 entries in front of you, you bite, you think, you judge. Talking is not permitted until after the round (is this right Brooklyn)

Way low or Way high, that is the luck of the draw. You (cook) use a salty rub. Judge A likes satly, so scores an 8, judge B, dislikes salt and scores a 4 (on overall taste).

Judgeing, to me (certified but virgin) seems to be the luck of the draw

only thing i can comment here is that after evaluating the details of the scores from the CT competition, i learned that there is no normalization. My brisket scores ranged from a 4-6-6 to an 9-9-8. It came in second, but one judge rated it very low, while the other scores were 6-8-8, 7-9-8, 7-8-9, 8-9-9. the ranges were pretty extreme.. Same on the ribs. Seesm its every judge for themselves. They are also not allowed to discuss the entries during judging.

Also, if I understood an explanation properly. All entries start at a 5 which is average and judges rate up or down. This is a new way of doing it and alot of judges still do it the old way where everything starts as a 9 and they rate downwards from there.

I agree with Phil. In our experience, our scores varied widley as well. One of the other competitors, with tons more experience and more than a few Grand Championships under his belt, said that with the old system if the scores were 5 and under, or something to that effect, the low scoring judge had to justify his score. In the current system there is no explanation required for low or widely varying scores.

That's how I remember it. maybe horn or bigmofo can correct me or add their recolections.

There is no talking allowed when you are judging. No hand gestures, no face gestures and almost no eye contact. We are allowed to discuss only AFTER the score cards go in. In CT, we did not discuss scores, we did talk about which we thought best, worst or odd.

We were told in CT to start at 6. "6 being average, 9 better than mom's apple pie and 2 being rot gut" were the exact words used. The old scoring system started at 9 and worked down.

I know that i gave one entry a 6,3,3 and no one questioned me about it. It is subjective and somewhat the luck of the draw.

The correct instruction to the judges is, "9 is excellent, 5 is average and 2 is just bad, we will start at 6 and judge up or down based on how you, the judge, feel about the entry". A 1 or DQ can only be given by the KCBS Rep and they will give you the correct way to score the entry.

This being the first season under the new guidelines it is not surprising that the scores are all over the board. It use to be that a total score of mid 700s would be needed to win a lot of competitions, now a score in the 630 to 650 range can get the job done.
The old system you would start at nine and take a point off if you found a problem with the entry, the range was 9-8-7 this leads to a large number of ties and that's the reason for the change to the new starting point and instructions.

According to this month's Bullsheet - (August 2004 -page 10):
"Contest reps have been instructed to repeat the following after the CD meeting is completed: 'The entry starts as a 6, and is judged up or down according to the quality of the sample'." It also says that "5 is considered average."

One thing I didn't know that was also in the article: "Cooks are welcome to listen to the Judge's meeting and Table Captain's meeting." That's something I didn't see any of the cooks take advantage of. I would if possible.

It was good that Greg and I started the judge thing this year.

I'm not sure if other judges/competitors (like Jim M) like the new system, but since it is the only one Greg and I know, it seems fair. Start at a mid point, and award better, and detract from lesser to score, rather than everything be 9 and detract only the bad points. This new system seems to award a superior product, where as if you started at 9 with a good product, but couldnt find alot wrong with it, you would end up with an 8. In the new system, a good product would be a 6, maybe a 7, but the "great" product would be an 8 or 9 being excellent.

So I like the new system just from a theoretical standpoint, even though I only have applied it in a "mock" situation at class.

Just as info, Greg and I were instructed by Mike and Theresa Lake who run the Illinois Championship (and something in Iowa) and the Butt to Butt competition you read in this months BullSheet.

Second sidebar: any Brethren who are or are Not considering competing, I would strongly suggest taking a judging class.

Not only can you see how its done if you chose to compete to see what they are looking for, you can also be a judge for future contests, or even just play around at home and rate your own Q's. Many a time have I found myself deducting points on a butt for tenderness or awarding myself for taste. Only thing I skip is appearance, since I eat it right off the grate :)

Having judged with the "old" system (start at 9 and go down) and the "new" system ( start at 6) I have to say that I liked the old system better. It was easy to look/taste a piece of meat and say to yourself ,"the meat looks burnt, thats a point off." It is much harder to label something as "average" and then decide how much above or below average it is. I also think this system makes you want to compare the entrys more, and that is a big no no.

I do understand the need for a system that has less ties in the final scores, and this might be the best way to do it.

From what I have heard around the judges tent after competitions, this new system still gives a fairly accurate final score among all entrys.

With any system there is bound to be some fluctuation in scores from one judge to another. My understanding is that if one judge is continualy scoring low on ALL entrys at more than one contest, he or she may be asked not to judge, or asked to explain thier judging method.

case in point: I had one judge at my table that didn't like coconut. When the entrys for the deserts (not an official KCBS catagory) came in, one persons entry was custard served in coconut shells. This guy said he marked lower scores just beacuse he didn't like coconut!

There are two Jim Ms, McGrath and Minion. I think the new system will be much better in the long run. Right now a lot of more experienced judges are going through a relearning process.

On the whole, teams that did well under the old system continue to do well. Average and medocore teams used to get clumped together. Now they are more spread out.

Jim

Oops
Both of you have some great knowledge though.

Thanks Jim McG!

Well, I "asked" and so now "I learn".
And, everything I learn helps.
Will put it to good use and make the Brothers proud!

Thanks to all.
TIM

not sure if I like the new system---yet! It is really hard to tell what ya are doing when the scores are all over the board.. difficult to get a handle on things when you get thre 6's and three9's for appearance on the same entry. Is it 3 old judges and 3 new judges dod the different folk reall see it that much differently??

all that said will cook again this weekend

Buzz
2fat Bikers

As a newbie, I think the new system makes more sense. Starting at 9 and looking for ways to knock down the score seems a lot harder than starting at 6 and moving either up or down depending on the entry.

I don't know if this helps Buzz, but when we compared scores in the judge's class people were pretty consistant. What I mean by that is, let's say there were 10 judges in the class, the instructor would ask for a show of hands about the scoreof an entry. How many of you gave it a 6 - let's say- and 5 hands would go up, 5 - one hand, 7 - two hands, 8 - one hand and maybe one hand for 9 or 4. The majority of the group seemed to always be predominately one number. So it seems like the new system SHOULD work just fine.

I guessit works in the sense that the smae good cooks are still winning and there are very few(maybe one in the four meats) 180's this year.. Folks were po'd after the contest in Nebraska last year!!

Last year you never would have seen three 6' and 3 nine's given to the same box--but thats the way it is--today

Buzz