Judge's Score Sheet

[Candy Sue]

On the Team of the Year thread, some mentioned that the board should focus on judging.

The board made an important first step to get information on how the judge judged to the judge. This is pulled from the November meeting quick notes:

... suggested that instead of a full info sheet on judges scores, an average is presented of the judge's average scores compared to table average. It protects the double blind in that no actual scores are given. This would need to be done for each category.

Judge would login to kcbs.us and go to judging information. A list of contests judged would appear. Select the contest desired. Four categories appear with CBJ average and table average. For example:
Category CBJ average Table Average
CK 33.44 30.25
RB 31.22 34.56
PK 32.85 35.85
BR 34.99 35.85
Proposed Motion: I move that KCBS implement a Judges scoring report which indicates the CBJ's average score and the average score of the table where judging was done. 2ND RICHTER. ROLL CALL: AYE: WEAVER, POLSON, HAYS. NAY: BIGLER, COLLIER, CAPSTACK, COMPTON, RICHTER, FULTON, SIMMONS, LOHMAN. MOTION FAILED.

DISCUSSSION REGARDING WHETHER THIS IS ENOUGH INFORMATION FOR JUDGES. PROPOSE AMENDING REPORT TO SHOW JUDGE AVERAGE FOR APPEARANCE, TASTE AND TENDERNESS FOR EACH CATEGORY. WEAVER: AMENDED MOTION TO SHOW GREATER DETAIL IN JUDGE REPORT. 2ND BIGLER

Category CBJ average Table Average
CK 33.44 30.25
APPEARANCE 9 8.75
TASTE 8.5 8
TENDERNESS 8.13 7.95

RB 31.22 34.56
APPEARANCE 9 8.75
TASTE 8.5 8
TENDERNESS 8.13 7.95

PK 32.85 35.85
APPEARANCE 9 8.75
TASTE 8.5 8
TENDERNESS 8.13 7.95

BR 34.99 35.85
APPEARANCE 9 8.75
TASTE 8.5 8
TENDERNESS 8.13 7.95

ROLL CALL VOTE: AYE: COLLIER, WEAVER, CAPSTACK, COMPTON, POLSON, RICHTER, FULTON, SIMMONS, LOHMAN, BIGLER. NAY: HAYS. PASSED.

 

[DUBBAGA]

Candy, I did read this, and believe it is a solid step.

For the purposes of tracking judges, and to provide the most accurate averages, would this system be able to remove any categories where "1" was utilized?

Scoring a "1" means that there is a disqualification, and it was authorized/directed by a Rep. The applicable "low score" should not be taken into account when tracking judges, or comparing them to the table, as it would skew the results.

 

[Roast Beast]

That's a good first step. Would be great to see the Board take the next logical step of using the historical average information to help balance tables when judges are seated. It might not be perfect, but I bet it would do a better job of smoothing out TODs and TOAs than the current (various and inconsistent) methods used to seat judges currently. :thumb:

 

[DR]

i have said this from day 1. WE NEED ACCOUNTABILITY WITH OUR JUDGES!!!!

 

[Okie Sawbones]

I would like to see judges 'scored' so to speak, tracking them as to performance. They did this when I was a certified wine judge, and those judges who were always counter to the table were not asked back.

The trouble is that not all BBQ societies use certified judges. KCBS and the PNWBA does, and I've judged with both of them. The IBCA, which is big in Texas, does not certify judges. I don't understand that.

 

[Jorge]

I would like to see judges 'scored' so to speak, tracking them as to performance. They did this when I was a certified wine judge, and those judges who were always counter to the table were not asked back.

The trouble is that not all BBQ societies use certified judges. KCBS and the PNWBA does, and I've judged with both of them. The IBCA, which is big in Texas, does not certify judges. I don't understand that.

The cream still seems to consistently rise to the top in IBCA, without certified judges.

 

[CBQ]

The cream still seems to consistently rise to the top in IBCA, without certified judges.

The cream rising to the top in KCBS too. Good teams win more consistently, but even good teams have been burned by the TOD and lifted by the TOA. Even though it all averages out, KCBS should still strive to reduce that variance, especially since it now has the data to do so.

I'm not advocated banning judges as cited in the wine example, but certainly seating judges based on tendency has some merits. New judges would be an exception, and excluding "1"s from a judge's historical scoring would also have to be done.

 

[egorham]

I don't know how to hold judges accountable for something this subjective. There are many reasons that a judges scores could be different from others at a table that have nothing to do with the quality of the judges. We know from looking at the results that there is a regional bias in judging, just look at the results and see how the GC scores within a region stay in a tight band but vary widely compared to other regions. The regional bias alone could explain why in a large comp, with more judges that travel, you would see a high variance in scores. There are also taste preferences that vary widely.

I have used this analogy before, at the beginning of every broadcast of a the major league baseball games, at least in my area, the commentator talks about how that days umpire has a "high strike zone", "narrow strike zone", "low strike zone", etc. If an organization with unlimited resources such as MLB can't get 150 or so paid umpires to call a strike zone, that could not be more clearly defined, the same way how can we expect KCBS to get thousands of volunteer judges to do it with something as subjective as BBQ.

I think we need to accept that there is a large variance in judging, there is no way to educate our way out of this and find other solutions. I like the idea of using some type of algorithm to assign judges to tables to help balance this out, I don't think it would be difficult to accomplish and could improve the outcomes.

 

[Smoke'n Ice]

Candy, the way I read this Motion is it is a place for JUDGES to go to see how they faired. NOTHING else. Is the intent to get them to be more average? Is this really what the cooks want?

I personally like the idea that a judge who gets a bad sample, so judges its and the hell with average.

As far as breakout of the categories, it is WORTHLESS when seating the judges unless you plan on a shuffle between each category.

What should be done with this information is KCBS, the reps or the organizer should balance tables based on the overall average of each judge and implemented ASAP.

 

[Okie Sawbones]

If judges are judging by taste preferences, they have no business judging IMO. If they are not familiar with regional differences, they need to get educated or not judge out of their area. I don't subscribe to the 'it's too subjective' mantra.

The trouble with BBQ judging classes in the USA is there are no tests. You take the class, raise your hand, and you are certified. Not good enough. When I took the wine certification course, we had a half day test that we had to pass. A large portion of the people failed. We had to identify grape varieties, wine flaws, acidity in order, sweetness in order, etc. Something similar needs to be done for BBQ judges IMO. Judges need to be able to identify flaws like fire management problems, undercooked meat, overcooked meat, etc.; they need to be aware that what is taboo in one BBQ association is just fine in another BBQ association.

 

[Gowan]

Judges need to be able to identify flaws like fire management problems, undercooked meat, overcooked meat, etc.;.

I strenuously disagree.

Everything you mentioned there would require the judge to make guesses about HOW the sample was prepared. This is absolutely not the role of a KCBS judge. The CBJ is tasked with evaluating the sample before them, not the process that got it there. In fact it is when judges try to do this that cooks become most agitated - comment cards that attempt to diagnose problems and tell the cooks how to correct them almost always are received with anger and derision.

Useful feedback in the above cases would be on the order of "bitter creosote taste", "chewy - no clean bite", "tough and dry", etc.

 

[Okie Sawbones]

I strenuously disagree.

Everything you mentioned there would require the judge to make guesses about HOW the sample was prepared. This is absolutely not the role of a KCBS judge. The CBJ is tasked with evaluating the sample before them, not the process that got it there. In fact it is when judges try to do this that cooks become most agitated - comment cards that attempt to diagnose problems and tell the cooks how to correct them almost always are received with anger and derision.

Useful feedback in the above cases would be on the order of "bitter creosote taste", "chewy - no clean bite", "tough and dry", etc.

I disagree. Every one of these problems is not a guess. It is a simple fact if the judge has a palate at all. It is indeed the role of a judge to identify flaws. To think otherwise makes no sense at all. If you are offended at negative feedback, perhaps you need to look at your entries. There is nothing wrong with a judge telling a cook that their entry was tough, or overcooked, or way out of the mainstream. The cooks know it themselves, but hope to slip it by an inexperienced judge. I never tell a cook what to do, I just state the reason I marked down their entry. They shouldn't get angry, they should learn and make corrective measures if they are honest with themselves. I am pro-cook. I give every possible consideration to a team, but I won't dismiss crap, and believe me, I have tasted stuff I'd be ashamed to feed my dogs. Too many cooks deride judges as if they are the enemy. This could not be farther from the truth. But as I have seen over the years, the worse teams are the ones crying the loudest about judges. It goes with the job. I don't let it bother me. I have a lot of friends on the circuit, and they know I try to give an honest opinion. They also know that I am a good cook myself (blowing my own horn), which I think is important as a judge.

 

[Gowan]

I disagree. Every one of these problems is not a guess.

It absolutely is a guess when you are presented with a blind entry.

How many times have we seen cooks complaining about Wrong Guesses?

"Tastes like lighter fluid" - (none used)

"Too much Blues Hog" - (none used)

"Alcohol flavor" - (none used)

"Undercooked" - (overcooked)

It is irrelevant to the judge why a sample is the way it is. Their job is to report on their experience tasting it.

It is up to the cook to decide if that feedback is relevant and if they want to take any action in the future because of it. Boiled down to the essence of the thing: CBJs are judges, not instructors.

 

[Okie Sawbones]

Nonsense. Just as I can tell a corked wine with 99% accuracy (seen the printout). I can tell a flaw in BBQ. If a judge cannot tell undercooked from overcooked, they should not be judging. This is a simple, basic judgment. It is not irrelevant to a judge what a sample is -- it is their job to evaluate it and provide feedback if they so desire. I only provide feedback when an entry is really bad, hoping the cook will take something to heart. Sometimes their ego gets in the way, which is their loss not mine. In all of the years I have judged, I have never said 'too much alcohol' or 'lighter fluid'. I have tasted some entries that taste light Matchlight charcoal, because I mistakenly used it once. But I don't know if the entrant used that, so I refrain from commenting in that regard. Same with Blues Hog. How would I know if they used that vs. their own concoction that mimics it? The comment card should simply say 'overpowered with sauce and I could not taste the meat'. There are a lot of those, trying to hide substandard entries. Next argument?

 

[Gowan]

How would I know if they used that vs. their own concoction that mimics it? The comment card should simply say 'overpowered with sauce and I could not taste the meat'.

Good. I'm glad you agree.

 

[sitnfat]

Judging wine and BBQ are two different things

 

[Pappy Q]

Judging wine and BBQ are two different things

This coming from a professional wine maker and BBQ cook....can't get anymore of an expert opinion than that.

 

[Okie Sawbones]

Judging wine and BBQ are two different things

They both require a trained palate that can identify flaws. They both require a trained palate that can distinguish a typical entry, vs. some atypical entry. They both require evaluating the body of the entry. They both require evaluating the appearance. They both require over all taste evaluation. You evaluate aroma in both. There are a LOT of similarities.

 

[Serious Hog]

If judges are judging by taste preferences, they have no business judging IMO. If they are not familiar with regional differences, they need to get educated or not judge out of their area.

Can you elaborate on this? Are you saying judges should score taste based on regional conventions rather than just whether they think something tastes good or not? If so, I totally disagree. Apologies if I misinterpreted.

 

[Okie Sawbones]

Can you elaborate on this? Are you saying judges should score taste based on regional conventions rather than just whether they think something tastes good or not? If so, I totally disagree. Apologies if I misinterpreted.

No need for any apologies. I run my mouth recklessly. :heh: It is hard to have a real conversation on the Internet. What I'm trying to say (as an example) is if a judge happens to be judging in Alabama, and they have no clue what a white sauce is, how can they fairly judge an chicken entry with white sauce vs. a standard entry? If all a judge has ever eaten is Memphis-style BBQ, how will they react to slap in your face vinegar BBQ from North Carolina? And what about a judge who has only eaten a savory brisket, when they are presented with a S&P Texas brisket?

I have been fortunate to travel a lot, 39 states and three territories, and 17 countries. I have a good feel for regional differences in BBQ except the far NE USA. They don't allow talking at the judging table, so a judge cannot ask a contest organizer, "I've never tasted a white sauce before. It this typical?"

I don't know if this clarifies anything, but I tried. :grin: