Is the KCBS BOD schizophrenic?

If a CBJ consistently scores well outside the norm of his tables
Maybe "honest" is not what he is

I am not a competitor. I took a CBJ class recently and the person who was supposed to cook for the class had a last minute personal emergency and replacement cooks were recruited from throughout the community. The replacements did not have to be competition cooks. Every piece of meat that I tasted that day, I would have sent back to the kitchen had it come from a restaurant. It was the worst BBQ I had tasted in my life. The class leader acknlowledged the problem and said that it would be valuable to us to know what the bad stuff really is. The scores on each sample were in the full range just at my table. There were 9's and 2's.

If there is a scoring problem, it is in the training of the new judges because there are more than 70 CBJ's that came out of my class that have not yet seen, felt or tasted a 7, 8 or 9. Further, from the comments above, some judges have their own "sub criteria" for scoring that are not represented in the KCBS rules.

Thom, I judged today, and there were swings in the scoring. 4 of us got sauced fat in the brisket box, 2 got nice crispy chunks. One rib box had 5 very nice ribs, yet the judge sitting beside me got a rib with barely enough meat to taste. His score was significantly lower than the rest of ours. Theoretically, with the tracking that should have started today, if he scores anything lower than the rest of the table within the next 12 months, he will be subject to evaluation. Should this judge be penalized for getting the one bad piece of chicken in a box on top of a rib with no meat?

SaucyWench said:

Thom, I judged today, and there were swings in the scoring. 4 of us got sauced fat in the brisket box, 2 got nice crispy chunks. One rib box had 5 very nice ribs, yet the judge sitting beside me got a rib with barely enough meat to taste. His score was significantly lower than the rest of ours. Theoretically, with the tracking that should have started today, if he scores anything lower than the rest of the table within the next 12 months, he will be subject to evaluation. Should this judge be penalized for getting the one bad piece of chicken in a box on top of a rib with no meat?

Click to expand...

And that happens a lot more than you know. If a judge gets a really bad rib compared to the others, why are they out of the norm? They are talked to if they are out of the same realm of the other judges, but if they have a legitamate reason, so there lies a true score. We are not out there to sway judges opinions, just making sure they understand our procedures.

I see judge number ident theft comming ? :crazy:

I judged the same contest as SaucyWench. I too had one rib that was not fit to feed to a dog. Couldn't have cut it off the bone with a razor blade. They got a low score compared to the rest of the box. The ribs in the box obviously came from several different slabs as the other judges at my table didn't have a like rib. The others were also a different color.
Sure not my inconsistancy as a judge that caused that. :hand:

Judges will not be penalized
Education is not a penalty

The Rib example given would be noted by the Rep
If this same CBJ were given a bad entry at another contest
it would be noted once more
There are times when one judge will get a substandard piece
and perhaps it will be discovered that 2 times outside table norm
is too frequent the action step

The problem I see with computerized judge tracking (which by the way, failed this week due to programing issues) is, how do we prove that a judge has been asked why a score, be it low or high in comparison with others at the table, is not within the norm? If a rep says it's a valid reason, will that override the tracking program? Since computers cannot know our reasons for scoring as we do, I fully expect to get a talking to next year, because I will not give a 6, 7 or 8 to something that deserves a 5 or less.

SaucyWench said:

The problem I see with computerized judge tracking (which by the way, failed this week due to programing issues) is, how do we prove that a judge has been asked why a score, be it low or high in comparison with others at the table, is not within the norm? If a rep says it's a valid reason, will that override the tracking program? Since computers cannot know our reasons for scoring as we do, I fully expect to get a talking to next year, because I will not give a 6, 7 or 8 to something that deserves a 5 or less.

Click to expand...

I would hope the program would have a way to note a Reps comments. I dont know that for sure but I am taking a wild guess here...

At a contest this weekend the following scores on ribs were posted for the top 3 finishers:

1 34.8572 23.4286 30.2856 36.0000 35.4286 34.8572
2 34.8572 24.0000 33.7142 30.2856 33.7142 36.0000
3 35.8572 25.1428 35.4286 32.0000 29.7144 32.0000

I have looked at all of the results and suspect I can pick out the additional entries judged by these judges based on the score pattern. The question is, "Which Judge Should Be Re-educated -- Judge 1 for consistently having no variation or judge 2 for consistently being well below the rest of the table or did judge 2 perchance get the one bad rib in each of the boxes he judged? It might be that he needs new glasses

Implementing the nuts and bolts of this will take time
Yes there are more questions that need to be answered here
I can only comment on the concept
Maybe we can get a current BoD member to address these points

Thanks for taking the time

BoD action to remove a CBJ is at the last resort I would hope

Smoke'n Ice said:

At a contest this weekend the following scores on ribs were posted for the top 3 finishers:

1 34.8572 23.4286 30.2856 36.0000 35.4286 34.8572
2 34.8572 24.0000 33.7142 30.2856 33.7142 36.0000
3 35.8572 25.1428 35.4286 32.0000 29.7144 32.0000

I have looked at all of the results and suspect I can pick out the additional entries judged by these judges based on the score pattern. The question is, "Which Judge Should Be Re-educated -- Judge 1 for consistently having no variation or judge 2 for consistently being well below the rest of the table or did judge 2 perchance get the one bad rib in each of the boxes he judged? It might be that he needs new glasses

Click to expand...

There's that dang judge #2 again!!! :-D

Seriously though, there is little chance that the top 3 were at the same table, and according to the "2 points +/- from the mean" plan, judge #2 is not the only one out of line-some of the high scorers are too.

SaucyWench said:

Seriously though, there is little chance that the top 3 were at the same table

Click to expand...

I would not be willing to bet on this:tape:

Having been a judge (CBJ) for 6 years at 70+- comps, and a competitor for 3 years
or so, and having experience with other sanctioned cooking events (chili for example),
I think I've seen it all a time or two.

Scoring systems with a wide range (1-9, 1-10, etc) allow the possibility of a wide
numerical variance (mathematics for a deep pile of feces if not monitored and
governed closely). Numerical variance, it and of itself, isn't a bad thing, it allows for
finer delineation between one that's pretty good and another that's perhaps very
good. It also allows (as described above) the variance of score in a piece of bbq
that one judge gets vs. a piece another judge gets.

HOWEVER, it also allows everything else. These "everything else" issues come to
light more clearly when watching judges scoring on items that are in the same pot
(ie not a different piece of meat, but part of the same), like in chili judging. I've
seen it all. Even with really good instruction, some judges still think (and I've heard
this said more than one time, directly out of the mouth of judges) "if they're in
a competition they must be great [or trying to prove something]", so their average
score is *average*-great; resulting in a truly good product ending up with very low
scores. It's like they're judging this entry in front of them against every entry
they've ever judged or eaten at a restaurant or made at home. Out of the same
bowls of chili, I've literally seen 1's to 9's. With better guidance I've seen 4-9's.
I'm not saying which is the correct score (the 4's, or the 9's). I am saying that the
product either sucked or it was pretty darned good, but both? No.

There is a very good argument that KCBS, by throwing out the lowest score, takes
care of this. It's a good argument, but honestly perhaps the lowest score was the
more correct score. I dont know. I'll also tell you that IF it were the result of a
rogue, uneducated, whatever judge, there are many tables that have 2 or 3 of
such folks (entry variance [as described wonderfully above] not withstanding).

Removing/reducing numerical variance you've virtually eliminated the problem. To
the folks against it, I highly suggest working with system with a very small numerical
variance for a while and see first hand whether they work or not. I'll tell you, as
a MIM/MBN judge, I cannot think of a time where the best BBQ didnt win. Not one
single time. I ask that you ask the teams and judges who either compete or judge
both systems and ask for their opinions rather than making assumptions about the
other without any experience.

By the way, I'm not anti-KCBS. I found out yesterday, for example, that I've pretty
much convinced an organizer to change her competition from an MBN format to a
KCBS format. I love judging and competing in both. It doesn't mean that any of them
are perfect or without opportunity for improvement.

Skinny Cook said:

I see judge number ident theft comming ? :crazy:

I judged the same contest as SaucyWench. I too had one rib that was not fit to feed to a dog. Couldn't have cut it off the bone with a razor blade. They got a low score compared to the rest of the box. The ribs in the box obviously came from several different slabs as the other judges at my table didn't have a like rib. The others were also a different color.
Sure not my inconsistancy as a judge that caused that. :hand:

Click to expand...

Just for clairification, did you judge down because it was tough or did you judge down because it was a different color or from a different slab?

Just a thought from little ole me: How often are the 2-3-4-5 scores used, and do they really make much of a difference among them? From what I've seen, scores start out at about 7 for all categories and can go up one or two places for good and great. The can also go down for "not good" and "Raw / innedible." Does it really matter whether you get a 2 or a 5? Both of them are lousy, with the 1 of course reserved for DQ. I for one would appreciate removal of any scores above about five total values plus special values (DQ and raw?). That's just me though.

This reminds me of one of my pet peaves about grading, but it's very similar: In college there is no realistic difference between a D and an F. Sure a D is passing and an F not, but D's don't count toward a degree or qualify as a pre-req, so why even give the grade? I don't think a 1.0 GPA even qualifies for graduation at most schools. Some schools don't even allow you to graduate with anything below a 3.0. Why stick with labels like "average" when average isn't good enough to graduate? The grades should be relabeld and D dropped totally. Sure this is partially a rant, but it's also related: Extra grades or scores can really confuse and complicate matters, and I don't think "because they've always been there" is a good enough reason to keep them.

dmp

I was a table captain this weekend. We had a rib entry that scored this;
9-7-7
9-7-7
9-9-9
9-9-9
8-8-8
8-8-8
Do you think that is to much variance between the judges?

By the way, there were 12 ribs in the box so I did get to try one. Obviously, with that many ribs they did not come from the same slab. The one I had would have been 9-9-9 if I had been judging, but apparently I got one of the good ones...

QN said:

I was a table captain this weekend. We had a rib entry that scored this;
9-7-7
9-7-7
9-9-9
9-9-9
8-8-8
8-8-8
Do you think that is to much variance between the judges?
By the way, there were 12 ribs in the box so I did get to try one. Obviously, with that many ribs they did not come from the same slab. The one I had would have been 9-9-9 if I had been judging, but apparently I got one of the good ones...

Click to expand...

No, pretty much dead on. They all pretty much agree that
the appearance was appealing and appetizing (perhaps more
appealing to a few, which is normal and expected). They had
a slight variance in tenderness and taste, that is easily explained
by different ribs on different slabs, different cuts, but also allows
for difference taste preferences. This is how it should look. It's
when you get (see below) that needs work:

977
678
999
688
989
788