Judging Parity

QansasjayhawQ

QansasjayhawQ

is Blowin Smoke!
Joined
Jun 21, 2006
Messages
1,515
Reaction score
255
Points
0
Location
Lawrence, Kansas - The Great American Outback
I've been thinking about this and I would like to hear everyone else's thoughts too.

I do both - cook on a couple of teams AND judge. I started judging simply because I realized after a couple of events that I had NO IDEA what it was that we were shooting for (other than simply having some fun cooking BBQ).

As it turns out, I really enjoy judging and I've been doing it for a couple of seasons and I've enjoyed getting to know 'the regular' judges who travel far and wide to provide their services.

One thing that bothers me is that the scores from a "celebrity" judge (like a local radio personality or the mayor or the fire chief, etc.) are weighted just the same as a non-certified judge that are weighted just the same as a 'rookie' judge (say, less than 10 competitions judged) and their scores are weighted the same as 'experienced' judges and all of them have just as much say in their scores as Master judges.

This doesn't seem like a very fair set up.

Couldn't the KCBS do some things to help level this out? For example;

1) Track all scoring by all certified judges.
2) Weight judge's scores by some kind of 'experience factor' giving greater credence to experienced and Master judge's scores.

That way, the celebrity or non-certified judges who are tasting competition BBQ for the first time don't just give everything a 9 across the board - as they tend to do. And the tendency for rookie judges to compare entries against each other could also be accommodated using a system like this.

This would require more data entry time on site by the rep doing the data entry. Are there any other problems this might cause that I am missing?

Thank you for your thoughts on this.
 
Good point you're making, and this is a large factor in why many/most sanctioned
cookoffs try hard to get/recruit only certified judges.

Nothing is more frustrating to a team than having someone with mediocre product
outscore their tried-and-true better product because a judge or two has no clue
as to what they're doing (or score theirs down for the same reason).

This is a problem with non-santioned cookoffs and other sanctioned cookoffs where
they cant get trained judges (ala. chili).
 
I'd say an organizer that wants to get some celebrity judges should get six of them. Then set them at a table together in a corner and have some volunteers cook their food. The celebrity table wouldn't get used for the comp teams and they'd be non the wiser.

If the mayor of Williamstonburgsonville doesn't have time to take the class, he doesn't have time to judge. :biggrin:

Am I kidding - sorta but if an organizer did it I'd laugh with him
 
I have had some requests for these "celeb" types to judge and I tell them they have to take the class and they usually back out. They usually don't have tht two hours to give anyways.
 
KC_Bobby said:
I'd say an organizer that wants to get some celebrity judges should get six of them. Then set them at a table together in a corner and have some volunteers cook their food. The celebrity table wouldn't get used for the comp teams and they'd be non the wiser.

If the mayor of Williamstonburgsonville doesn't have time to take the class, he doesn't have time to judge. :biggrin:

Am I kidding - sorta but if an organizer did it I'd laugh with him
Click to expand...
This is the best idea I've heard yet. Seriously.

As far as "fairness", the entire field is subjected to the judging pool. Reps here try hard to spread less-experienced judges evenly between tables, so no table is too hard hit. But for sure, teams ask about percentage of CBJs, and are cognizant that if a class was held the night before...

The other judging system I've seen that works around this somewhat is that used by the American Rose Society. In that organization at that time, a newly-minted judge had an apprenticeship period. They would score shows in the company of an experienced judge, to make sure that their initial judging experiences were in line with the milieu in which they judged.

Whether it's through appenticeships or continued education, there are things to be said for doing what can be done to ensure that the greater body of available judges share a common and current worldview.
 
QansasjayhawQ said:
I've been thinking about this and I would like to hear everyone else's thoughts too.

I do both - cook on a couple of teams AND judge. I started judging simply because I realized after a couple of events that I had NO IDEA what it was that we were shooting for (other than simply having some fun cooking BBQ).

As it turns out, I really enjoy judging and I've been doing it for a couple of seasons and I've enjoyed getting to know 'the regular' judges who travel far and wide to provide their services.

One thing that bothers me is that the scores from a "celebrity" judge (like a local radio personality or the mayor or the fire chief, etc.) are weighted just the same as a non-certified judge that are weighted just the same as a 'rookie' judge (say, less than 10 competitions judged) and their scores are weighted the same as 'experienced' judges and all of them have just as much say in their scores as Master judges.

This doesn't seem like a very fair set up.

Couldn't the KCBS do some things to help level this out? For example;

1) Track all scoring by all certified judges.
2) Weight judge's scores by some kind of 'experience factor' giving greater credence to experienced and Master judge's scores.

That way, the celebrity or non-certified judges who are tasting competition BBQ for the first time don't just give everything a 9 across the board - as they tend to do. And the tendency for rookie judges to compare entries against each other could also be accommodated using a system like this.

This would require more data entry time on site by the rep doing the data entry. Are there any other problems this might cause that I am missing?

Thank you for your thoughts on this.
Click to expand...


I like your line of thought, but it would be very hard to calculate what the weights should be. It would take mountains of data and a real statistician to try and work it out. And it would only work if there really is a relationship between judging experience and scores. If on average the new judge always scores low then ok, but if it goes both ways, some new judges score high and some low then a weight won't work. The mean would be about the same as a experienced judge only there variance would be larger. Almost impossible to adjust for.
 
Avctually I think FBA does the best job at this. They track all their judges and categorize them according to their judging style. Those that are tough markers those that mark favorably and those who are middle of the road. When they go to seat these judges they do their best to offer each table a equal amount of each type so that each table is "fair". Of course it doesn't always work but it does do something to change that table with four friends who haven't given a 9 in their career or adversely that group of newbies, that gauge BBQ on Charlie Browns thursday night BBQ special, who "can't believe how good all of this food is. I gave all 9's because each one was better then the next." Its these judges I worry most about not the celebs.
 
I do like the tracking idea, you could weight specific judges (which is pretty analogous to what the FBA does by spreading them out).
 
goodsmokebbq said:
I do like the tracking idea, you could weight specific judges (which is pretty analogous to what the FBA does by spreading them out).
Click to expand...


The only problem I see with weighting each judge is that specific areas yield different flavor profiles. A judge could mark more favorably in one area then the next yet still be a tough judge. I think the mixing of judging styles would even out more then weighting. Even if they like the flavor profile they are still judging the same whereas weighting to a specific number may artificially bias the scoring.

Sorry I tried to explain myself. Does that make sense to anyone?
 
I really like the spreading out idea based on classification (high marker, average marker, etc.). By doing this you are in effect weighting the judges to create parity across the tables. You are using the judges classification to weight the other judges on the table.

I want to hit the table of all high markers!

Weighting each judge would be very ineffiecnt and very impratical, and it would fail if you don't truely understand a judges distribution of scores (like scoring higher in the south, etc). Any sort of weighting is biasing the data, you just need to make sure you are biasing in the right direction, towards the population average. Weighting can be very tricky business.
 
Do you see that with reps? There are some reps I like to go to contest they are at because the scoring is a little higher and more consistent then others.
 
Buster Dog BBQ said:
Do you see that with reps? There are some reps I like to go to contest they are at because the scoring is a little higher and more consistent then others.
Click to expand...

I can't think of a way that particular reps could affect scoring. Only through illegal ways and I won't make that claim at all. The only thing I can think of is the reps have a following of judges and these judges are either easier scorers or the percentage of CBJ's goes up so the caliber of judging goes up.
 
They can if and how they "charge the judges", like by stressing what a 9 means, what
an 8 means... Stressing. If not, even with certified judges, you end up with some
pretty bizarre variances. I do this for CASI (charge the judges) every time and stress
it. Pretty much the scores on the same chili rarely if ever vary by more than a point
or two...
 
Lake Dogs said:
They can if and how they "charge the judges", like by stressing what a 9 means, what
an 8 means... Stressing. If not, even with certified judges, you end up with some
pretty bizarre variances. I do this for CASI (charge the judges) every time and stress
it. Pretty much the scores on the same chili rarely if ever vary by more than a point
or two...
Click to expand...


Ah that makes a lot of sense. Thank you.
 
goodsmokebbq said:
I have had some requests for these "celeb" types to judge and I tell them they have to take the class and they usually back out. They usually don't have tht two hours to give anyways.
Click to expand...

Just because they're local celebs (usually VERY local) doesn't mean they're any busier than anyone else. If you want free bbq, at least take the time to get an idea what you're doing.

BTW, great logo!
 
#1 the KCBS has no control over selection of judges. The Jack is a good example. Good cooks still win with celebs. If we tell the organizer they need to have all CBJ's then the opranizer says I'll run it without you unless you provide the judges I need. Not going to happen.

#2 there are a lot of cbj's out there myself included that have not judged a contest in years but we're still cbj's. Doesn't make us better judges.

#3 I know a lot of cbj's that smoke as in cigars, etc. Now I would far prefer a non smoking non certified judge to taste my entry. Not cutting at smokers here as I enjoy cigars at contests but fdact is after a couple of cigars my taste buds are messed up. That's why I like having a program and sticking to it every time. I don't need to taste for the best slab of ribs.

#4 new cbj's that took the class the night before are often tough scorers. But others are not. there's no formula to apply here
 
I think it's a mistake placing so much importance on the judges and sometimes it seems, from what I read about it, self importance. IMO If you cook the best BBQ you will still win. It's been shown over and over again with cookers that cook nationwide and cross sanction. IMO CBJs are not necessary nor do we need experienced "professional" judges making the tail wag the dog at competitions. I believe the CBJ and their training is starting to shape comp BBQ turn in's and to me that's wrong. This endeavor we love so much should be about the cooks not the judges
 
I judged my first comp this last Fall and the reps asked the judges if there were any newbies and made sure that we were not all sitting at the same table and had an experienced judge sitting next to us in case we needed assistance. I thought that was really good and fair.
 
My feeling is that the difference between a judge who took a class and one that hasn't is not significant. The criteria to be a BBQ judge is ridiculous. One day where you are taught how to fill out a form and told that ribs shouldn't fall off the bone - that's hardly training. To become a Certified Beer Judge you have to take a series of classes and then pass a difficult test. Plenty of people are willing to go through the hassle of getting certified. And the reality is that it costs less than $100 to brew a beer that will be judged. It can cost several hundreds to participate in a BBQ contest. Doesn't it make sense that people who spend that kind of money have judges who have REAL training?

BYW, I'm judge # (oh well, I forgot)
 
You must log in or register to reply here.

Similar threads

J
An idea whose time has come.....
Replies
3
Views
382
bschoen
B
L
Sca charlotte
Replies
2
Views
677
LanceM88
L
F
Reverse Flow Smoker Question
Replies
2
Views
271
seattlepitboss
S
FloridaBBQGuy
New to Florida and the beach but not to BBQ
Replies
8
Views
2K
oldgfbbq
O
J
Sanctioned Competition vs Local "Rib Burn" Flavor Profiles
Replies
9
Views
860
Jlems
J
Back
Top