The BBQ BRETHREN FORUMS.


Forum Portal Recipes Smoke Signals Magazine Welocme Merchandise Associations Purchase Subscription Brethren Banners
Go Back   The BBQ BRETHREN FORUMS. > Discussion Area > Competition BBQ > For the Board

Notices

For the Board *On Topic Only* Strictly moderated. NO BAD KARMA! This forum is for questions and discussions you would like reviewed by members of the KCBS(or other BBQ orgs) Board of Directors. A clean and direct place where they do not have to wade thru day to day chatter.


Reply
Thread Tools
Unread 05-25-2013, 01:55 PM   #1
accuseal
On the road to being a farker

 
accuseal's Avatar
 
Join Date: 08-17-11
Location: Strongsville, Ohio
Downloads: 0
Uploads: 0
Default Calibration of Judges

I'm a bit new at this but I am a CBJ and have cooked my first contest. It seems to me that judges could be calibrated by taking their scores and after a few contests, giving them a report on how they stack up against the rest of the judging population. I remember the first time I judged being paranoid about whether I was in line with the others or was going to be #6 that was thrown out. Just had an experience where I got horribly inconsistent judging. How can you get 999866 ? Other folks at the same contest had similar observations. I think a calibration tool for the judges would go a long way towards fairness and objectivity. Has this ever been considered? Or was it too time consuming and / or expensive. I have been involved with Statistical Process Control for years in a manufacturing environment. It seems to me that Judges falling outside of standard deviation should be informed; whether high or low.
__________________
RUB-A-RIB-RIB BBQ
Two Modified WSM 22, iQue 110, CyberQ, Traeger Texas, Smokey Joe, Weber Gasser
CBJ
accuseal is offline   Reply With Quote
Unread 05-26-2013, 12:01 PM   #2
drbbq
is One Chatty Farker

 
drbbq's Avatar
 
Join Date: 10-26-04
Location: Saint Petersburg, Florida
Downloads: 0
Uploads: 0
Default

IMO that's the kind of thinking that is needed. But no cooks ever want to accept anything but 8-9 and an occasional 7 so all that's ever done is nudging judges into using only the top three numbers. No point in trying to sort that out. I'm guessing a real statistician would want to use most of the scale.
__________________
Ray Lampe
Dr. BBQ
The Honey Badger of BBQ. I just don't give a sh!t.
drbbq is offline   Reply With Quote
Thanks from: --->
Unread 05-26-2013, 02:58 PM   #3
Smoke'n Ice
is one Smokin' Farker
 
Join Date: 05-08-09
Location: Plano, TX
Downloads: 0
Uploads: 0
Default

Rumor has it that the new "Score" program, while very limited in its ability, has this as a feature. Can't keep track of coment cards, hence their elimination, but it can compare judges and tables. Remember this is just rumor and maybe some one on the software committee could post the SRS if they exist.
Smoke'n Ice is offline   Reply With Quote
Unread 05-26-2013, 03:21 PM   #4
carlyle
Full Fledged Farker

 
carlyle's Avatar
 
Join Date: 10-06-10
Location: Owatonna, MN
Downloads: 0
Uploads: 0
Default

Not on the committee or the board. I was there at Kansas City in January when there was a presentation about the new program.

New software is supposed to track judges. That is why score cards now have table and seat number plus judge number.

This kind of feed back is essential. Everybody needs to get an idea of how they are doing. Outliers, either too high or too low, need to be identified, informed, and efforts made to rehabilitate them.

Right now - at least at our contest, I encourage talk between judges after cards have been turned in. Table captains have their eyes open, and reps have a quiet talk with judges that are more than 2 points off from the rest of the table. Sometimes valid reason, sometimes not.

Before the contest, I pre assign table assignments in an effort to balance experience for each table. I do not want a hot table or a cold table. Right now the only tools are my previous experiences with the judges, plus their experience. Then with inexperienced judges or judges I don't know, I can fill them in around known quantities.

When people are involved who are doing subjective tasks - either cooking or judging-
the idea that you can calibrate them like a piece of machinery or a robot is an illusion
IMO.

Does not mean we should not make efforts to improve the status quo.

I am hopeful that the new scoring system will take a step in the right direction.

Time will tell.
__________________
Carlyle, KCBS, CBJ, Organizer ( Judge Chair)
carlyle is offline   Reply With Quote
Thanks from: --->
Unread 05-26-2013, 03:42 PM   #5
Smokin' D
is One Chatty Farker
 
Join Date: 12-11-07
Location: N. Andover, MA
Downloads: 0
Uploads: 0
Default

I am a judge and some things are difficult to control. Some judges, just like regular people, like what they like and do not like what they don't like. Here is my rib taste score from last weekend: 899949, Really? Texture: 997959, appearance was pretty even though. Looks like Judge #5 was in a hateful mood, or was judge #6 in disguise.

I do like to idea of calibrating the judging, if for no other reason than letting the outliers know they are off the norm.
Smokin' D is offline   Reply With Quote
Unread 05-27-2013, 07:42 PM   #6
ModelMaker
is One Chatty Farker
 
ModelMaker's Avatar
 
Join Date: 04-21-06
Location: Lake Ponderosa-Montezuma, IA
Downloads: 0
Uploads: 0
Default

Quote:
Originally Posted by accuseal View Post
I'm a bit new at this but I am a CBJ and have cooked my first contest. It seems to me that judges could be calibrated by taking their scores and after a few contests, giving them a report on how they stack up against the rest of the judging population. I remember the first time I judged being paranoid about whether I was in line with the others or was going to be #6 that was thrown out. Just had an experience where I got horribly inconsistent judging. How can you get 999866 ? Other folks at the same contest had similar observations. I think a calibration tool for the judges would go a long way towards fairness and objectivity. Has this ever been considered? Or was it too time consuming and / or expensive. I have been involved with Statistical Process Control for years in a manufacturing environment. It seems to me that Judges falling outside of standard deviation should be informed; whether high or low.

What makes you think the 3- 9's aren't the oddballs and the 8 was a gift?
Why is it so hard to accept that 2 judges thought your entry was just average and the rest were the dreaded high scoring, meat nibbling, cooler toting sort?
I have no problem scoring your entry as average if it is lacking somewhere, the "kiss of death 6" just means you need to up your game some. I'm not a 7,8,9 judge but rest assured if you give me an excellent piece of meat you get the 9. I give out lots of 9's and also a lot of 6's.
The day comes I have to justify a 6 is the day I just go back to eatin and judgin my own stuff, because 6 is just average not bad BBQ.
Ed
__________________
Designer of a custom drum smoker
Customized Aussie gas passer
Turkey fryer
Extremely seasoned hand hammered Wok
Pit Boss-HART BURN Competition BBQ Team
KCBS certified judge


And Your not the boss of me either!!
ModelMaker is offline   Reply With Quote
Unread 06-10-2013, 12:10 PM   #7
Porcine Perfection
is one Smokin' Farker

 
Porcine Perfection's Avatar
 
Join Date: 04-13-12
Location: KANSAS CITY MISSOURI
Downloads: 2
Uploads: 1
Default

Quote:
Originally Posted by Smokin' D View Post
I am a judge and some things are difficult to control. Some judges, just like regular people, like what they like and do not like what they don't like. Here is my rib taste score from last weekend: 899949, Really? Texture: 997959, appearance was pretty even though. Looks like Judge #5 was in a hateful mood, or was judge #6 in disguise.

I do like to idea of calibrating the judging, if for no other reason than letting the outliers know they are off the norm.
Perfect example of why the low score is thrown out.
__________________
Old user name was JP7794

Jim - KCBS CBJ

Porcine Perfection BBQ Team - Follow us on Twitter and Facebook

18.5" WSM (1983 model) | 18.5" WSM (2010 model) | Packer Green Thermapen | Auber PID Temp Controller | QMaster Senior Temp Controller | Mini WSM | Green Bay Packer UDS
Porcine Perfection is online now   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Loading



All times are GMT -5. The time now is 10:46 AM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.
vBulletin Optimisation provided by vB Optimise v2.6.0 Beta 4 (Lite) - vBulletin Mods & Addons Copyright © 2014 DragonByte Technologies Ltd.
2003 -2012 © BBQ-Brethren Inc. All rights reserved. All Content and Flaming Pig Logo are registered and protected under U.S and International Copyright and Trademarks. Content Within this Website Is Property of BBQ Brethren Inc. Reproduction or alteration is strictly prohibited.
no new posts