ブタ
The BBQ BRETHREN FORUMS.


Forum Portal Recipes Smoke Signals Magazine Welocme Merchandise Associations Purchase Subscription Brethren Banners
Go Back   The BBQ BRETHREN FORUMS. > Discussion Area > Competition BBQ

Notices

Competition BBQ *On Topic Only* Discussion regarding all aspects of Competition BBQ. Experiences competing or visiting, questions, getting started, Equipment, announcements of events, Results, Reviews, Planning, etc. Questions here will be responded to with competition BBQ in mind.


Reply
Thread Tools
Unread 12-18-2009, 12:51 PM   #1
QansasjayhawQ
is One Chatty Farker

 
QansasjayhawQ's Avatar
 
Join Date: 06-21-06
Location: Lawrence, Kansas - The Great American Outback
Downloads: 0
Uploads: 0
Lightbulb Judging Parity

I've been thinking about this and I would like to hear everyone else's thoughts too.

I do both - cook on a couple of teams AND judge. I started judging simply because I realized after a couple of events that I had NO IDEA what it was that we were shooting for (other than simply having some fun cooking BBQ).

As it turns out, I really enjoy judging and I've been doing it for a couple of seasons and I've enjoyed getting to know 'the regular' judges who travel far and wide to provide their services.

One thing that bothers me is that the scores from a "celebrity" judge (like a local radio personality or the mayor or the fire chief, etc.) are weighted just the same as a non-certified judge that are weighted just the same as a 'rookie' judge (say, less than 10 competitions judged) and their scores are weighted the same as 'experienced' judges and all of them have just as much say in their scores as Master judges.

This doesn't seem like a very fair set up.

Couldn't the KCBS do some things to help level this out? For example;

1) Track all scoring by all certified judges.
2) Weight judge's scores by some kind of 'experience factor' giving greater credence to experienced and Master judge's scores.

That way, the celebrity or non-certified judges who are tasting competition BBQ for the first time don't just give everything a 9 across the board - as they tend to do. And the tendency for rookie judges to compare entries against each other could also be accommodated using a system like this.

This would require more data entry time on site by the rep doing the data entry. Are there any other problems this might cause that I am missing?

Thank you for your thoughts on this.
__________________
We should be too big to take offense and too noble to give it. - Abraham Lincoln
-------
One Moocow UDS! (Thanks, Shane!)
http://www.moocowbbq.com/
Two Weber 22.5" kettles One Brinkmann Smoke'N PitMaster Deluxe One LARGE WSM! (special black glossy finish) w/Stoker! WooHoo! KCBS Member/KCBS Master CBJ #24295
Sweaty, smokey and enjoying every minute.
QansasjayhawQ is offline   Reply With Quote


Unread 12-18-2009, 01:07 PM   #2
Lake Dogs
Quintessential Chatty Farker

 
Lake Dogs's Avatar
 
Join Date: 07-14-09
Location: Lake Sinclair, GA
Downloads: 0
Uploads: 0
Default

Good point you're making, and this is a large factor in why many/most sanctioned
cookoffs try hard to get/recruit only certified judges.

Nothing is more frustrating to a team than having someone with mediocre product
outscore their tried-and-true better product because a judge or two has no clue
as to what they're doing (or score theirs down for the same reason).

This is a problem with non-santioned cookoffs and other sanctioned cookoffs where
they cant get trained judges (ala. chili).
__________________
Hance - Lake Dogs Cooking Team - MiM/MBN/GBA CBJ and comp cook
Lake Sinclair, GA (strategically about an hour from darn near anywhere)
Started competing in chili cookoffs back in the 1990's and have competed in more than I care to count. I became a CBJ in MiM in 2005, then MBN and in GBA in 2010. I've probably judged 130+- BBQ comps (sanctioned and unsanctioned) over this time. That said, I really enjoy competing more than I enjoy judging, and hope to get back to doing 4 or 5 a year in the near future.
Lake Dogs is online now   Reply With Quote


Unread 12-18-2009, 01:12 PM   #3
KC_Bobby
Babbling Farker
 
KC_Bobby's Avatar
 
Join Date: 05-03-07
Location: Gardner KS
Downloads: 0
Uploads: 0
Default

I'd say an organizer that wants to get some celebrity judges should get six of them. Then set them at a table together in a corner and have some volunteers cook their food. The celebrity table wouldn't get used for the comp teams and they'd be non the wiser.

If the mayor of Williamstonburgsonville doesn't have time to take the class, he doesn't have time to judge.

Am I kidding - sorta but if an organizer did it I'd laugh with him
__________________
Boondoggle BBQ
Country Smokers 450
Weber Smokey Mountain 18"
Proud KCBS Member & Certified Judge

KC_Bobby is offline   Reply With Quote


Unread 12-18-2009, 01:23 PM   #4
goodsmokebbq
is one Smokin' Farker
 
goodsmokebbq's Avatar
 
Join Date: 09-17-07
Location: Rochester, NY
Downloads: 0
Uploads: 0
Default

I have had some requests for these "celeb" types to judge and I tell them they have to take the class and they usually back out. They usually don't have tht two hours to give anyways.
__________________
Brian

There are 10 kinds of people in the world, those who understand binary and those that don't

http://www.goodsmokebbq.com
http://www.roccityribfest.com
http://www.nybbqchamp.com
Backwoods Party,Backwoods Competitor,Weber 26
goodsmokebbq is offline   Reply With Quote


Unread 12-18-2009, 01:24 PM   #5
Alexa RnQ
is One Chatty Farker

 
Alexa RnQ's Avatar
 
Join Date: 09-12-07
Location: the Ninth Ring of Hell, cleverly disguised as Phoenix
Downloads: 0
Uploads: 0
Default

Quote:
Originally Posted by KC_Bobby View Post
I'd say an organizer that wants to get some celebrity judges should get six of them. Then set them at a table together in a corner and have some volunteers cook their food. The celebrity table wouldn't get used for the comp teams and they'd be non the wiser.

If the mayor of Williamstonburgsonville doesn't have time to take the class, he doesn't have time to judge.

Am I kidding - sorta but if an organizer did it I'd laugh with him
This is the best idea I've heard yet. Seriously.

As far as "fairness", the entire field is subjected to the judging pool. Reps here try hard to spread less-experienced judges evenly between tables, so no table is too hard hit. But for sure, teams ask about percentage of CBJs, and are cognizant that if a class was held the night before...

The other judging system I've seen that works around this somewhat is that used by the American Rose Society. In that organization at that time, a newly-minted judge had an apprenticeship period. They would score shows in the company of an experienced judge, to make sure that their initial judging experiences were in line with the milieu in which they judged.

Whether it's through appenticeships or continued education, there are things to be said for doing what can be done to ensure that the greater body of available judges share a common and current worldview.
__________________
.

Don't practice until you get it right; practice until you can't get it wrong.
Don't overthink or undercook.

180 Brisket -- Sam's Club, Salt Lake City, UT 2014
180 Chicken -- Sam's Club National, Bentonville, AR 2013


Rhythm 'n QUE
Alexa RnQ is online now   Reply With Quote


Unread 12-18-2009, 01:40 PM   #6
goodsmokebbq
is one Smokin' Farker
 
goodsmokebbq's Avatar
 
Join Date: 09-17-07
Location: Rochester, NY
Downloads: 0
Uploads: 0
Default

Quote:
Originally Posted by QansasjayhawQ View Post
I've been thinking about this and I would like to hear everyone else's thoughts too.

I do both - cook on a couple of teams AND judge. I started judging simply because I realized after a couple of events that I had NO IDEA what it was that we were shooting for (other than simply having some fun cooking BBQ).

As it turns out, I really enjoy judging and I've been doing it for a couple of seasons and I've enjoyed getting to know 'the regular' judges who travel far and wide to provide their services.

One thing that bothers me is that the scores from a "celebrity" judge (like a local radio personality or the mayor or the fire chief, etc.) are weighted just the same as a non-certified judge that are weighted just the same as a 'rookie' judge (say, less than 10 competitions judged) and their scores are weighted the same as 'experienced' judges and all of them have just as much say in their scores as Master judges.

This doesn't seem like a very fair set up.

Couldn't the KCBS do some things to help level this out? For example;

1) Track all scoring by all certified judges.
2) Weight judge's scores by some kind of 'experience factor' giving greater credence to experienced and Master judge's scores.

That way, the celebrity or non-certified judges who are tasting competition BBQ for the first time don't just give everything a 9 across the board - as they tend to do. And the tendency for rookie judges to compare entries against each other could also be accommodated using a system like this.

This would require more data entry time on site by the rep doing the data entry. Are there any other problems this might cause that I am missing?

Thank you for your thoughts on this.

I like your line of thought, but it would be very hard to calculate what the weights should be. It would take mountains of data and a real statistician to try and work it out. And it would only work if there really is a relationship between judging experience and scores. If on average the new judge always scores low then ok, but if it goes both ways, some new judges score high and some low then a weight won't work. The mean would be about the same as a experienced judge only there variance would be larger. Almost impossible to adjust for.
__________________
Brian

There are 10 kinds of people in the world, those who understand binary and those that don't

http://www.goodsmokebbq.com
http://www.roccityribfest.com
http://www.nybbqchamp.com
Backwoods Party,Backwoods Competitor,Weber 26
goodsmokebbq is offline   Reply With Quote


Unread 12-18-2009, 01:42 PM   #7
Skip
is One Chatty Farker
 
Skip's Avatar
 
Join Date: 07-24-07
Location: Wantagh, NY
Downloads: 0
Uploads: 0
Default

Avctually I think FBA does the best job at this. They track all their judges and categorize them according to their judging style. Those that are tough markers those that mark favorably and those who are middle of the road. When they go to seat these judges they do their best to offer each table a equal amount of each type so that each table is "fair". Of course it doesn't always work but it does do something to change that table with four friends who haven't given a 9 in their career or adversely that group of newbies, that gauge BBQ on Charlie Browns thursday night BBQ special, who "can't believe how good all of this food is. I gave all 9's because each one was better then the next." Its these judges I worry most about not the celebs.
__________________
WannaBeBBQueen's Prince Consort

Swamp Pit BBQ's freeloader

To part or not to part: that is the question. Whether ‘tis nobler in the mind to part and suffer the slings and arrows of ostentatious brothers or to not part when faced with a sea of dissent, and by compromise end it? To part: to succeed. Forever more: and by succeed to say we win.

Yeah it looks good...but how does it smell?

Member of The Big Guns BBQ Team Semper Fumantem

"you're eating tree RAT, tree RAT, you're eating RAT!" -- landarc
Skip is offline   Reply With Quote


Unread 12-18-2009, 01:52 PM   #8
goodsmokebbq
is one Smokin' Farker
 
goodsmokebbq's Avatar
 
Join Date: 09-17-07
Location: Rochester, NY
Downloads: 0
Uploads: 0
Default

I do like the tracking idea, you could weight specific judges (which is pretty analogous to what the FBA does by spreading them out).
__________________
Brian

There are 10 kinds of people in the world, those who understand binary and those that don't

http://www.goodsmokebbq.com
http://www.roccityribfest.com
http://www.nybbqchamp.com
Backwoods Party,Backwoods Competitor,Weber 26
goodsmokebbq is offline   Reply With Quote


Unread 12-18-2009, 02:00 PM   #9
Skip
is One Chatty Farker
 
Skip's Avatar
 
Join Date: 07-24-07
Location: Wantagh, NY
Downloads: 0
Uploads: 0
Default

Quote:
Originally Posted by goodsmokebbq View Post
I do like the tracking idea, you could weight specific judges (which is pretty analogous to what the FBA does by spreading them out).

The only problem I see with weighting each judge is that specific areas yield different flavor profiles. A judge could mark more favorably in one area then the next yet still be a tough judge. I think the mixing of judging styles would even out more then weighting. Even if they like the flavor profile they are still judging the same whereas weighting to a specific number may artificially bias the scoring.

Sorry I tried to explain myself. Does that make sense to anyone?
__________________
WannaBeBBQueen's Prince Consort

Swamp Pit BBQ's freeloader

To part or not to part: that is the question. Whether ‘tis nobler in the mind to part and suffer the slings and arrows of ostentatious brothers or to not part when faced with a sea of dissent, and by compromise end it? To part: to succeed. Forever more: and by succeed to say we win.

Yeah it looks good...but how does it smell?

Member of The Big Guns BBQ Team Semper Fumantem

"you're eating tree RAT, tree RAT, you're eating RAT!" -- landarc
Skip is offline   Reply With Quote


Unread 12-18-2009, 02:07 PM   #10
goodsmokebbq
is one Smokin' Farker
 
goodsmokebbq's Avatar
 
Join Date: 09-17-07
Location: Rochester, NY
Downloads: 0
Uploads: 0
Default

I really like the spreading out idea based on classification (high marker, average marker, etc.). By doing this you are in effect weighting the judges to create parity across the tables. You are using the judges classification to weight the other judges on the table.

I want to hit the table of all high markers!

Weighting each judge would be very ineffiecnt and very impratical, and it would fail if you don't truely understand a judges distribution of scores (like scoring higher in the south, etc). Any sort of weighting is biasing the data, you just need to make sure you are biasing in the right direction, towards the population average. Weighting can be very tricky business.
__________________
Brian

There are 10 kinds of people in the world, those who understand binary and those that don't

http://www.goodsmokebbq.com
http://www.roccityribfest.com
http://www.nybbqchamp.com
Backwoods Party,Backwoods Competitor,Weber 26
goodsmokebbq is offline   Reply With Quote


Unread 12-18-2009, 02:43 PM   #11
Buster Dog BBQ
is One Chatty Farker
 
Join Date: 07-20-08
Location: Clive, IA
Downloads: 0
Uploads: 0
Default

Do you see that with reps? There are some reps I like to go to contest they are at because the scoring is a little higher and more consistent then others.
__________________
FEC-100 Right Handed, FEC-100 Left Handed, Large BGE x2, Mini BGE
Buster Dog BBQ - [url]http://www.busterdogbbq.com[/url]
Buster Dog BBQ is offline   Reply With Quote


Unread 12-18-2009, 02:47 PM   #12
Skip
is One Chatty Farker
 
Skip's Avatar
 
Join Date: 07-24-07
Location: Wantagh, NY
Downloads: 0
Uploads: 0
Default

Quote:
Originally Posted by Buster Dog BBQ View Post
Do you see that with reps? There are some reps I like to go to contest they are at because the scoring is a little higher and more consistent then others.
I can't think of a way that particular reps could affect scoring. Only through illegal ways and I won't make that claim at all. The only thing I can think of is the reps have a following of judges and these judges are either easier scorers or the percentage of CBJ's goes up so the caliber of judging goes up.
__________________
WannaBeBBQueen's Prince Consort

Swamp Pit BBQ's freeloader

To part or not to part: that is the question. Whether ‘tis nobler in the mind to part and suffer the slings and arrows of ostentatious brothers or to not part when faced with a sea of dissent, and by compromise end it? To part: to succeed. Forever more: and by succeed to say we win.

Yeah it looks good...but how does it smell?

Member of The Big Guns BBQ Team Semper Fumantem

"you're eating tree RAT, tree RAT, you're eating RAT!" -- landarc
Skip is offline   Reply With Quote


Unread 12-18-2009, 02:52 PM   #13
Lake Dogs
Quintessential Chatty Farker

 
Lake Dogs's Avatar
 
Join Date: 07-14-09
Location: Lake Sinclair, GA
Downloads: 0
Uploads: 0
Default

They can if and how they "charge the judges", like by stressing what a 9 means, what
an 8 means... Stressing. If not, even with certified judges, you end up with some
pretty bizarre variances. I do this for CASI (charge the judges) every time and stress
it. Pretty much the scores on the same chili rarely if ever vary by more than a point
or two...
__________________
Hance - Lake Dogs Cooking Team - MiM/MBN/GBA CBJ and comp cook
Lake Sinclair, GA (strategically about an hour from darn near anywhere)
Started competing in chili cookoffs back in the 1990's and have competed in more than I care to count. I became a CBJ in MiM in 2005, then MBN and in GBA in 2010. I've probably judged 130+- BBQ comps (sanctioned and unsanctioned) over this time. That said, I really enjoy competing more than I enjoy judging, and hope to get back to doing 4 or 5 a year in the near future.
Lake Dogs is online now   Reply With Quote


Unread 12-18-2009, 02:54 PM   #14
Skip
is One Chatty Farker
 
Skip's Avatar
 
Join Date: 07-24-07
Location: Wantagh, NY
Downloads: 0
Uploads: 0
Default

Quote:
Originally Posted by Lake Dogs View Post
They can if and how they "charge the judges", like by stressing what a 9 means, what
an 8 means... Stressing. If not, even with certified judges, you end up with some
pretty bizarre variances. I do this for CASI (charge the judges) every time and stress
it. Pretty much the scores on the same chili rarely if ever vary by more than a point
or two...

Ah that makes a lot of sense. Thank you.
__________________
WannaBeBBQueen's Prince Consort

Swamp Pit BBQ's freeloader

To part or not to part: that is the question. Whether ‘tis nobler in the mind to part and suffer the slings and arrows of ostentatious brothers or to not part when faced with a sea of dissent, and by compromise end it? To part: to succeed. Forever more: and by succeed to say we win.

Yeah it looks good...but how does it smell?

Member of The Big Guns BBQ Team Semper Fumantem

"you're eating tree RAT, tree RAT, you're eating RAT!" -- landarc
Skip is offline   Reply With Quote


Unread 12-18-2009, 02:59 PM   #15
cmcadams
Quintessential Chatty Farker

 
cmcadams's Avatar
 
Join Date: 02-15-06
Location: Waynesville, Ohio
Downloads: 0
Uploads: 0
Default

Quote:
Originally Posted by goodsmokebbq View Post
I have had some requests for these "celeb" types to judge and I tell them they have to take the class and they usually back out. They usually don't have tht two hours to give anyways.
Just because they're local celebs (usually VERY local) doesn't mean they're any busier than anyone else. If you want free bbq, at least take the time to get an idea what you're doing.

BTW, great logo!
__________________
Curt McAdams
Livefire Blog

Livefire Whisky

CBJ, some comps, a few awards, Certified Artisan Steak Taster.

XL Big Green Egg, Bubba Keg Convection Grill, Weber Kettle, Charbroil Infrared and mad livefire cooking skillz! :)
cmcadams is online now   Reply With Quote


Reply

Similar Threads
Thread Thread Starter Forum Replies Last Post
American Royal Judging.... Who's going? and a question for invitational judging veterans linhardt Competition BBQ 28 10-12-2010 01:22 PM
Judging Priority KC_Bobby Competition BBQ 32 02-21-2008 11:30 AM
First judging n-2-que Competition BBQ 12 07-16-2007 05:49 PM

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Loading



All times are GMT -5. The time now is 07:18 PM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.
vBulletin Optimisation provided by vB Optimise v2.6.0 Beta 4 (Lite) - vBulletin Mods & Addons Copyright © 2014 DragonByte Technologies Ltd.
2003 -2012 © BBQ-Brethren Inc. All rights reserved. All Content and Flaming Pig Logo are registered and protected under U.S and International Copyright and Trademarks. Content Within this Website Is Property of BBQ Brethren Inc. Reproduction or alteration is strictly prohibited.
no new posts