PDA

View Full Version : BBQ Contest Judging


Smoke Ring
08-23-2012, 12:47 PM
I competed in a Lone Star BBQ Society sanctioned contest last weekend which got me to thinking again about the KCBS judging process. I believe that the way the Texas rules contests are judged relies less on luck and is probably a more accurate way to pick the best meat entry.

First, there is no garnish to worry about. You lay the meat in the box on top of a sheet of aluminum foil. There is no distraction for the judges as to who made the prettiest bed of parsley, or if some of the lettuce leaves are a different color, or heaven forbid they used red tip lettuce or some other "illegal" garnish. The focus is on the meat.

All teams put the meat in the box in the same way. Chicken is half a chicken in one piece. Ribs are laid parallel to the box hinge with 5 ribs on the bottom and two more on top starting at the back. Brisket is 7 slices and the ends cannot be trimmed. It's an even playing field.

Sauce may not be added after the meat is removed from the smoker. It must be cooked on if used.

There are no "trained" or "certified" judges. Judges are selected from the public on site. They are judging the meat only, not lettuce and parsley, based on what they like and think is good, and have no pre-conceived notion about how to score, starting at 6, starting at 9, scoring down because the entry has sauce and they wonder what the cook is trying to hide. No luck of the draw on which table your entry lands. No worry about will you get a table where judges are mostly liberal with their scores (start at 9) or mostly conservative (start at 6) or will you get an experienced "master" judge with pre-conceived notions about what barbecue should or shouldn't be and guessing the motivations of the cook.

There were 31 teams. Entries went through two rounds of judging. In the first round the 14 best entries are chosen to move to the finals round where the top 10 are decided.

Scoring - Points are awarded based on where you placed in each category. First place gets 10 points and 10th place gets 1 point. Another team that places first in a different category gets the same number of points.

It is true blind judging. Boxes are identified by a raffle ticket. The cook takes one half and the other half is taped to the box with the number down. No one knows whose box is whose, including the contest official. At the awards the boxes are stacked in the order they finished, the ticket is removed from the box and the number is called out. The team holding the matching ticket claims the award. The overall GC and RGC aren't known until all the categories have been claimed, points assigned and totaled.

Is this the ultimate way of judging BBQ? I don't know. All I know is that the KCBS judging system is flawed and needs to be re-evaluated. Yes, I know there is a handful of teams that win regularly. But they also finish way down the list sometimes. You not only have to cook well, you need luck. A cooking competition should not rely on luck. Luck doesn't identify the best entry on any given day. I think KCBS needs to take a look at their judging process. The board seems to feel it is just fine the way it is. But there is always room for improvement.

SDAR
08-23-2012, 01:35 PM
I cook mostly Texas contests. I cooked next to you in Pueblo. Team name is Stop, Drop and Roll BBQ. Congrats on your Red River GC. We wanted to go but my wife is a teacher and couldn't get off that Friday.

As far as scoring goes, I like not having to garnish in Texas.

I don't think Texas scoring is as flawless as you might think though. With three different types of scores in KCBS, the entries don't result in a tie score very often. In Texas, you'd probably be surprised how many times the top in each category result in a tie score. When a tie occurs, there is actually a provision to "shuffle" those teams score cards and they "pick" the winner like trying to draw an ace from the hole. To me, that has a lot more luck than the KCBS system.

In other words, although you scored 1st in Brisket in Red River; you might have tied with 2nd and maybe even 3rd or 4th. This is especially true depending on how many teams are in the field. We got a 1st in Brisket out of 84 teams at one contest this year and I doubt very seriously that my brisket was scored higher than all 84 teams. I felt really good winning the 1st, but I'm also pretty sure I got the "luck of the draw".

Allen

Carnivorous Endeavors BBQ
08-23-2012, 02:09 PM
I agree with SDAR. No judging system is as perfect as they are made out to be. I believe they all have their upsides and downsides. Seemingly the biggest problem is that no organization wants to be like "that other organization" so there is a huge reluctance to change anything in all organizations because..."it works". With the usual texas judging styles there can be luck as SDAR stated. Generally with 25+ teams there are at least two preliminary tables. Boxes would usually get shuffled up and split between the two. If the shuffle happens to result in you getting paired with the top teams in the comp, you might not get off the preliminary table. Then again if you make the final table you can get into SDARs scenario where you make the winner lottery. Also, as far as random off the street judges are concerned, once you watch some of the "eaters" (think they are there for a meal) judge you could imagine seeing their scores drop after they are full and have 12 more boxes to judge.You just have to hope that you are not box #16 getting sampled. Or even better, when they are drunk and just smoked a pack of cigarettes, now that makes for excellent judges.
Again, NONE of the systems are perfect. If you like to compete, you just have to accept whatever that organizations methods and rules are, cook your best and roll those dice!

Outnumbered
08-23-2012, 02:47 PM
I like the theory of the KCBS judging. I don't care for the execution. Most judges score in the 7-9 range, but then there are outliers that score a 6 as average.

Judges are told they have to fill out a comment card if they score below a 6...not by any rule, it's just the way it is. So, you don't get below a 6 because most judges can't articulate why they give the scores they give.

This results in frustration. As a cook, how am I to know if the 6 I get is from a judge who believes my 6 is average, as the scoring system suggest...or does it really suck? Could be, but why did the guy next to that judge give the same sample a 9?

So, I don't know that I'm for wholesale changes in judging, but I would like the execution to be a little more consistent from judge to judge and contest to contest.

Just my 2 cents.

SDAR
08-23-2012, 03:30 PM
Oh I think they are both pretty good scoring systems and I didn't mean to trash the Texas systems. I have enjoyed cooking in them. Drinking is supposed to be prohibited but then what are they going to get kicked out of? A free meal?

Let me say that I think most folks will be sincere in judging. Texas comps are truly blind unless you go to one of the comps where they need judges so bad that they allow anyone but a head cook to judge...that is including team members, wife of head cook etc. I've actually seen it. I don't really like it but then again, we enjoy it and roll those dice.

We don't spend as much on Texas comps as we do when we do KCBS comps. Low this year was $75 entry to $150 for the high entry in Texas this year. We have done two KCBS this year and I believe it was $250 each time along with the cost of the extra meat category...which is not much for a pork butt but still. We rarely get electricity in the comps around the area I cook in and have never been offered water. There is no ice vendor or cart coming around to see if you need more.

I like KCBS. It is just that most of them are $200 worth of gas at a minimum for me to take the motorhome now days because none are local to here. I mostly stay local as a result. The trip from Amarillo to Pueblo cost us at a minimum $700 round trip. I would have literally been mad had I spent that kind of money and saw someone asking your cooking partner to come judge. :shocked:

We are going to Bedford next weekend. It's going to hurt the pocket book but man am I looking forward to it! It's KCBS!! It does feel like the big time compared to :behindsofa: the redneck cooks we do around here.

Smoke Ring
08-23-2012, 03:35 PM
Thanks Allen. I wasn't trying to imply that the Texas system was perfect, just that there are alternatives and room for improvement and I believe the KCBS judging system could be improved. As for ties, I was at the Blue Ridge BBQ Festival in Tryon, NC one year and the top 5 brisket scores were 180s.

Smoke Ring
08-23-2012, 03:38 PM
Even though ties may not be as common in KCBS contests, that's because scores are given to 4 decimal places. Do you really believe the scoring is that accurate? We have lost by .0002 points before, which is very common. I don't see much difference between that and a tie. I would just about as soon call it a tie and go to the tie breaker.

Bentley
08-23-2012, 03:39 PM
Entries went through two rounds of judging. In the first round the 14 best entries are chosen to move to the finals round where the top 10 are decided.

It is true blind judging.

Is this the ultimate way of judging BBQ?


When I Master the Art of Making my Competition meat taste good after sitting in ambient temperatures for 2 hours I will feel more comfortable competing in more IBCA type events.

It is a True Double Blind scoring system, that is the best thing about the contests.

No it is not.

SDAR
08-23-2012, 03:42 PM
Thanks Allen. I wasn't trying to imply that the Texas system was perfect, just that there are alternatives and room for improvement and I believe the KCBS judging system could be improved. As for ties, I was at the Blue Ridge BBQ Festival in Tryon, NC one year and the top 5 brisket scores were 180s.

No problem. I agree. At least in KCBS you know you tied though. In the scenario above with 5 180's, if that was a Texas comp and you wound up in 5th...the thought is that 4 other teams beat you.

The judging is blind...but so is the scoring. If you are DAL in KCBS...probably time to change something. If you ain't in the top ten in Texas, you might have been DAL...unless they read final table...then if you did't make final table...you might have been DAL and not even know it.

SDAR
08-23-2012, 03:48 PM
When I Master the Art of Making my Competition meat taste good after sitting in ambient temperatures for 2 hours I will feel more comfortable competing in more IBCA type events.

Ever thought it may be an acquired taste on the judges part instead of good tasting cold food? Some of the bbq dives we set our taste buds with don't have the best warming equipment. :hand:

Smoke Ring
08-23-2012, 03:50 PM
No problem. I agree. At least in KCBS you know you tied though. In the scenario above with 5 180's, if that was a Texas comp and you wound up in 5th...the thought is that 4 other teams beat you.

The judging is blind...but so is the scoring. If you are DAL in KCBS...probably time to change something. If you ain't in the top ten in Texas, you might have been DAL...unless they read final table...then if you did't make final table...you might have been DAL and not even know it.

Not getting feedback on the scores, and how you placed if you weren't in the top 10, is one of the weaknesses of the Texas system and one of the pluses for KCBS. I'm not condemning KCBS. I'd just like to see some out of the box thinking when it comes to judging and scoring. The current system has some obvious flaws.

As for DAL, do you really want to know? :grin:

ique
08-23-2012, 03:55 PM
I competed in a Lone Star BBQ Society sanctioned contest last weekend which got me to thinking again about the KCBS judging process. I believe that the way the Texas rules contests are judged relies less on luck and is probably a more accurate way to pick the best meat entry.


Does this have anything to do with the fact that you won the contest? :wink:

SDAR
08-23-2012, 03:57 PM
[QUOTE=As for DAL, do you really want to know? :grin:[/QUOTE]

Nope. Not at all. I am still embarassed from the 3rd from DAL ribs they threw back at me in Pueblo. :shock:

I think you ought to be able to pick up a rib to eat it by the way instead of trying to cut off a piece with a plastic fork and knife like we do in Texas. What's the bone for anyway? Try cooking for awhile giving good meat that will cut off the bone and then try cooking against someone wanting a white bone that leaves perfect teeth marks on a rib where the meat just barely pulls away...sat WHAT??

Smoke Ring
08-23-2012, 04:00 PM
Does this have anything to do with the fact that you won the contest? :wink:

Not really. I've won KCBS contests too. :shocked:

And if you remember I've had a bad experience with a Texas rules contest in the past. No system works without a competent official overseeing things backed by an organization that wants to make things right.

I'm giving them a second chance and doing another one of their contests next month. Not the same head judge, though.

Carnivorous Endeavors BBQ
08-24-2012, 11:12 AM
Honestly, not releasing scores and the letting team members/immediate family members judge are my two biggest complaints of IBCA. I would guess not releasing scores is a good way of not letting people know how many tie breaker lottery draws actually happen because i don't see any other good reason to not let adults, who just spent a lot of effort and money, know how they were scored. As for the other issue, Head Judges are supposed to now ask if anyone is a team or immediate family member and not let them judge Final Table, however, it is not technically against the rules. So as long as you have good Head judges then you should get a fair shot if you get off the preliminary table.
It is not bashing IBCA, and I don't believe KCBS is superior, comparing them is an apples and oranges comparison.

Lake Dogs
08-24-2012, 11:26 AM
Sounds like a nice combination of MBN (without the onsite), GBA (without the finals), IBCA, and CASI (chili). :-)

Not bad, except... I found over time with IBCA and CASI that I actually preferred trained judges. Even trained some will go rogue, but it seems more likely with untrained.

Jorge
08-24-2012, 11:39 AM
Trained or not, judges usually tend to get it right. There are too many consistent winners in each organization to argue otherwise, as well as teams that do well consistently in multiple organizations.

stl-rich
08-24-2012, 11:59 AM
I used to judge KCBS - in what way is their system NOT double blind? Neither the Table Captain nor the individual Judges know who cooked that box

Jorge
08-24-2012, 12:32 PM
I used to judge KCBS - in what way is their system NOT double blind? Neither the Table Captain nor the individual Judges know who cooked that box

Anybody that knows the alternate # knows who the box belongs to. Technically it is not a double blind system.

IBCA uses tickets, and no record is kept of which team gets a ticket. If a team loses a ticket, or doesn't claim a box that scores there is no way to determine who it belonged to. That is double blind.

ModelMaker
08-25-2012, 07:35 AM
Anybody that knows the alternate # knows who the box belongs to. Technically it is not a double blind system.

IBCA uses tickets, and no record is kept of which team gets a ticket. If a team loses a ticket, or doesn't claim a box that scores there is no way to determine who it belonged to. That is double blind.


Uh, I don't think so Tim...
Yes, a table captain you could figure out the "add to" number but no judge sitting at the table could come up with the magic number.
The only way to know whos box is whos is to know the teams box number and then the "add to" number. I think a judge at a table would have no way to complete this task.
The double blind is safe and secure.
Ed

Jorge
08-25-2012, 11:32 AM
Uh, I don't think so Tim...
Yes, a table captain you could figure out the "add to" number but no judge sitting at the table could come up with the magic number.
The only way to know whos box is whos is to know the teams box number and then the "add to" number. I think a judge at a table would have no way to complete this task.
The double blind is safe and secure.
Ed

KCBS does not use a true double blind system. It's a blind system. It's unlikely that a table captain or judge would ever learn the alternate # prior to judging, that's true. But the fact is that the system is not double blind. There will be a handful of people that have access to that info prior to judging.

IBCA is an example of a double blind system. The box given to a team has two tickets with matching numbers attached. The team retains one and the other remains on the box. The judges, head judge, volunteers have NO list to determine which team was given a particular #. That is a double blind system.

Either way, I think this has diverged from the original topic.

Lake Dogs
08-25-2012, 05:06 PM
There are drawbacks to a double blind system. First, it makes for a very unusual awards ceremony. Second, there are ways for teams to cheat so that one particular team makes GC's; enough to qualify them for things like The Royal, The Jack, etc. I've seen competitions in CASI where teams have done that very thing to qualify for Terlingua, only to be found out and the contest loses it's sanctioning...

Every system has its strengths and weaknesses. There's benefits of a 7-10 scoring system, and there are benefits to a 1-9 system. Each different. There are benefits to systems with finals, and benefits to those without. Benefits to on-site (with blind too), and benefits to purely blind.

Choose those that you prefer and compete away.

CBQ
08-25-2012, 09:15 PM
There are drawbacks to a double blind system. First, it makes for a very unusual awards ceremony. Second, there are ways for teams to cheat so that one particular team makes GC's; enough to qualify them for things like The Royal, The Jack, etc. I've seen competitions in CASI where teams have done that very thing to qualify for Terlingua, only to be found out and the contest loses it's sanctioning...

Every system has its strengths and weaknesses. There's benefits of a 7-10 scoring system, and there are benefits to a 1-9 system. Each different. There are benefits to systems with finals, and benefits to those without. Benefits to on-site (with blind too), and benefits to purely blind.

Choose those that you prefer and compete away.

Yes. Since nobody knows who has the tickets, what's to prevent teams from collaborating and switching tickets to help a team win? (You win this week, you help me win next week...) We see that kind of stuff happen in people's choice all the time.

gmholler
08-26-2012, 07:53 AM
Yes. Since nobody knows who has the tickets, what's to prevent teams from collaborating and switching tickets to help a team win? (You win this week, you help me win next week...) We see that kind of stuff happen in people's choice all the time.

What's to prevent that? The head cook has to sign his ticket, which has to be validated by contest official before they can receive their award. Sure, it can happen in people's choice - that's usually a lot less stringent(around here, "just for bragging rights") and the rules are up to the organizer.

Lynn H.