PDA

View Full Version : BBQ Judging - Newsletter from Bill Anderson of Chatham Artillery BBQ Team


Bogus Chezz Hawg
01-29-2010, 04:43 PM
I think this might stir the pot...

Here's the link to the newsletter...

http://www.bbqsuccess.com/barbecue-judging/

I also pasted the newsletter below because it displays improperly in Mozilla 3.6, (but it does display properly in Internet Explorer).

Dear BBQ Friend:
What better way to start out the new year than complaining about all the things that need improvement in our little barbecue industry...
I've been meaning to write a newsletter about this for a couple of years now. The last newsletter I did on barbecue judging was way back in December 2006 (http://www.bbqsuccess.com/bbq-competitions/) - and it hasn't changed much in the last 3 years! In that Dec 2006 newsletter, I basically explained how things work so you would know what to expect at a contest. This newsletter is a bit different... I am going to offer some suggestions for improvement.
Note: There are good barbecue judges out there and a few not so good ones. This newsletter is not intended to offend anyone. Nor is it intended to take anything away from barbecue contest winners. We are all in the same judging boat and all wins are well deserved.
I can assure you, not many people besides barbecue competitors know how much work is involved in competing in a barbecue contest. Even some seasoned barbecue judges do not have a clue. Even some Master Certified Barbecue Judges don't have a clue... all they are required to do is help a team at a contest. Note I said "at a contest". Well, what about the 2 days of preparing for a contest? And what about the 2 days of cleanup after a contest? And what about the many, many hours of practice? Just helping a team at a contest does help a judge understand what goes into a contest, but it is still only the tip of the iceberg. If a judge really wants to become a master and understand fully what is involved, let them go out and buy all of their own equipment, spend years learning how to cook competition quality barbecue, pay the entry fee, buy the meat and other supplies, spend 2 days preparing for a contest, driving many miles to get there, working your ass off at a contest, and then driving home and cleaning up. And... why do we do this again? Oh... that's right - for the fun of it!


(http://www.barbecuecoach.com/)
Maybe we would get better scores if all judges were required to go through all that! But I understand that that's not possible. So... how can we go about improving barbecue judging? I think we can all agree that the current judging is not exactly fair and after spending around $700+ for a contest, I think we deserve fair judging. A lot of my scores ranged from 9's and 8's to 6's and 7's at some tables last year. So... my first question is - "Is it even possible to improve barbecue judging?" After all, everybody has their own tastes even though they are taught to ignore their personal tastes. But is that even possible? Probably not.
Then there is the question of certified Barbecue Judges (CBJ). I personally think we are slightly better off with CBJs, but I say only slightly because they are not always fair either. Some of them have tasted the best and hold everybody else up to a higher standard - thus low scores for everyone. Don't know if this is right or wrong to be honest with you. And using people off the street could result in a lot of low scores or too high scores - who knows!? I think my point is... we need a better system to overcome these innate "discrepancies". So, here are my suggestions...




Use more judges - Use 8 judges per table instead of just 6 and throw out the high AND the low scores. Most sports that require human judging use a system like this (i.e. ice skating, body building, gymnastics, beauty pageants, etc). It's no problem for barbecue competitors to turn in 8 pieces instead of 6. Most of us do this anyway. The argument over wether CBJs are better than people off the street or celebrity judges is diminished because the extremes are thrown out. You are left with 6 good scores that are added up instead of 5 - hopefully giving you a better "average" of the best judges present at that table. This way, one or two or even three out of line scores will not hurt as much. All the other rules would stay the same. Most contests would only need between 10-20 extra judges and it would not matter as much if they were CBJs or not. And about the "boneyard" - that's the extra barbecue in the turn-in box for the volunteers and table captains. There's plenty of barbecue left over at the team's site that can't fit in the box. If organizers went around collecting barbecue donations for the volunteers, they would get plenty. Or competitors could just turn in a separate boneyard box when they turn in their regular box.
Instead of a total disqualification for a late entry, just start deducting points for each second late. I think that is much more fair after all the work they did to turn in that category. You could lose a whole contest and waste an entire weekend just because you were a few seconds late in one category. That is not fair... especially when other circumstances may be involved like walking distance and having to dodge pedestrians. In one contest, we had to wait for a golf cart to drive us the 1/4 mile to the turn in station. Without that, we would have been at a big disadvantage.
Don't use any judges - Say what, Bill? Have you lost your mind!? Nope... there's plenty of very well qualified people around barbecue contests to do the judging. Who? I'm talking about the contestants themselves! Who else is more qualified? Here's how you would go about it...




After cooking their barbecue, contestants are usually just sitting around drinking beer for hours until the awards ceremony. Plenty of time to do a little judging.
Since you only need one judge per team (maybe a few extra if you use 8 judges per table), there will be plenty of judges. Anyone on a team can be assigned the task of judging. The rest of the team can still be cleaning up or they can clean up after judging is complete.
Contestants are there anyway, so organizers don't have to worry about securing judges.
I think a lot of contestants would actually enjoy this part of a competition. I know I would because we don't get to taste a competitor's barbecue very often and I think it would be a great learning experience.
Anti-cheating mechanism: A mathematical formula could be used to prevent a judge from purposely giving other teams undeserved bad scores. If their scores are way out of line on the low side than all the other judges at that table, then their own scores will be penalized by a deduction. And remember... it would have to be obviously WAY out of line with the rest of the table. And this would only be done with low scores and not high scores. Judges would have no choice but to be fair.
Cheating is possible in the present system... A friendly judge could be planted who could easily recognize their friend's box. They would give good scores if they got their friend's box and bad scores otherwise. I hate to say it, but if it is possible to cheat, in my experience, there will be cheaters. As Ronald Reagan always said... "Trust, but verify". If they get half a chance, they cheat in NASCAR, the NFL, the NBA, MLB, bicycle racing, and just about every other major sport. What makes you think cheating is not going on at barbecue contests?
Forget about garnishments. It will be too much of a hastle to mess with garnishments when doing all 4 meat categories in that last hour before turn-in. Most teams don't like garnishments. Most judges don't like garnishments. And it isn't a parsley contest - it's a barbecue contest. I think garnishment also gives rise to the possibility of a judge being able to recognize a box easier.
You'll have to implement a system where nobody's box goes to their own table. Should be easy to do with a simple computer program and a known seating arrangement.
All 4 categories would be turned in at one time. Let's say a 30 minute window.
There will be some lag time between turn in and actual judging, so all entries would be put on a tray and slipped into a warming oven. They should only have to be kept warm for about 1 hour. Even something as simple as an electric blanket could be used for this. And if you don't allow any garnishment, it will not wilt. We all know that in the current system, turn-in boxes can easily sit around for an hour before being judged. Is it fair to let a competitor's turn-in box get ice cold before being judged? I think not.
And last but not least... I know a lot of judges will be "unemployed" if this idea is implemented. So, I suggest that there be some way for the general public to taste or sample competitor's barbecue. I think it would make visiting a barbecue contest much more enjoyable for everyone. Of course, some details would have to be worked out like health code regulations and fairly compensating teams for their contribution. Maybe a buffet area could be set up where teams would deposit their meats. The buffet area would conform to all health codes. The public would pay to try out the samples. Not exactly sure if this would conform to the health code or not. Whatever you do, don't make the teams deal directly with the visitors - that is too much to ask of teams that are not set up for that kind of thing. And they are usually too busy with competing anyway. Another option for unemployed judges is to move to another sanctioning body or become a volunteer at a contest. Volunteers, table captains, turn-in box receivers, contest reps, etc will still be needed.





So... there you have it! Those are my crazy ideas. Take 'em or leave 'em. If you are a contest organizer, contest rep, on the board of directors, or work for a sanctioning body, feel free to suggest these ideas to the powers that be. After all, in my opinion, the main purpose of a barbecue sanctioning body is to provide the judging at barbecue contests. Think about that for a second... their main job is to provide fair judging at barbecue contests - and they get it wrong all too often. If someone were to come along and "get it right", someone may be looking for a job in a tough environment. There's a huge need in a relatively very young sport... I was always taught in business - "find a need and fill it". I think you all know this could happen. And one or two big time contest organizers are already thinking about it. There were many different racing leagues in the early 1940's before Bill France Sr started NASCAR. Ever heard of any of them? No? Point made.
While I'm on my soapbox... let's see some increases in prize money! I am sick and tired of seeing these contests which only have $4000-$5000 total prize money! That is a joke! First place would not even cover my expenses. I won 4 Grand Championships in 2009 and over $10,000 in prize money and I barely covered my expenses for the 10 contests I entered last year. There's a whole lot of room for improvement if you ask me. Like I said... somebody's gonna come along real soon and "find the need and fill it" . I watched 4 contestants on TV do some kind of chocolate sculpture contest. The winner got a check for $10,000. I about fell out of my Lazy Boy! I work my ass off for the better part of a week and can only dream about winning $10,000 at one contest. Those TV networks film a barbecue contest and don't put up a dime in prize money. Are we a bunch of suckers? Try to film a NASCAR event or a NBA game without paying big bucks - good luck with that! Again... our sanctioning bodies are not doing their job right. Who's the next BBQ Bill France Sr? This industry is ripe. It's in it's infancy. Find a need and...



I better shut up now,
Bill Anderson
Chatham Artillery BBQ Team

Stoke&Smoke
01-29-2010, 05:27 PM
I got that email too. Don't see his idea happening any time soon

Bbq Bubba
01-29-2010, 05:44 PM
I like it.

HBMTN
01-29-2010, 07:04 PM
I did a local contest this year where we were required to give out samples to the spectators. It made things 10 times more crazy than a normal contest. I won't do it again.

I have also looked into starting a contest and found how much expense is involved with sanctioning, orgaizing, insurance and more before you every get around to prize money. While I would like to see more money also, for the majority of contests across the nation it ain't gonna happen. I think if popularity gets big and there are 15-20 contests per year paying say $100,000 then once they get all of the teams they can handle and there are still teams trying to get in, that is when you will see entry fees go through the roof. Your team will need a major sponsor and the entry fee say $2000-$5000, you'll have a circuit of the TLC Pitmaster guy's and gal touring the nation like NASCAR

Buster Dog BBQ
01-29-2010, 07:28 PM
So what are the odds of getting a friends box at a judging table? Not great unless it is a small contest and if their box is so unique would it not be marking?

A 10 minute window of turn in seems fair especially when you have Friday to start.

Now I do see issues when judges give a 2 or 3 and the rest is all 8 or 9, but things happen like a dried piece of meat. Would be nice for
comment cards then.

Alexa RnQ
01-29-2010, 07:34 PM
Jesus. I think Bill needs a cookie.

BRBBQ
01-29-2010, 08:39 PM
He said. Don't use any judges - Say what, Bill? Have you lost your mind!? Nope... there's plenty of very well qualified people around barbecue contests to do the judging. Who? I'm talking about the contestants themselves! Who else is more qualified? Here's how you would go about it...((I have news for him, not all contestants cooking, cook good food, so how can they judge properly:!:). Sounds like he got a bad score, must not of been 100% kcbs judge certified:biggrin:

QansasjayhawQ
01-30-2010, 08:33 AM
"We all know that in the current system, turn-in boxes can easily sit around for an hour before being judged."

This comment tells me that this person's perspective is severely skewed from reality. From what I've seen, it's rarely two minutes from the time the sample box hits the turn in table until the table captain delivers it to the table.

Also - sure! It would be great to have more prize money to go for, if you have a shot at winning. There are competitors like me who do it for the fun. Sure, it's kind of a masochistic kind of fun . . . but I really enjoy the challenge!

I also realize that these contests are most often conducted as a benefit to raise money for charities. Hospitals, March of Dimes, animal shelters, the Sertomas raising money for the hearing impaired, Shriners causes . . . I've seen all kinds of deserving non-profit programs benefit from KCBS contests. So, when I shell out the $700 or so - and I think that's a fair estimation - I consider it all a DONATION to whatever the chosen cause is and go about my business knowing that my participation in my chosen hobby is having a larger effect on the world than simply an opportunity to win prize money.

Anyone competing strictly to win prize money is wacky anyway, from a strictly business accounting point of view . . . so give up the struggle, relax and have fun. Isn't that what it's all supposed to be about anyway?

ModelMaker
01-30-2010, 10:13 AM
wow! All I can say is wow!!
ModelMaker

BigJimsBBQ
01-30-2010, 10:49 AM
If Bill feels that strongly he should start his own Sanctioned BBQ Association.

Like every other BBQ competitor, I simply do my best and hope for the rest.

thenewguy
01-30-2010, 12:48 PM
Although...is he correct in that the producing of "pitmasters" there was no money given to kcbs to allow the filming being done. I DO think that they (producing co) should have,if they indeed didn't, give some money to the contest to help up the ante in the winnings department.
After all, it can be constued as an inconvenience when the camera guy is standing up in front of you at the awards ceremony so he can get the money shot from the guy they've been following all weekend...I'm just sayin'...

Mo-Dave
01-30-2010, 01:10 PM
Instead of having 8 judges, why not just use the now required 6 and through out the lowest and highest score with the remaining 4 scores you could use the highest 3 for scores and the 4th for the tie breaker if needed.

I don't agree with an hour wait for judges to get boxes but I have not done a huge contest like the AR so not sure about that.

I have seen the need for the rep to go out and get extra judges from the teams and don't think it has caused any problems. But with most teams the extra members are needed up to the last turn in so that could get a little tight time wise.

I am all for no garnish as long as everyone is doing the samething.
Dave

LindaM
01-30-2010, 05:14 PM
I don't agree with an hour wait for judges to get boxes but I have not done a huge contest like the AR so not sure about that.

Dave

How can one figure an hour wait to get to the tables when there is only 30 minutes between categories. Realize that there would be tables judging chicken and tables judging pork at the same time. IT DOESN'T HAPPEN, no matter how big the contest.

ique
01-30-2010, 05:19 PM
That made my brain hurt

Smokin' Gnome BBQ
01-30-2010, 05:35 PM
That made my brain hurt


gld Im not the only one....... and really how does a box sit for 1 hour when turn ins are 1/2 apart????

uziel5000
01-30-2010, 09:49 PM
Of all the stuff in there, the one about having 8 judges and dropping the higher and lowest score made sense to me for some reason. The whole system is built around trying to give fair average scores so doing it that way concentrates on averages even more.

Of course, I could be wrong.

Coz
01-30-2010, 09:54 PM
I have only judged 1 time but I bet all turn ins were on the tables no more then 6 or 7 minutes at the very most after the window closed.

Mo-Dave
01-31-2010, 01:30 AM
How can one figure an hour wait to get to the tables when there is only 30 minutes between categories. Realize that there would be tables judging chicken and tables judging pork at the same time. IT DOESN'T HAPPEN, no matter how big the contest.


I concur.
Davd

CivilWarBBQ
01-31-2010, 01:55 AM
I like Bill and Lee, and I'm glad they had such success last season. However, I disagree with most of what he's saying in that newsletter. I think it shows a lack of experience and understanding of the many issues that cooks don't get involved with.

A lot of folks who only have one or two pieces to the overall puzzle of what makes KCBS comps tick share his thought process, so he's not unique. To illustrate, take just a couple of his ideas and look at them realistically:

Have 8 judges for every table - There is a very good reason KCBS settled on 6 judges: that's what you can fit at a standard eight-foot table, leaving one side free for the table captain to work. Logistically, increasing to 8 judges would be a logistical nightmare.

Have the competing cooks act as judges - This is impossible for many reasons. First off, KCBS would go broke after the 3/4 of membership that is made up of judges stop sending in their dues. Then there is the fact that cooks who have been standing in smoke for 6-15 hours have had their palates saturated and are no longer able to judge accurately, even IF they aren't sick of the thought of BBQ by judging time. (What do YOU want to eat Saturday night of a contest?) And of course the logistics would again be a mess. How are cooks supposed to prepare their box for ribs if they are judging chicken? Bill's response is to hold entries in an oven for an hour, but then cooks would have no control over texture, and where would all these warming ovens (and the power to run them) magically come from?

Most of the rest of the proposals in the newsletter have the same weaknesses. In the end, I'd say like most of the mid-winter soapboxing, Bill's ideas are best left behind with the snow once BBQ season gears up again.

-gf

Jeff Hughes
01-31-2010, 12:06 PM
I cook alone often, I have zero interest in judging...

mobow
01-31-2010, 03:35 PM
I want some of what he is smoking. lol keith

Gadragonfly
02-01-2010, 12:11 AM
I can assure you, not many people besides barbecue competitors know how much work is involved in competing in a barbecue contest. Even some seasoned barbecue judges do not have a clue. Even some Master Certified Barbecue Judges don't have a clue...I struggle with this concept quite a bit. I hear it at almost every event and from a number of experienced table captains. "When judging keep in mind all of the work that is put into cooking for competition". First let me say that I do understand all of the work that is involved. My husband and I raced cars as a hobby and had a well prepared, winning race team so I understand what it takes to put together a winning competition team. But that said, why should that affect the scores I give an entry? Quite often I've had table captains that emphasize this amount of work and that entries deserve high scores for this reason. It seems to me that I should be judging the product for what it is, not the amount of perceived work that goes into preparation of the contest. It is suggested that a judge shouldn't give a score of 7 because at that point you've taken a team out of the contest. Thus you have scores all over the board. The newly trained judges are judging the way they are trained and then the more seasoned judges are judging with their hearts. It seems to me that a product which is outstanding deserves a score of 9 but a mediocre entry does not deserve a score of 7. This weekend I had a chicken entry that was bland. It didn't have any flavor profile at all but I was reluctant to give it a 6 - much less the 5 that it probably deserved. I really believe we should take the concept of "the amount of work going into an event" out of the judging arena and concentrate on the product itself. After all, almost every team puts in the same amount of time and preparation and if they don't it shows in their entry.

Signed,
A very conflicted judge :icon_smil

JimmyB
02-01-2010, 01:06 AM
I concur, the amount of work that goes into a BBQ contest should have no impact on your scores. The only thing that should be judged is the box that is turned in. Whether there is good weather or bad, whether the effort put in was high or low, and whether it is your first contest or your thousandth contest- the only thing that should be judges, or should be expected to be judged, is your final product. The team that handed in bland chicken and deserved a 5 or 6 didn't work harder than the team that turned in a great product and got a 9, or the team that did a decent job and deserved a 7.

I understand the thought, if you see your scores are 4's and 5's, you're probably not going to keep throwing money at competitions. But by saying everyone deserves higher scores, you actually punish those that do. A team that earns a 7 is thus put on even par with a team that earned a 5, and that's one thing that I'm confident isn't fair.

Finney
02-01-2010, 07:10 AM
Maybe this newsletter is why Lee isn't cooking with Bill this year. :shock:

Lake Dogs
02-01-2010, 08:20 AM
I like Bill and Lee, and I'm glad they had such success last season. However, I disagree with most of what he's saying in that newsletter. I think it shows a lack of experience and understanding of the many issues that cooks don't get involved with.

A lot of folks who only have one or two pieces to the overall puzzle of what makes KCBS comps tick share his thought process, so he's not unique. To illustrate, take just a couple of his ideas and look at them realistically:

Have 8 judges for every table - There is a very good reason KCBS settled on 6 judges: that's what you can fit at a standard eight-foot table, leaving one side free for the table captain to work. Logistically, increasing to 8 judges would be a logistical nightmare.

Have the competing cooks act as judges - This is impossible for many reasons. First off, KCBS would go broke after the 3/4 of membership that is made up of judges stop sending in their dues. Then there is the fact that cooks who have been standing in smoke for 6-15 hours have had their palates saturated and are no longer able to judge accurately, even IF they aren't sick of the thought of BBQ by judging time. (What do YOU want to eat Saturday night of a contest?) And of course the logistics would again be a mess. How are cooks supposed to prepare their box for ribs if they are judging chicken? Bill's response is to hold entries in an oven for an hour, but then cooks would have no control over texture, and where would all these warming ovens (and the power to run them) magically come from?

Most of the rest of the proposals in the newsletter have the same weaknesses. In the end, I'd say like most of the mid-winter soapboxing, Bill's ideas are best left behind with the snow once BBQ season gears up again.

-gf

MIM/MBN Certified Judge here. I now cook MBN & KCBS. First off, this
is certainly the time of year to discuss strengths and weaknesses of any
system and ideas should be entertained. To that I applaud. No system
is perfect; never will be. Judging will never be perfect. 80% of judges
are superior; the other 20% I could do without. (Did anyone pay attention
to the on-site judges in the BBQ PitMasters show? Recall they lady
judge chatting up the guy saying how wonderful it was, etc. Major
League NO NO. Supposed to be quiet, listen, and as stoic as possible. Anyway....) However, that's life, isn't it? 6 at a table, logistically, is
all you can really get there. Also, you'd now need 8 pieces vs. 6 (box
stuffing problems in some cases). I really like the 6 throw out the lowest
(KCBS style). MBN style is 5 judges, nothing gets tossed. That's it.

However, I do like MBN's scoring model vs. KCBS; 7 8 9 10 with a definition of what is a 7, what
is an 8, what is a 9, and what is a 10 vs. KCBS's 1 - 9. Numeric variance
(old math guy here) can be a disaster, and he's right about the more
experienced judges grading against everything they've ever had vs. what
they have presented before them today. That's another reason MBN's
scoring system is better, IMHO. 10 is the best you have today, regardless
of what you had in the past. I wont go into all the scoring, but it's
different, defined, and not nearly as much variance.

The having competitors judge themselves, believe it or not, in some
chili cookoffs (I wont go in to what a Pod cookoff is in CASI) we do
this, and frankly I like it!! HOWEVER, it's really not suited for BBQ at
all. In chili, we make 1 pot and go with it. In BBQ, there are multiple
events with multiple turn-ins, etc. In chili we dont stay up all night.
BBQ, some of us certainly DO. It's just not suited. Good idea, just
not practical and wont work IMHO.

ThomEmery
02-01-2010, 09:27 AM
He should cook a IBCA
It would give him much of
what he is asking for

weconway
02-01-2010, 02:36 PM
Have 8 judges for every table - There is a very good reason KCBS settled on 6 judges: that's what you can fit at a standard eight-foot table, leaving one side free for the table captain to work. Logistically, increasing to 8 judges would be a logistical nightmare.

Have the competing cooks act as judges - This is impossible for many reasons. First off, KCBS would go broke after the 3/4 of membership that is made up of judges stop sending in their dues. Then there is the fact that cooks who have been standing in smoke for 6-15 hours have had their palates saturated and are no longer able to judge accurately, even IF they aren't sick of the thought of BBQ by judging time. (What do YOU want to eat Saturday night of a contest?) And of course the logistics would again be a mess. How are cooks supposed to prepare their box for ribs if they are judging chicken? Bill's response is to hold entries in an oven for an hour, but then cooks would have no control over texture, and where would all these warming ovens (and the power to run them) magically come from?

First off, let me say that I just took my CBJ class and have no actual experience. Take what I say with a grain of salt...

Very thoughtful responses to his letter. I'm cautious to second-guess a process that has been carried out hundreds of times successfully.

That being said, I found it very confusing on how to judge based on one CBJ class. I am in no way saying my instruction was inadequate, just that the KCBS doesn't exactly do a good job telling you what the scoring system is supposed to mean.

I also especially didn't like the "take effort into account" mentality. That's just wrong. Should we congratulate mediocre BBQ because they tried hard? I plan on cooking competitively one day, and I'd much rather get a 4 and a bunch of meaningful comments than a 7 or 8 and think I'm doing it all right. My preference is to judge honestly and give good, constructive comments on the scores. Heck, we have a half hour to taste 6 entries, so there's no rush.