Judging Parity

[Alexa RnQ]

IMO CBJs are not necessary

And in my opinion, they are more than "necessary" -- they are the reason I will go to a particular contest, or not.

Yes, good BBQ will rise to the top. We've been fortunate enough to cook well across two sanctioning bodies this past year. But the fact remains that a team must cook differently to please "judges" from the general public who think boiled Chili's and mushy McRibs are the bomb, than they do for a Master CBJ.

Contests/organizers that make lackluster or nonexistent efforts to recruit experienced judges will not see me return. Contests that rely on holding a judging school the night before a contest to make up their "100% CBJs" will not be at the top of my list of places I want to compete again.

Too much goes into our product to not care whether any attempt has been made to educate the judges to a common standard. Do teams deserve something more? Yes, and that's why we value experienced judges so highly. Do I think it could be better than that? Absolutely, through apprenticeship and continuing education.

To become a Certified Beer Judge you have to take a series of classes and then pass a difficult test. Plenty of people are willing to go through the hassle of getting certified. And the reality is that it costs less than $100 to brew a beer that will be judged. It can cost several hundreds to participate in a BBQ contest. Doesn't it make sense that people who spend that kind of money have judges who have REAL training?

This is tremendously interesting, to hear how other sanctioning bodies train judges. Following that model, wouldn't it make sense to have a class entirely on each of the four meats?

There's a HUGE amount of information a prospective new BBQ judge has to absorb, so many variables in how the meat can present, and it's just human nature that nobody's going to sponge it all up and apply it correctly in one go. It seems like either splitting up the category classes, or establishing continuing education, would make sense.

 

[monty3777]

There's a HUGE amount of information a prospective new BBQ judge has to absorb, so many variables in how the meat can present, and it's just human nature that nobody's going to sponge it all up and apply it correctly in one go. It seems like either splitting up the category classes, or establishing continuing education, would make sense.

At the VERY LEAST let this level of training be the criteria for being a Master Judge. What is the criteria right now? I think it is the number of contests judged - though I may be wrong.

Here's a link to the Certified Beer Judge program, for comparison. http://www.bjcp.org/index.php

 

[Alexa RnQ]

Thirty contests, plus cooking with a team -- invaluable! -- and I'm not certain of the testing involved. In any event, it's not a cakewalk.

 

[BigJimsBBQ]

I have cooked in a competition and was scored the lowest every in the chicken category, with turn-ins that I thought were my best ever. Thank goodness the FBA shows the tables by numbers, and the judges at that table by numbers (Table 1, Judge 1, 2 3 4 5 & 6). When I looked at the Table and judges score that judge my chicken, I looked at the other 5 Teams that were judged for the chicken at that table and found out I was the highest scores of the 6 of us. 7 was the highest given in all 3 categories and saw several 5 and majority 6s. I then followed this Table in the other 3 categories and they were consistantly the lowest scorers in all 4 categories.

If any one thinks a low scoring table would not cause a consistant top 5 Team be in the bottom with same product, wake up. The majority of the time it is fair, but there is always that table of judges at a event that are all very low scorers and piss your $1K down the drain for you.

I respect Judges, Organizers very much. I just expect the same respect back as a very serious competitor that is there to win fairly.

 

[landarc]

Getting certified to be a BJCP judge is extremely difficult, and you have to be able to identify all of the recognized styles of beers, by taste and description. A lot more than 4 meats, for certain. This does not make the judging any less controversial when someone gets lower scores than they expect. The one KCBS judge I know locally has quit, as he became fed up with the lack of thought and carefree attitudes he saw as a judge.

 

[Rich Parker]

My feeling is that the difference between a judge who took a class and one that hasn't is not significant. The criteria to be a BBQ judge is ridiculous. One day where you are taught how to fill out a form and told that ribs shouldn't fall off the bone - that's hardly training. To become a Certified Beer Judge you have to take a series of classes and then pass a difficult test. Plenty of people are willing to go through the hassle of getting certified. And the reality is that it costs less than $100 to brew a beer that will be judged. It can cost several hundreds to participate in a BBQ contest. Doesn't it make sense that people who spend that kind of money have judges who have REAL training?

BYW, I'm judge # (oh well, I forgot)

I was told that comps already have a hard time getting 100% CBJ participation so wouldn't more classes and more money make that even harder?

 

[QansasjayhawQ]

#1 the KCBS has no control over selection of judges. The Jack is a good example. Good cooks still win with celebs. If we tell the organizer they need to have all CBJ's then the opranizer says I'll run it without you unless you provide the judges I need. Not going to happen.

#2 there are a lot of cbj's out there myself included that have not judged a contest in years but we're still cbj's. Doesn't make us better judges.

#3 I know a lot of cbj's that smoke as in cigars, etc. Now I would far prefer a non smoking non certified judge to taste my entry. Not cutting at smokers here as I enjoy cigars at contests but fdact is after a couple of cigars my taste buds are messed up. That's why I like having a program and sticking to it every time. I don't need to taste for the best slab of ribs.

#4 new cbj's that took the class the night before are often tough scorers. But others are not. there's no formula to apply here

I've also thought that there's another way that judges could be tracked.

If a judge volunteers to judge - and then doesn't show up - that should be kept track of so that organizers have an idea of how many judges are likely to show up.

Tracking a judge to see if it's been a couple of years since they judged would be another way to ensure that the pool of judges is better. If a judge hasn't judged in a couple of years, they should have to take a refresher course, or take the judging class all over again.

 

[QansasjayhawQ]

Thirty contests, plus cooking with a team -- invaluable! -- and I'm not certain of the testing involved. In any event, it's not a cakewalk.

Yes - and there is a written test. I haven't seen the test - only heard about it from some other people judging last year as they hit their Master level.

I can say that the skills of a judge that has judged less than about 10 contests are radically different than they are of a judge who has judged several more.

I think that judges are necessary and that it's a good system of judging that we (the KCBS) has created. However, I also see a HUGE number of opportunities for improvement and refinement.

Thanks to everyone who has thrown in their comments here - I really appreciate them.

P.S. The efforts that I see to improve the judging are mostly coming from the serious, experienced judges.

 

[CajunSmoker]

I personally would prefer my box to hit a table of newer judges than one with supposedly "experienced judges" who think they know everything there is to know about BBQ. Some of the opinions I've heard in the judges tent make me cringe when I turn a box in. One that sticks out in my mind was from a judge with 20+ contests who said before the brisket turn ins started "Well it's time for brisket next. I just hate brisket":!:

 

[BigJimsBBQ]

I personally would prefer my box to hit a table of newer judges than one with supposedly "experienced judges" who think they know everything there is to know about BBQ. Some of the opinions I've heard in the judges tent make me cringe when I turn a box in. One that sticks out in my mind was from a judge with 20+ contests who said before the brisket turn ins started "Well it's time for brisket next. I just hate brisket":!:

Roger - I 2nd that Opinion.

 

[QansasjayhawQ]

I personally would prefer my box to hit a table of newer judges than one with supposedly "experienced judges" who think they know everything there is to know about BBQ. Some of the opinions I've heard in the judges tent make me cringe when I turn a box in. One that sticks out in my mind was from a judge with 20+ contests who said before the brisket turn ins started "Well it's time for brisket next. I just hate brisket":!:

Well, that's one judge . . . that doesn't mean that ALL judges are that way.

I can tell you that the vast majority of judges that I know are open minded and certainly not picky about what they like or don't like. Most of the judges I know are aware that, one day, pork with white sauce may hit their table and they have to be ready to judge that entry on its own merits - while keeping their own preferences for KC style (or whatever) in check.

Heck, I LOVE hot, spicy foods! But if someone has created an excellent work of BBQ art with no heat at all in it, I will NOT score it down. I am judging that BBQ on its own merits - not comparing it to what I would personally prefer.

Good judges practice objectivity.

Non-certified and inexperienced judges lack that practice of objectivity and it seems like there should be a way to accommodate for that.

 

[monty3777]

I was told that comps already have a hard time getting 100% CBJ participation so wouldn't more classes and more money make that even harder?

I suppose so - but with all that work there are tons of people who want to put forth the effort to be beer judges. And because of that training the beer judges are taken very seriously in the comp beer world!

The other big difference is that each entry is given comments by the judge that help the competetor better understand scores. That just seems logical. I think the reason that doesn't happen in KCBS is that cooks would explode if they got a 7 on taste because "it ain't what I like." Better to leave it all a mystery.

 

[monty3777]

case in point: (this is a link to another forum. Please come right back!!

http://www.rbjb.com/rbjb/rbjbboard/messages/730928.html

 

[ZILLA]

And in my opinion, they are more than "necessary" -- they are the reason I will go to a particular contest, or not.

Yes, good BBQ will rise to the top. We've been fortunate enough to cook well across two sanctioning bodies this past year. But the fact remains that a team must cook differently to please "judges" from the general public who think boiled Chili's and mushy McRibs are the bomb, than they do for a Master CBJ.

Contests/organizers that make lackluster or nonexistent efforts to recruit experienced judges will not see me return. Contests that rely on holding a judging school the night before a contest to make up their "100% CBJs" will not be at the top of my list of places I want to compete again.

Too much goes into our product to not care whether any attempt has been made to educate the judges to a common standard. Do teams deserve something more? Yes, and that's why we value experienced judges so highly. Do I think it could be better than that? Absolutely, through apprenticeship and continuing education.

I'd rather have my BBQ judged like a Tennis game rather than like an Olympic Ice Skating routine. With judges that call the boundrys and fouls and that's it! The other thread sheds light on what I mean.

 

[QansasjayhawQ]

I suppose so - but with all that work there are tons of people who want to put forth the effort to be beer judges. And because of that training the beer judges are taken very seriously in the comp beer world!

The other big difference is that each entry is given comments by the judge that help the competetor better understand scores. That just seems logical. I think the reason that doesn't happen in KCBS is that cooks would explode if they got a 7 on taste because "it ain't what I like." Better to leave it all a mystery.

In 2009 the KCBS introduced Comment Cards on each table for just such an occasion.

The judges have been instructed all year long to NOT leave vague, general comments such as "it ain't what I like". The comments on the cards are to be specific comments on the details with the intent that the comments are to help the teams improve their art.

The comment cards are separate from the scoring card.

I liked the fact that we started out leaving comments on the back of the scoring cards but that those comments were separated out this year. I hope the KCBS continues this.

 

[Falcon 83]

This is my first post or comment so forgive mistakes and I hope I don't offend anyone.

I just became a KCBS certified master judge. The requirements are judging 30 KCBS sanctioned events, cook with a team and pass a 51 question written test with a score of 90% or better. Some of the questions are true/false, some are multiple choice and some are essay. Believe me its not easy.

I'm also a certified table captain. Part of my duties are to check each score sheet at my table for completeness before giving them to the rep. I can also see the scores from the various judges. Relative to the discussion about new judges certified the night before, I can honestly say from my experience that some score hard, some score in the middle and some score easy. The same can be said for master judges.

Previous comments above relative to some kind of parity would seem to me to be labor and cost prohibitive. KCBS would have to pass these costs on to the organizers and they would add these to entry fees for the cooking teams. Judges do receive recurrent training at each competition as at the judges meeting, we listen to a 10 minute CD recapping the scoring system and the judging particulars for each catagory. If judges are required to much more training periodically, they're going to want to be paid; again adding to your entry fee. This would bring in judges who are only in it for the money. At least now, the volunteer judges are there for the love of BBQ.

I have two suggestions to keep things simple for the time being until some more elaborate system can be devised and acceptable to all parties involved.

When judges are walking around the comp area after the close of scoring, invite them in, explain your procedures, answer their questions and invite them to cook with you at a future comp, even if they are not ready for master judge certification. Each little bit of training will help. I know I'm a better judge after cooking with a team.

Second, the comment slip program is already in service, although it is voluntary on the judges part. I try to submit a slip for any score 5 or below; my philosophy being if I can help you make better BBQ, it will benefit my stomach at a future comp.

The rules committe is meeting in Phila. next month. Get your e-mails into them asking for mandatory comment slips for a score of 5 or below. At least then, if a judge gives a low score, you'll know why. This pressure is going to have to come from the cooking teams; volunteer judges are not going to ask for more work.

I've sai enough for now. Thanks.

 

[Rich Parker]

I suppose so - but with all that work there are tons of people who want to put forth the effort to be beer judges. And because of that training the beer judges are taken very seriously in the comp beer world!

The other big difference is that each entry is given comments by the judge that help the competetor better understand scores. That just seems logical. I think the reason that doesn't happen in KCBS is that cooks would explode if they got a 7 on taste because "it ain't what I like." Better to leave it all a mystery.

Don't get me wrong - I think it would be awesome if they offered that type of training for judges so then you would only have qualified judges. I just think KCBS is to big to go back and change that.

 

[QansasjayhawQ]

This is my first post or comment so forgive mistakes and I hope I don't offend anyone.

I just became a KCBS certified master judge. The requirements are judging 30 KCBS sanctioned events, cook with a team and pass a 51 question written test with a score of 90% or better. Some of the questions are true/false, some are multiple choice and some are essay. Believe me its not easy.

I'm also a certified table captain. Part of my duties are to check each score sheet at my table for completeness before giving them to the rep. I can also see the scores from the various judges. Relative to the discussion about new judges certified the night before, I can honestly say from my experience that some score hard, some score in the middle and some score easy. The same can be said for master judges.

Previous comments above relative to some kind of parity would seem to me to be labor and cost prohibitive. KCBS would have to pass these costs on to the organizers and they would add these to entry fees for the cooking teams. Judges do receive recurrent training at each competition as at the judges meeting, we listen to a 10 minute CD recapping the scoring system and the judging particulars for each catagory. If judges are required to much more training periodically, they're going to want to be paid; again adding to your entry fee. This would bring in judges who are only in it for the money. At least now, the volunteer judges are there for the love of BBQ.

I have two suggestions to keep things simple for the time being until some more elaborate system can be devised and acceptable to all parties involved.

When judges are walking around the comp area after the close of scoring, invite them in, explain your procedures, answer their questions and invite them to cook with you at a future comp, even if they are not ready for master judge certification. Each little bit of training will help. I know I'm a better judge after cooking with a team.

Second, the comment slip program is already in service, although it is voluntary on the judges part. I try to submit a slip for any score 5 or below; my philosophy being if I can help you make better BBQ, it will benefit my stomach at a future comp.

The rules committe is meeting in Phila. next month. Get your e-mails into them asking for mandatory comment slips for a score of 5 or below. At least then, if a judge gives a low score, you'll know why. This pressure is going to have to come from the cooking teams; volunteer judges are not going to ask for more work.

I've sai enough for now. Thanks.

Welcome to the friendliest most helpful forum I've ever been a part of on the Internet, Falcon. Please be sure to stop by Cattle Call and introduce yourself.

How do you envision parity values being assigned to a judge as being cost prohibitive?

If I were required to take an on-going recertification class, that would be fine with me. Heck, it might even cull out the recreational judges. Myself and many other CBJs consider ourselves professional and take our volunteer responsibilities very seriously. It's true that we do get that ten minute reminder before we start, but I think that the most training comes from actually judging. Those who have judged less than 10 contests have far less of a clue as to what is going on than those who have judged 20 or 30.

I think that it's not fair to the cooks (which includes myself - I also cook on two teams now) that a rookie judge's score counts for just as much as a Master judge's score. I've been around these judges and I can say that the experienced judges who are professional about their duties score much more accurately and fairly than most rookie judges. That's why I started the discussion about judging parity.

I do like your idea about a comment card being required for any score 5 or below. There are a few teams out there that need the help and if I ever hit a 5 in my results, I would REALLY like to know what I could do to improve it.

Thanks for your comments!

 

[Falcon 83]

Hello QansasjayhawQ,
I think the cost for the parity of judges would have to come from the time and research to establish the program, keeping track of each judge's value, and plugging it into the scores for each competition. Each competition would have a different field of judges so it would take extra time for the Rep to establish the computer scoring program for a particular contest. Last minute substitution of a few judges would also add to the Rep's pre-comp preparation. Regional records couldn't be kept and used as I have judged comps in 10 different state ranging from New York to North Carolina and two trips to Kansas for The Great American comp.

Then there would be the upgrading of each judge's value as he judge's more contests. Can anyone say that two judges who have judged 20 contests have the same parity value? Does a judge who judges 10 comps in 5 years deserve the same value as a judge who does 10 comps in one year? Does a newer master judge deserve a higher value than an older master judge who attained this position before the rules were added about cooking with a team and passing a written test? This list of situations could go on forever. Evaluation tests would have to be given on a regular basis to determine if a judge deserves and receives a higher value.

In closing this part of my response to your honest question, I think KCBS would have to hire more staff and supply another Rep or two to each comp for the added paperwork.

And finally, I do take my judging seriously, as seriously as your comments imply you do. Yes, there are judges I would like to see "fired," especially those filling their take home cooler while I'm still scoring my 3rd sample (my personal pet peeve). I have asked discrete questions of several Reps about just this subject but there is no established procedure for firing a judge unless it is for serious rule infractions, etc.

Somebody is going to have to build the better mouse trap. Until then, I'll judge the best I can and enjoy the BBQ.