Comparitve Judging??
 

I would appreciate some thoughts on "Comparative Judging" as used by MIM/MBA and Best of the Best.

I am developing my own opinion, but would love to hear from some Judges with more experience in that process.

Thanks,

TIM

MIM/MBN certified judge here. MBN first judges the 3 categories (appearance, tenderness, and taste) on their own merits aside from any BBQ you've had in the past or any others on the table (same for onsite presentation and appearance). It's the overall (aka. final) impression score this is (and is designed to be) comparative. It's a good way, IMHO, to avoid ties (because any tie-breaker is tough for the loser of it) and rank order the ones you've judged. It, IMHO, provides a clearer picture of what's what. That way the top ones are definitely at the top, even to teams. I enjoyed it
a few weeks ago competing in an MBN comp, where we got mostly 10's and a few 9's
at the table, then 4 9.9's and a 9.8 overall impression. It told me that we were a
very close 2nd to someone at that table (who ended up GCing the comp). I actually
enjoy knowing we were a close 2nd to someone (at the table) who GC'd at this particular comp.

I like it from both a competitors perspective and a judges perspective.

I went to an MBN event this weekend and my only issue was the time factor. 99% of the crowd and 30-40% of the competitors had left by the time awards took place.

I judge in GBA which also uses the comparative judging method. GBA's system is a little bit different than MBN. There is no on-site judging. Its all blind box & the top 3 in each category make finals & prepare a second blind box for judging.

The GBA system is totally comparative. You judge each entry in appearance, texture, taste & overall impression against the other entries on your table.

Appearance, texture and taste are scored on a 7 - 10 scale (whole numbers only) You have to award at least one 10 in each criteria. You can award multiple 10's. For example, of the 5 boxes on the table, the appearance scores can be 10, 10, 9, 9, 8. They could be all 10's. If you give a 7, you must include a comment on the score sheet.

Overall impression is the tiebreaker. It is scored from 9.0 - 10. Each entry much receive a unique score with your top choice receiving a 10. Scoring for overall impression could be 10, 9.9, 9.7, 9.5, 9.3.

Each criteria has a multiplier to arrive at the final score. Taste has the biggest multiplier with appearance being the smallest.

IMHO, it is a very fair system. You judge each entry on it merits against the other entries on your table that day. Not against what you tasted 2 weeks ago or against some subjective standards.

Comparative judging has no direction. Each "contest" is unique but doesn't promote advancement of the art because the entries vie with each other, not a standard. In a worst case scenario a well-connected team could avoid competitions with known good teams and attend only competitions with inexperienced teams. Non-comparative judging is the only way to progress -- figure skating is a great example of how discipline in judging has advanced the sport.

On the surface I like the idea of comparative judging - your stuff is compared to all the other cooks and the best product on that day wins. This works great at chili cookoffs where every judge tastes every entry.

The problem is that can't be done at BBQ contests because of the sheer volume of food at an event of any size. So now you have to break the field into smaller groups to fit them all on tables of a smaller group of judges. In addition to the X factor introduced by different judges scoring some entries than others (which is unavoidable), you are adding a Y factor of which teams land on your table.

Consider a scenario where your entry is compared against Jack's, Yazoo & Red Hot vs a handful of novice local teams. Obviously these are two wildly different levels of competition.

The two-tier judging helps to even things out by pitting the winner from the "easy" table against the winner from the "hard" table, but the team who never made the finals because they were unlucky on the blind box tables even though they had great product still gets shorted.

In the end, ALL judging systems are flawed because they all have to deal with the problems of the subjective opinions and limited eating capacity of judges. I don't think MBN is any better or worse than other systems; what is most important is that all participants understand the system completely and that rules are enforced equally and consistently.

Let me see if I understand this so far.
Lets use a contest of 60 teams which would be 10 tables, right?
And, one shot at blind judging for a final score.

Now, I as a judge would get 6 samples to rank.
Appearance should not be difficult. Probably one that really stands out, one or two that are not appealing and the balance are in the middle.
So, I guess I assign a value to the ones in the middle on a random mental coin toss if they all look of equal quality.
Am I right so far?

Now, for taste and tenderness, it is obvious that I must sample each entry and then rank order both taste and tenderness.
I guess one or two might stand out for taste and one or two might be lacking in taste. So, once again I do a mental coin toss to rank the entries that are not solidly on one end or the other.

Then I still need to rank order the entries for tenderness using the same process. But, the best and the worst are probably not the same as they ranked for taste. So, I have to remember the relative tenderness of all 6 entries so I can accurately rank them.

I am thinking that I am not smart enough to keep all of this straight in my mind. I am not sure that I could really rank all of them fairly since that is a lot of information to process.

Am I missing something here??

Thanks for the input so far.

TIM

Let me see if I understand this so far.

Lets use a contest of 60 teams which would be 10 tables, right?
And, one shot at blind judging for a final score.

Now, I as a judge would get 6 samples to rank.
Appearance should not be difficult. Probably one that really stands out, one or two that are not appealing and the balance are in the middle.
So, I guess I assign a value to the ones in the middle on a random mental coin toss if they all look of equal quality.
Am I right so far?

No, appearance is judged on it's own merit. All 6 could get 10's (using
the MBN scoring model) if they were beautiful/perfect.


Now, for taste and tenderness, it is obvious that I must sample each entry and then rank order both taste and tenderness.

Not done that way either. Again, if all taste perfect, all could get 10's.
Or, if they dont taste good they could all get 8's or 7's...

I guess one or two might stand out for taste and one or two might be lacking in taste. So, once again I do a mental coin toss to rank the entries that are not solidly on one end or the other.

<see above>

Then I still need to rank order the entries for tenderness using the same process. But, the best and the worst are probably not the same as they ranked for taste. So, I have to remember the relative tenderness of all 6 entries so I can accurately rank them.

<see above> If all are perfect they all get 10's. This part isn't
comparative.


I am thinking that I am not smart enough to keep all of this straight in my mind. I am not sure that I could really rank all of them fairly since that is a lot of information to process.

Am I missing something here??

Yep. They all get judged on their own merits on appearance, tenderness,
and taste (not comparative). If you had an unbelievable table and all
were perfect, all could get straight 10's (never seen it, but I have been
at a table where we had 3 entries get straight 10's this far from every judge).

You're missing the final overall impression score. This is the part that's
comparative. One will get the 10. The rest are rank ordered. If they're
all extremely close the scores will be 10, 9.9, 9.8, 9.7, 9.6...

Thanks Hance,

So, it appears that the blind box portion of MIM is just like KCBS/FBA.
The judge evaluates the entry based on their opinion as to how it fits into their perception of how that entry should be scored-- without reguard to the rest of the entries before them.

The actual system I am trying to figure out is the Best of the Best comparative judging. It is described as pure comparative judging, which would be more like what I described above. And, there is only one judging, not a finals table or top 3 finals. One shot and it is done.

I am trying to contact a couple of the BotB judges to see if my understanding is correct, or if I am off base.

Any BotB judges here today???

TIM

The only information on BotB judging that I can find is posted here:
http://www.nationalbbqfestival.com/_...EN%20Rules.pdf

Item #13

I can't copy/paste cause it is a .pdf file.

TIM

Actually, all categories in MBN are comparative, and they are all assigned differing multipliers to express their relative importance in winning BBQ. Area/Personal appearance and Presentation are the two categories that most non-MBN cooks worry about most, but they have the lowest multiplier. Also, most judges give 10's in these categories as long as your area is clean and you give them some kind of story about your cooking process.

The other criteria are comparative; however that does not mean you must pick a best and then score down from there. A sweet rib that is done well can score a 10 in flavor just as a spicy rib on the same table can receive a 10 from the same judge.

A judge may NOT duplicate a score in overall impression. But typically they assign a 10 for their best product, then a 9.9, and so forth. Many judges mistakenly believe that this is a huge difference in scoring, but the overall impression is multiplied by 3 while flavor is multiplied by 6.

No system is perfect, however the Memphis scoring system in my opinion allows for a good product to withstand a "bad table" or low scoring judge a little more than the KCBS system.

As far as the comments about comparative judging not advancing the sport, I would totally disagree. When is the last time you had a judge give you some feedback in KCBS? Very common for judges to come back and talk to the teams they judged in MBN- especially new teams- and give them pointers and feedback. Also, breaking down the score into more criteria helps pinpoint deficiencies in your product quicker.

Hope this helps

Kapn, let me walk you thru my process of judging & how I keep things straight in my mind as you say.

In GBA, most tables have 5 entries per table. All 5 boxes are placed on the table & opened. I inspect them all for appearance & score each box. It is not unusual that I will have 3, 4 or even 5 10's in appearance. I rarely see a box that justifies an 8 in appearance. I have never seen a 7.

After all the judges at my table have marked the appearance scores, I take a sample from each box & place it in the appropriately numbered section on my platter. I will poke, pull & bite the first sample for texture/tenderness. I make notes along side each section of my platter as I am sampling. For example, if the first rib gives me a perfect bite make with just a bit of tug and the bone is clear & white, I give it 2 check marks for tenderness. If it is just slightly over or under cooked, it gets 1 check. If it is way over or under cooked, I will mark an X. For taste, I mark down 1 to 3 plus signs. Now I go back to my score sheet & record the scores for texture & taste on all 5 entries. If all the samples on my platter had 2 check marks, they all get 10s for texture. 3 plus signs = a 10 in taste. 2 would be a 9.

Now I have to rank the five samples from best to least for overall impression. I start with the samples that have 3 plus signs & taste each one again. When I decide which one I like best, I mark that one a 10 on my score sheet. Now is the toughest part for me. If the second best is very, very close to my favorite, it gets a 9.9. If there was a significant different IMHO, it may get a 9.8 or even a 9.7. I continue working down from there. It is not mandatory that you score overall impression 10, 9.9, 9.8, 9.7, etc. You can score 10, 9.8, 9.5, 9.4, 9.1.

When judges score the overall properly, it does not penalize a box for being on the same table with another strong entry. In that case, both boxes with probably get all 10s in appearance, texture & taste. One box will get three 10s & two 9.9s in overall. The other will get two 10s & three 9.9s. Both those teams will more than likely be in the top three scores & advance to the finals.

Warthog (and all).

Great explanation about GBA procedures.

I have sent an e-mail request to the Phelps to find out how they instruct the judges for BotB since they use "blended and merged" rules from many sanctioning bodies.

Really curious about some things.

Thanks a million.

TIM

I just got off the phone with a BotB/FBA/KCBS/MIM/GBA judge.

The process I described in post #7 above is the one used by Best of the Best.
And, the judges are "encouraged" to score high and give the best on the table a "100" or very close to it.
No duplicate scores allowed in Taste, Tenderness, and Texture.

Now, I can make an informed decision about attending the event next year.

A big THANKS to all who posted the various rules.

TIM