New KCBS Scoring Program Test

[Sauced!]

I saw this posted by Pellet Envy on Facebook and didnt see a thread on here so I figured I would share. I like the idea. I know FBA does this and I think KCBS would benefit from it.

Here is the comment Rod posted along with a couple of pics of the scoresheets

"At this past weekend's KCBS sanctioned contest in Quapaw, Oklahoma, the new scoring system was unveiled for beta testing on an actual contest and teams received print-outs from the new program along side the traditional score sheets.

Some interesting data at first glance. Table numbers of where your food was judged are shown for all teams and all categories, so you can see the teams on your table. Scores by judge for each category are listed, along with each of those judges average score, so you can see how the score you received compares to the other scores that judge gave on that day. With some simple calculations, you can also prove the "good table, bad table" theory. In this case, 23 of the top 40 places were awarded by 3 tables, while only 6 of the top forty places were awarded by 4 tables. Translation: If you his tables 1 or 10 this weekend, odds were strongly against you winning this contest."

Overall
http://i271.photobucket.com/albums/jj155/FultonPub/294444_10151191287420923_255402537_n.jpg

Team
http://i271.photobucket.com/albums/jj155/FultonPub/538930_10151191293245923_962904238_n.jpg

Chicken
http://i271.photobucket.com/albums/jj155/FultonPub/377232_10151191286860923_1583624169_n.jpg

Ribs
http://i271.photobucket.com/albums/jj155/FultonPub/377232_10151191286860923_1583624169_n-1.jpg

Pork
http://i271.photobucket.com/albums/jj155/FultonPub/304390_10151191287015923_1794621823_n.jpg

Brisket
http://i271.photobucket.com/albums/jj155/FultonPub/523246_10151191287240923_1977830575_n.jpg

[Sledneck]

About friggin time

[pat]

Looks like a great improvement...if it happens.

[Fatback Joe]

I like it.

That (and no garnish :biggrin1:) was one thing I really liked about FBA over KCBS. Nice to know if you won your table.

[Sauced!]

Funny part is a BOD member posted a comment saying the BOD had no knowledge of this test being done and that they had not even seen the program yet.....Once again another example of the dysfunction of the BOD.

[Teamfour]

Preface: I don't compete yet so take my comments with that in mind.

All this does is provide data that throws a now proven variable into your cook. While you may have suspected poor judging did you in, now you know. So, do you adjust your cook profile for next comp, or leave it the same since the judging was most likely suspect?

I hope KCBS would use the data to identify judges who may need further training. Personally, I think some teams may use the data to hunt down the judges and "retrain" them personally. :wink:

[carlyle]

At first glance, this looks like a good improvement.

Rather than bash the BOD, give them credit for sticking to this project even after it became more difficult that it first appeared.

As an organizer and judge chair, I am really excited about getting this new detailed information to evaluate the job my judges are doing. I can see using the data along with assigned seating in advance of the following years contest as an aid in deploying the returning judging resources evenly to avoid "good table/ bad table".

As a judge, getting feedback with data, will help evaluate my performance relative to the rest of the table more concretely than by just talking between categories.

Anxious for the program to be rolled out.

[carlyle]

Teamfour,

I hope the program will be blinded to preserve judge anonymity the same way

renumbering turn in boxes keeps judges from knowing what team they are judging.

Retaliation and confrontation would be bad for bbq and for KCBS.

[SCSmoke]

I for one really like it. I hope it gets implemented.

[Teamfour]

Teamfour,

I hope the program will be blinded to preserve judge anonymity the same way

renumbering turn in boxes keeps judges from knowing what team they are judging.

Retaliation and confrontation would be bad for bbq and for KCBS.

That part of my post was sincerely tongue-in-cheek. Honestly, I love data and hope this implemented. My point is teams will have to make even more educated decisions on what changes to make based on scores.

[tmcmaster]

Funny part is a BOD member posted a comment saying the BOD had no knowledge of this test being done and that they had not even seen the program yet.....Once again another example of the dysfunction of the BOD.
The reason (well, ONE of) I am no longer a member of KCBS.:mad2:

Retaliation and confrontation would be bad for bbq and for KCBS.
We all may joke about that sort of thing, but I really can NOT see anyone that I have EVER met in BBQ actually doing anything like that.

[Big Mike]

I like the Team Detail Report. I like the fact that I can see each judges avg score and who was at your table. The only thing that confuses me is the "Rank" column. What is that supposed to show. It is obviously not his rank at that table. Anybody have an idea

[Bourbon Barrel BBQ]

As much as I love the feedback I see this as creating a serious amount of complaints to reps and organizers. Its like eating at a real dive of a restaurant. Sometimes you just don't want to see whats going on behind the scenes. Im hoping for the best though.

[Lake Dogs]

As much as I love the feedback I see this as creating a serious amount of complaints to reps and organizers. Its like eating at a real dive of a restaurant. Sometimes you just don't want to see whats going on behind the scenes. Im hoping for the best though.

Many of us with years and years of experience in cookoffs (not just BBQ) have known this and some of us have preached and tried to inform teams of this. There is a luck factor involved. Even if you train and train and get very well "educated" judges, group a few together with the same tendencies and same biases (we all have them; we're human) and the same types of preferences, and what you'll get are "good" tables and "bad" tables. It happens. I'm sure there's some scientific law here that applies.

For this reason, MANY sanctioning bodies force tables to change judges (mix up the mix) every category. You'll still end up with good and bad tables, but they've shifted and mixed.

There's still the luck of who else's BBQ landed on your table. Trust me, even if you judge each on their own merits, a judge, after tasting 3 or 4 fairly average or below average entries gets one good one, watch that score. It'll go WAY up. Deserved that high? Possibly; possibly not. Same on the other side, if yours is 4th in their tasting and they've had 3 in front of you that were off-the-chart awesome and yours is then very good too, the odds of getting that perfect score are fairly low.

It happens. We're dealing with humans.

For these reasons, many/most seasoned competitors will counsel new competitors to not read too much into any 1 competition scores, and dont change recipes, etc. until you've had 3 or 4 or 5 competitions saying the same thing. THEN and only then should you consider making changes.

Kudos to KCBS for showing this (like many others have done for a while).

[TooSaucedToPork]

Love it. It solves the age old problem of guessing what went wrong. It let's us zero in on and focus on improving our product. By seeing who was against you, you can tell what judges like and don't like. That is...If this actually makes it out of beta without politics breaking it down into a useless program.

[mobow]

I like the Team Detail Report. I like the fact that I can see each judges avg score and who was at your table. The only thing that confuses me is the "Rank" column. What is that supposed to show. It is obviously not his rank at that table. Anybody have an idea

It is what place he came in over all. keith

[Carnivorous Endeavors BBQ]

I wonder how this would work out if all certified judges scores were kept and averaged over a period of time or several contests. Judges could then be assigned to tables so that all tables have near the same cumulative average for scoring. If all tables had a close to the same average in scoring, it would seemingly(on the surface) start a contest with a more level playing field. This way you shouldn't get 1 table who likes to score all 9's for practically everything. Just a thought....

[MAP]

Look at Judge 3 on table 2. Either he is a low scorer or the others are High scorers.

[ModelMaker]

The reason (well, ONE of) I am no longer a member of KCBS.:mad2:


We all may joke about that sort of thing, but I really can NOT see anyone that I have EVER met in BBQ actually doing anything like that.


There is a commitee in charge of designing, debugging, testing, and implementing the new KCBSCORE program. It is headed by BOD member Mike Budai who when each step is accomplished then makes a monthly report to the entire BOD at the monthly meeting.
I don't find it suprising that the entire BOD was not aware of a beta test of the new scoring system as Mr. Budai was just doing his job getting the system up and running.
I'm very certain the board will be fully informed of progress at the next BOD meeting.
Don't be all hating at KCBS for doing what we ask of them.
Ed

[djqualls]

I cooked this contest and we were told at the cooks meeting that it was going to be used side by side the original system and then used as a comparison as well. They answered any questions by the teams and then added the results along with the original (Official) to our sheets at awards.

I really thought it was a good report and the awards came off early so I suppose the software worked goof. There were 44 teams in this contest soI'm guessing it was stable since we got a copy and awards were early. I did hear Mike thanking Dennis Polson who was there to judge for helping him with the data that day.

[Grizmt]

I found the info most interesting, really drills down to almost the bottom of the available data.
As always when getting this much detail depending on how it's used will determine whether it'll be a good thing or a bad thing in the long run.

Personally I've always liked the idea of mixing things up in the tent by moving TC's around as well as shuffling judges since it matters little if you're at the same table for any given turn. I've also been a big proponent of whomever is working the entry table shuffling the boxes after they've been renumbered before they're brought in the tent to the TC's to better assure truly blind judging.
As for the luck thing, there's always going to be luck involved when it gets into the tent, there's no real way to avoid it unless you clone all of the judges from one person but by shuffling everything I'd guess the luck factor (good table/bad table) could be reduced a bunch.

[dhuffjr]

I like the Team Detail Report. I like the fact that I can see each judges avg score and who was at your table. The only thing that confuses me is the "Rank" column. What is that supposed to show. It is obviously not his rank at that table. Anybody have an idea
Mike that is the team in questions ranking overall as I'm reading the sheets.

I have been told that good reps make sure that each of your categories hit a different table so your not being judged by the same folks. Sounds about the same as moving judges around.

[Patrickkva]

Love it. It solves the age old problem of guessing what went wrong. It let's us zero in on and focus on improving our product. By seeing who was against you, you can tell what judges like and don't like. That is...If this actually makes it out of beta without politics breaking it down into a useless program.

Could not say it any better than this. :clap:

[Slamdunkpro]

<Debbie Downer mode>
This reminds me of when casinos added history boards to the roulette tables. They gave the impression of being helpful but were statistically insignificant.
back to BBQ
Even if you know the table# and who was at that table you still have far too many variables to draw accurate conclusions based on that one sample.</Debbie Downer Mode>

[Smoke Ring]

This confirms what I have been saying about KCBS contests involving luck. Yes, to win you have to cook well and consistently but you also have to be lucky. Someone commented this solves the problem of guessing what went wrong, but unfortunately it doesn't solve the problem that something went wrong.

[Jorge]

As for the luck thing, there's always going to be luck involved when it gets into the tent, there's no real way to avoid it unless you clone all of the judges from one person but by shuffling everything I'd guess the luck factor (good table/bad table) could be reduced a bunch.

By shuffling boxes AND moving judges you actually increase the chance of one or more teams being judged by high/low scoring judges more than once.

[Candy Sue]

Why mix up judges on tables between categories? A better solution, IMO, is to make certain that boxes don't land on the same table in front of the same judges multiple times. How would you be certain that the judges that judged you in chicken wouldn't be judging you in ribs, pork and brisket, if you mix them up on tables too?

It's my understanding that there will be minor tweaks on the reports but the info on the reports above will be there when KCBScore replaces BQwin next year. Also, ranking change (approved last year) will happen for next year as well.

This has been a very long and frequently painful development process. I'm really looking forward to the release of KCBScore 1.0. As issues arise, they are being noted for inclusion in the next version release.

[Slamdunkpro]

By shuffling boxes AND moving judges you actually increase the chance of one or more teams being judged by high/low scoring judges more than once.
+1

The distribution of entries process between the turn in table and entries hitting the judging tables works really well. It's rare that an entry goes to the same table twice.

[tmcmaster]

I wonder how this would work out if all certified judges scores were kept and averaged over a period of time or several contests. Judges could then be assigned to tables so that all tables have near the same cumulative average for scoring. If all tables had a close to the same average in scoring, it would seemingly(on the surface) start a contest with a more level playing field. This way you shouldn't get 1 table who likes to score all 9's for practically everything. Just a thought....
Very good idea! Almost assures the BOD won't even consider it.

[thaberberger]

Many of us with years and years of experience in cookoffs (not just BBQ) have known this and some of us have preached and tried to inform teams of this. There is a luck factor involved. Even if you train and train and get very well "educated" judges, group a few together with the same tendencies and same biases (we all have them; we're human) and the same types of preferences, and what you'll get are "good" tables and "bad" tables. It happens. I'm sure there's some scientific law here that applies.

For this reason, MANY sanctioning bodies force tables to change judges (mix up the mix) every category. You'll still end up with good and bad tables, but they've shifted and mixed.

There's still the luck of who else's BBQ landed on your table. Trust me, even if you judge each on their own merits, a judge, after tasting 3 or 4 fairly average or below average entries gets one good one, watch that score. It'll go WAY up. Deserved that high? Possibly; possibly not. Same on the other side, if yours is 4th in their tasting and they've had 3 in front of you that were off-the-chart awesome and yours is then very good too, the odds of getting that perfect score are fairly low.

It happens. We're dealing with humans.

For these reasons, many/most seasoned competitors will counsel new competitors to not read too much into any 1 competition scores, and dont change recipes, etc. until you've had 3 or 4 or 5 competitions saying the same thing. THEN and only then should you consider making changes.

Kudos to KCBS for showing this (like many others have done for a while).

100% agree.

[Lake Dogs]

By shuffling boxes AND moving judges you actually increase the chance of one or more teams being judged by high/low scoring judges more than once.

You're right, but the "good" tables and the "bad" tables are shuffled. The odds increase that any 1 judge sees your Q does go up, but the odds are 0 that the same group of judges judges it. It's about mixing up the grouping and minimizing the effect of good/bad tables. It does this. It doesn't prevent one of the same judges judging your Q again.

Mind you, I applaud the new scoring results posting.

[Lake Dogs]

This confirms what I have been saying about KCBS contests involving luck. Yes, to win you have to cook well and consistently but you also have to be lucky. Someone commented this solves the problem of guessing what went wrong, but unfortunately it doesn't solve the problem that something went wrong.

This isn't a KCBS problem so much as it is a problem with every cookoff with multiple tables. It's a bit of the nature of the beast. You will have "hot" tables, and you will have "cold" tables. Your entry will land on a table with what should be the top 5 or 6 entries in the whole contest (and they aren't scored that way), and your entry will land on a table where everyone else around yours is aweful (so your good entry stands out just that much more). Your entry will also be tested after an entry that leaves a bad taste in the tasters mouth and yours will get a lesser score as a result. It happens, regardless of the sanctioning body, regardless of the type of cookoff.

[Jorge]

You're right, but the "good" tables and the "bad" tables are shuffled. The odds increase that any 1 judge sees your Q does go up, but the odds are 0 that the same group of judges judges it. It's about mixing up the grouping and minimizing the effect of good/bad tables. It does this. It doesn't prevent one of the same judges judging your Q again.

Mind you, I applaud the new scoring results posting.

That can be accomplished by sending boxes to different tables, and retaining the opportunity for each team to land on good/bad tables.

I want each team to have the same opportunity to land on the SAME table. Otherwise you just penalized the six teams that landed on a 'bad" table for chicken if you change the makeup for ribs. If you are randomly moving people from table to table there doesn't appear to be any safeguard to prevent the formation of another bad table, which one or more teams from the first category may land on again further hurting their chances. The converse is also true. You may create a judging pool where one or more teams gain an advantage in the event they land on more than one high scoring table.

[mobow]

I wonder how this would work out if all certified judges scores were kept and averaged over a period of time or several contests. Judges could then be assigned to tables so that all tables have near the same cumulative average for scoring. If all tables had a close to the same average in scoring, it would seemingly(on the surface) start a contest with a more level playing field. This way you shouldn't get 1 table who likes to score all 9's for practically everything. Just a thought....

I am not a statistician nor do I play one on TV. But several contest would not be enough to draw any meaningful conclusions. Most researchers consider 30 to be a minimum number of participants for a study to have any validity. 100 is better. Given the nature of this beast I think 30 or so is a good number to consider. It will then average out across the board for those times that really good or really bad food crosses the judging table.

And in my opinion the all 9's everytime judge is as much and maybe more of a problem than the lower scoring judge. keith

[olewarthog]

That can be accomplished by sending boxes to different tables, and retaining the opportunity for each team to land on good/bad tables.

I want each team to have the same opportunity to land on the SAME table. Otherwise you just penalized the six teams that landed on a 'bad" table for chicken if you change the makeup for ribs. If you are randomly moving people from table to table there doesn't appear to be any safeguard to prevent the formation of another bad table, which one or more teams from the first category may land on again further hurting their chances. The converse is also true. You may create a judging pool where one or more teams gain an advantage in the event they land on more than one high scoring table.

But shouldn't the objective be NOT to have "good" & "bad" tables? Assigned seating means that a "good" or "bad" table remains as such throughout the contest. Random seating means that the same 5 or 6 "good" judges and the same 5 or 6 "bad" judges do not determine the winners & losers.

[Smokin' Joe]

Also, ranking change (approved last year) will happen for next year as well.



Could someone please refresh my memory on what this change is?

[Lake Dogs]

...
The converse is also true. You may create a judging pool where one or more teams gain an advantage in the event they land on more than one high scoring table.

But, they do now (in KCBS), or negative tables. In the test they had 6 tough tables (dont like the word bad) and they had 10 easy tables apparently. By keeping the tables as-is, there's a good chance that a few of the teams consistently landed on one of those 10 good/easy tables, putting the others at a distinct disadvantage. I'd personally prefer, for lack of a better word, busting the good and bad tables by shuffling judges. JMHO.

[Basted moral support guy]

Seeing those scans reminded me of how much we overcooked our chicken. I tried to block it out. :icon_blush:

[stl-rich]

Are the Judges given similar data? Back when I was judging I always wanted to know how myscores compared

[Big Mike]

Are the Judges given similar data? Back when I was judging I always wanted to know how myscores compared

Excellent idea. As a cook, I don't really care that I can tell who was on the table with me. What I like is seeing the judges avg score. I think that would be good for a judge to see as well

[Carnivorous Endeavors BBQ]

Mobow,

I write this with all due respect but and I understand what you are saying but it was more about the general concept than trying to get caught up in the details. If the new system won't be effective until 30-100 contests judged then it would seemingly be useless as it takes most folks are really long time to achieve that.

[EatonHoggBBQ]

A better solution, IMO, is to make certain that boxes don't land on the same table in front of the same judges multiple times.


Gee, I may be wrong and correct me if I am but..... I think this is what they are already doing. Or at least it is what I have experienced in the past six years that I have been a certified judge. :confused:

[tmcmaster]

Mobow,

I write this with all due respect but and I understand what you are saying but it was more about the general concept than trying to get caught up in the details. If the new system won't be effective until 30-100 contests judged then it would seemingly be useless as it takes most folks are really long time to achieve that.
I think he meant KCBS testing this at 30 - 100 contests. They should be able to knock that out in a few weeks next year. .

[Grizmt]

By shuffling boxes AND moving judges you actually increase the chance of one or more teams being judged by high/low scoring judges more than once.
I think there's some clarification needed to what I was saying. Once the boxes get to the TC they are (at every event I've ever been at) checked for duplication from a list each TC has and swapped out if clam on the tray is from a team that has already hit the table.
The reason I suggested the shuffle of clams (boxes) prior to getting to the TC is to pretty much kill off a fairly easy way to "help" a friend win. Before anyone gets wound up, I'm not saying it's ever been done as I've not been at every event but I know it's possible to do. Shuffling the boxes prior to getting to the TC wouldn't stop clams from hitting a good or bad table, that's not why I suggested it.
Shuffling judges and TC's would certainly help the good/bad,cold/hot table issues.
Just my thoughts, your mileage may vary.

[Slamdunkpro]

I think there's some clarification needed to what I was saying. Once the boxes get to the TC they are (at every event I've ever been at) checked for duplication from a list each TC has and swapped out if clam on the tray is from a team that has already hit the table.
Interesting, I've never seen it done at the TC level. That would seem to be a lot of box handling with the potential for dropped boxes. Here the boxes are always shuffled and checked for repeat tables before the TC's get their trays.

[Outnumbered]

Like it a lot. Not sure I see the value in knowing the number vs. the blind number however. Seems like overkill to me. What value does that really provide?

[Grizmt]

Interesting, I've never seen it done at the TC level. That would seem to be a lot of box handling with the potential for dropped boxes. Here the boxes are always shuffled and checked for repeat tables before the TC's get their trays.

The procedure I've seen is this with a couple of minor variations:

1. Turn in hits the turn in table and is placed in the tray along with whatever number of clams that table will be judging which could be anywhere from 5 to as high as 8 (that I've seen) depending on how many judges are on site.

2.The numbers are changed from team number to KCBS number using whatever "factor" was generated for that contest.

3.Most times it is double checked for accuracy.

4.The tray is brought inside or set on a TC table right inside the tent depending on the configuration of the tent and how many TC's there are.

5.The TC's then check the box #'s for boxes that have hit their table before and swap out with other TC's if there are any repeats.

6. TC's then carry the tray to the table and start the judging process.

[Outnumbered]

There is a commitee in charge of designing, debugging, testing, and implementing the new KCBSCORE program. It is headed by BOD member Mike Budai who when each step is accomplished then makes a monthly report to the entire BOD at the monthly meeting.
I don't find it suprising that the entire BOD was not aware of a beta test of the new scoring system as Mr. Budai was just doing his job getting the system up and running.
I'm very certain the board will be fully informed of progress at the next BOD meeting.
Don't be all hating at KCBS for doing what we ask of them.
Ed

Sounds to me like, and I'm totally uneducated in the BOD history here, but it is operating as a BOD should. Boards set strategy, not tactics.

The smaller group is testing a new system from a tactical standpoint after the BOD set the strategy that they may need to look at improving the scoring system.

If it works, they can bring it the entire BOD for approval and full adoption then. If there is no beta test, how can the BOD have any data to support their decision either way.

[Jorge]

But shouldn't the objective be NOT to have "good" & "bad" tables? Assigned seating means that a "good" or "bad" table remains as such throughout the contest. Random seating means that the same 5 or 6 "good" judges and the same 5 or 6 "bad" judges do not determine the winners & losers.

It absolutely should be. My point has been that if you begin moving judges around, from table to table, you risk creating additional tables that score high or low. I haven't seen any safeguards mentioned to prevent that from happening.

If a system that moves judges based on how they score a category, in comparison to others at the same table, you are looking at additional issues. Once you start singling judges out based on their scores you are likely to see a change in the way they score, and there is no guarantee that it will be a reasonable correction. If a judge has been a point low, how does anyone know that they won't start scoring two points higher to avoid being singled out and moved to another table in front of their peers?

The new software can become a very powerful tool. With power comes responsibility. It will take some time to really find out what the data says. Some trends are likely to become very evident early on, and some of those trends may change dramatically over time as the data gathered grows to proved a more complete and accurate view of what is actually happening.

There are also risks associated with telling a judge how far outside the norm they are without sufficient data. Being an outlier one day may indicate that 5 other people were off that day, if over time that outlier is more consistent than those other 5 judges. If we create a system where judges adjust their scoring from week to week, the data and system will suffer.

My personal opinion is that the responsible thing to do is take a look at a reasonable data set, see what can be learned from it, and then assess the options available to improve the system.

My .02

[mobow]

Mobow,

I write this with all due respect but and I understand what you are saying but it was more about the general concept than trying to get caught up in the details. If the new system won't be effective until 30-100 contests judged then it would seemingly be useless as it takes most folks are really long time to achieve that.

I do not disagree with you at all. The points I am wanting to add to this conversation are that good decisions require accurate conclusions and accurate conclusions can only occur if we have good data. Good data takes time. Yes, maybe a year or so. That does not make it useless. Just not as fast as we might like it. keith

[CivilWarBBQ]

The procedure I've seen is this with a couple of minor variations:

1. Turn in hits the turn in table and is placed in the tray along with whatever number of clams that table will be judging which could be anywhere from 5 to as high as 8 (that I've seen) depending on how many judges are on site.

2.The numbers are changed from team number to KCBS number using whatever "factor" was generated for that contest.

3.Most times it is double checked for accuracy.

4.The tray is brought inside or set on a TC table right inside the tent depending on the configuration of the tent and how many TC's there are.

5.The TC's then check the box #'s for boxes that have hit their table before and swap out with other TC's if there are any repeats.

6. TC's then carry the tray to the table and start the judging process.

Close, but that's not the procedure we use in Georgia.

1) Numbers are changed first. Usually multiple people are involved to avoid delays and double check for errors.

2) Turn-in personnel sort the boxes into the trays, referring to the checkoff list the Table Captain brought back after the last category to avoid placing the any box number on a table twice. (If it's the first category, boxes are sorted randomly.)

3) TC retrieves their tray along with the checkoff list. They cross off each box in the tray from the list. If any repeats are encountered, they confer with the Rep to see if a swap is warranted. (sometimes due to low team counts, etc. a repeat is unavoidable)

4) TC presents the boxes to judges for scoring.

5) When all score cards are turned in to the Rep, TC returns the tray with the checkoff list to the turn-in table for the next category.

[CivilWarBBQ]

Almost forgot:

I do have one big problem with the beta scoring sheets. The alternate number should never be listed, as this breaks the blind. If you cross reference the alternate number to team names like that, then you will have teams and judges comparing notes on entries between judging and awards.

If you are going to do that, you may as well forget renumbering completely.

[Uomograsso]

Close, but that's not the procedure we use in Georgia.

1) Numbers are changed first. Usually multiple people are involved to avoid delays and double check for errors.

2) Turn-in personnel sort the boxes into the trays, referring to the checkoff list the Table Captain brought back after the last category to avoid placing the any box number on a table twice. (If it's the first category, boxes are sorted randomly.)

3) TC retrieves their tray along with the checkoff list. They cross off each box in the tray from the list. If any repeats are encountered, they confer with the Rep to see if a swap is warranted. (sometimes due to low team counts, etc. a repeat is unavoidable)

4) TC presents the boxes to judges for scoring.

5) When all score cards are turned in to the Rep, TC returns the tray with the checkoff list to the turn-in table for the next category.


In all the contests I have judged or TC'ed the TC's receive the boxes and check to make sure that their table has not received a box from that team before. I have only heard of one pair of reps that do the duplicate checking/re-ordering.

I usually do one or two contests a month. I will admit that even though I try to maintain consistent standards, my judging can vary from contest to contest. Some contests I may judge "harder" than others depending on my mood. Judging for appearance and tenderness allow for the most objectiveness. Does a box look good, is the meat presented neatly, does the rib meat come off the bone cleanly, can you pull the brisket apart or is it a rubber band, is the pork mushy, etc. etc. Taste is almost purely subjective. How can you train someone what good BBQ should taste like? If you can come up with a consensus about what good BBQ should taste like I'll do my best to apply it when I judge.

[pahutchens]

As a Judge and Table Captain I've seen the luck and had a few discussions with other judges/TC. Sometimes you have a hot a table because your table 'lucked' out and got a string of contenders. One contest this summer had the best string of ribs that I could recall and another contest ribs were politely described as dismal.

Perhaps a better example. Sam's club we had 5 of 6 pork just money muscle only. Two other judges (different tables) I remarked about it too said there's were more 'standard' pulled pork and little to no money muscle. Not saying Miron Mixon is right that you need money muscle to win. Butt I thinks some cooks have really bought in to it.

As a TC I was told I'm the Rep's rep. If my table has wide swings in scores I bring it to their attention and the judges are monitored a little closer to see it a trend is there that is not supported by the rest of the table. They don't tell them what to score but remind them of HOW to score.

[smoke-n-my-i's]

As a Judge and Table Captain I've seen the luck and had a few discussions with other judges/TC. Sometimes you have a hot a table because your table 'lucked' out and got a string of contenders. One contest this summer had the best string of ribs that I could recall and another contest ribs were politely described as dismal.

Perhaps a better example. Sam's club we had 5 of 6 pork just money muscle only. Two other judges (different tables) I remarked about it too said there's were more 'standard' pulled pork and little to no money muscle. Not saying Miron Mixon is right that you need money muscle to win. Butt I thinks some cooks have really bought in to it.

As a TC I was told I'm the Rep's rep. If my table has wide swings in scores I bring it to their attention and the judges are monitored a little closer to see it a trend is there that is not supported by the rest of the table. They don't tell them what to score but remind them of HOW to score.

I totally do not agree with you on this one.... my last contest, 30 teams, my pork was overdone and the MM was pretty much mush... I used larger than normal pulled, and got a 5th place..... so, with NO money muscle at all, how would you explain your theory of you have to have MM to win?

[Jorge]

I totally do not agree with you on this one.... my last contest, 30 teams, my pork was overdone and the MM was pretty much mush... I used larger than normal pulled, and got a 5th place..... so, with NO money muscle at all, how would you explain your theory of you have to have MM to win?

I think his point was that many teams seem to have bought into that theory, not that it was accurate.

[pahutchens]

I think his point was that many teams seem to have bought into that theory, not that it was accurate.

Yeah that's what I thought I said too :thumb:

[crd26a]

In theory (okay, the theory in my mind on this one), the issue will be addressed in the seating of judges at the beginning of a contest. If they are able to track high judges, low judges, etc., for the experienced judges at a contest, each table should be "balanced" prior to seating the judges. This will take away allowing a judge to sit where they please, however, it would put the tables on an even keel for the known judges. To me, this will become where the evening out is.

This will still allow for the new or inexperienced judges to be wild cards. But if we can get 4 of the 6 on average down to a known balance, we "should" see more consistent judging across the tables going forward. I'd much rather compete knowing that the average scores on most of the judges is balanced and taking a shot at 1-2 wild cards vs. a table of who knows what.

And as a side note, I'm still willing to bet we'll see the historically strong teams still be strong. It just could help level out some of the playing field.

[Grizmt]

It'll be interesting to see how the data plays out over time, especially when up until recently there were some (what I consider) wide swings in what judges were told about scoring when in training class and at some events.

This brings up a question for y'all, is a judge who's considered a hard or low scorer incorrect or following what he/she was taught just like the easy/high scoring judge is doing.
The reason I ask this is as many people know for a while (at least out here in the west) judges were instructed to basically not score below a 5/6 unless it's completely and totally inedible. The reason given was "it would be an insult after all the hard work and money that the teams put in to get to the judges tent with an entry".

Given that instruction wouldn't or couldn't that cause a certain section of judges to automatically score higher and the judges that had not heard that instruction actually use the full 2-9 system meaning they could be scoring consistently lower?
I'd suggest that regional data sampling would probably be a good idea. I'd almost bet that you'd see the data swing from one region to another and most likely from one category to another like chicken being scored harder than let's say beef in TX or beef harder than pork in TN. Just thinking about regional tastes is all and no, I will NOT give an example of how the West Coast would score since KCBS doesn't have a "Tofurkey" category.:laugh::laugh::laugh::tongue:

[Rooter-Tooter BBQ]

ultimately the only way to resolve this in which i have no idea how to do it and im sure nobody else does is: when our boxes go to a table we are competing against those six boxes. To show who is truly the best is to let each judge taste 40 pieces of chicken, ribs, pork, and brisket. Then you will really know who is the best. But dang, i would get full plus i would forget what the other 39 pieces tasted like, but hey that could be a good thing. Who knows, i just like being able to be there even if we don't win. I mean there is nothing like having stories to tell after a comp...like just by chance your trailer flips and the cooker door is open and 3 of your 4 slabs of ribs fall on the ground or maybe just maybe fall into the soapy sink. Im just saying, not that happened or nothing. LOL

[stl-rich]

At well run comps judges are not allowed to just sit as they please - good reps try to balance the master judges, the experienced, the semi-experienced and the noobs.

Good reps do a good job :grin:

[Grizmt]

At well run comps judges are not allowed to just sit as they please - good reps try to balance the master judges, the experienced, the semi-experienced and the noobs.

Good reps do a good job :grin:

That's been my experience at almost every comp I've been at. Hey, that's it! Take the judges newly discerned rankings and mix 'em up at each table, tough,easy and undecided.:wink:

[Funtimebbq]

That was suggested many pages ago but will only work after judges have had several contests for the system to calculate their scoring pattern. Unfortunately, by the time that happens, all judges will be scoring so high, it will not matter. Once judges see themselves being sent to re-education camps, those who are on the bubble will get the message and start scoring higher, defeating the system.

[smoke-n-my-i's]

I think his point was that many teams seem to have bought into that theory, not that it was accurate.

Yeah that's what I thought I said too :thumb:

sorry abt that one... I totally read it too fast and missed the point.... I seem to do that some times, and regret it later....

[CBQ]

I wonder how this would work out if all certified judges scores were kept and averaged over a period of time or several contests. Judges could then be assigned to tables so that all tables have near the same cumulative average for scoring. If all tables had a close to the same average in scoring, it would seemingly(on the surface) start a contest with a more level playing field. This way you shouldn't get 1 table who likes to score all 9's for practically everything. Just a thought....

That might be an interesting idea. Talking to reps, they sometimes KNOW there are certain judges that always score low or score high. Some reps have told me they dread certain judges showing up at a contest because they know a lot of 5s and 6s are going to be handed out for some good bbq. The reps, however, don't have any power to DO anything about that. The above suggestion might have to mitigate some of the risk - but I wouldn't try to group like judges together.

Winning the table...when it's a low scoring table...doesn't help you. Spreading out low scoring judges evenly across all the tables would be more fair to the teams.

[Grizmt]

That might be an interesting idea. Talking to reps, they sometimes KNOW there are certain judges that always score low or score high. Some reps have told me they dread certain judges showing up at a contest because they know a lot of 5s and 6s are going to be handed out for some good bbq. The reps, however, don't have any power to DO anything about that. The above suggestion might have to mitigate some of the risk - but I wouldn't try to group like judges together.

Winning the table...when it's a low scoring table...doesn't help you. Spreading out low scoring judges evenly across all the tables would be more fair to the teams.

This brings up some food for thought. Why is it that someone scoring 5-6 is considered "low scoring"? Doesn't the rating scale go from 2-9 or has this truly turned into 6-9 scoring? Is there now an attitude like in T-Ball where there's a trophy for just being there?
The reps said "they know a lot of 5-6's are going to be handed out for some good BBQ", in whose opinion (theirs?) and why is their opinion any more valid that the person judging it?
Then you have the question of who's palate is correct, could the people who're "low scoring" actually have a better palate than the "high scoring"?
Would y'all prefer false high scoring? If so then maybe the scoring and judges instructions should be changed to reflect that.

I hope you don't take this as me jumping on you or what you posted, it's not meant that way, I think you bring up some good points for discussion.

I keep hearing how this judge or that judge scored low so the judge "must" be the issue, why isn't the same said for judges that score crazy high when the rest of the table doesn't? I've never,ever seen a judge taken aside for scoring too high, just too low.
A good example of this whole argument is Hells Kitchen. You have all the chefs make a dish and invariably each one thinks it tastes fantastic. When it's tasted by Gordon or a selected group of celebrity chefs they have a way different opinion. Same thing on Chopped,Cupcake wars,wine tasting etc.

Until such a time where an absolute set of iron clad taste criteria can be laid out so every person tastes the same way/thing AND the judges are all instructed the absolute same way then you're going to have personal taste variations/rule interpretations that are not explainable nor changeable.

I guess one thing could be done which is having to pass a palate test before you're allowed to be a judge. :idea:

[smoke-n-my-i's]

Here is a section from my TC pamphlet that was given to us at our class. Dated 4-14-2011

And as a cook, I have seen this posted in areas used by judges before, so it is not top secret.

Judges are told the scoring system is from 9 to 2.
9 is excellent
8 very good
7 above average
6 average
5 below average
4 poor
3 bad
2 inedible

So, there seems to be some judges that think that most meat coming across their score sheet is average or below. So, what do they deem would be a 7, 8, or even a 9? That is a good question.

Is the judges given a different scoring sequence in their classes? I would hope not.

[New Pal Frank]

In the CBJ class my wife and I took 2 1/2 years ago, we were told that (average) 6, was what every backyard cook (meaning home cook) would produce. Not what the average team would produce. Therefore, most of the Q that has come accross my tables in the 25 contests we have judged has been 7 or higher.
Also if i score below a 7, I will give a coment card so the team isn't left scratching their head as I have since we started competing this year.
As a couple of Reps have told us in the judges meeting before hand, ther is NO limit to how many 9's you can give out in a contest. If everything presented to you is top shelf product, it is a 9 and I agree. We are judging on criteria, not comparison.

[Grizmt]

In the CBJ class my wife and I took 2 1/2 years ago, we were told that (average) 6, was what every backyard cook (meaning home cook) would produce. Not what the average team would produce. Therefore, most of the Q that has come accross my tables in the 25 contests we have judged has been 7 or higher.
Also if i score below a 7, I will give a coment card so the team isn't left scratching their head as I have since we started competing this year.
As a couple of Reps have told us in the judges meeting before hand, ther is NO limit to how many 9's you can give out in a contest. If everything presented to you is top shelf product, it is a 9 and I agree. We are judging on criteria, not comparison.
I have been instructed on several occasions that 5 is the average, other times it was said to be 6.
Where I kind of take issue is the idea that just because a "team" cooked it that it would automatically start at any number. It's either good or not according to my palate and is scored that way obviously in degrees dependent on how good or how bad it is. If some meat comes across my plate that tastes real heavy of "jet fuel" or "creosote" (over smoked) then the likelihood of it getting a 6 or 7 out of me is slim to none. With that being said do I still score it a 5 or 6 to not insult someone?
Just to be clear, I don't have any problem handing out 9's to everyone who's food I'm judging IF the product deserves it and conversely, no judge should be discouraged or afraid to hand out 3's if so deserved but unfortunately that's the current culture in some places.

[gettinbasted]

The extra information was very useful. Unfortunately my pork was judged at table 10.... we won the most difficult table at least!

Without the new scoring I might be tinkering with my pork technique, but looking at the extra info I will probably stay the course.

[Ringburner]

Sounds like judging is almost as complex as cooking!

Sent from my GT-I9100 using Tapatalk 2

[THE Meatman]

looks nice and clean...easy to understand

[McEvoy AZ]

I may have missed this earlier in the thread, as I have not read the whole thing. It seems to me that what they are doing could be achieved by making sure there is a short commit on every score. Might even come up with a coding system that would expedite the scoring, as in the number 1 means to salty. With this system you would know why a judge scored you the way they did.
I was at a contest that another sanctioning body put on where most judges commented and I found I agreed with some of the judges and some of the judges contidicted other judges. This way you could look at the comment and decide for yourself if the judge was on point or not.
I think there is always going to be luck in this as we are facing human judges and thank god that is the case or the same teams would always win and what would be the point of showing up if you are not one of the winning teams.

[bover]

That was suggested many pages ago but will only work after judges have had several contests for the system to calculate their scoring pattern. Unfortunately, by the time that happens, all judges will be scoring so high, it will not matter. Once judges see themselves being sent to re-education camps, those who are on the bubble will get the message and start scoring higher, defeating the system.

It stands to reason that judges that overscore would be handled in the same manner as those that underscore. Anyone consistently straying from the norm should be subjected to re-education.

[bover]

I'm also in the camp of not including the alt number on the overall report. What purpose does it serve? To me it just seems like an opportunity for collusion at future events assuming the same renumbering methodology is used year after year.