New KCBS Scoring Program Test

Grizmt · Sep 6, 2012

I found the info most interesting, really drills down to almost the bottom of the available data.
As always when getting this much detail depending on how it's used will determine whether it'll be a good thing or a bad thing in the long run.

Personally I've always liked the idea of mixing things up in the tent by moving TC's around as well as shuffling judges since it matters little if you're at the same table for any given turn. I've also been a big proponent of whomever is working the entry table shuffling the boxes after they've been renumbered before they're brought in the tent to the TC's to better assure truly blind judging.
As for the luck thing, there's always going to be luck involved when it gets into the tent, there's no real way to avoid it unless you clone all of the judges from one person but by shuffling everything I'd guess the luck factor (good table/bad table) could be reduced a bunch.

dhuffjr · Sep 6, 2012

Big Mike said:
I like the Team Detail Report. I like the fact that I can see each judges avg score and who was at your table. The only thing that confuses me is the "Rank" column. What is that supposed to show. It is obviously not his rank at that table. Anybody have an idea

Mike that is the team in questions ranking overall as I'm reading the sheets.

I have been told that good reps make sure that each of your categories hit a different table so your not being judged by the same folks. Sounds about the same as moving judges around.

Patrickkva · Sep 6, 2012

TooSaucedToPork said:
Love it. It solves the age old problem of guessing what went wrong. It let's us zero in on and focus on improving our product. By seeing who was against you, you can tell what judges like and don't like. That is...If this actually makes it out of beta without politics breaking it down into a useless program.

Could not say it any better than this. :clap:

Slamdunkpro · Sep 6, 2012

<Debbie Downer mode>
This reminds me of when casinos added history boards to the roulette tables. They gave the impression of being helpful but were statistically insignificant.
back to BBQ
Even if you know the table# and who was at that table you still have far too many variables to draw accurate conclusions based on that one sample.</Debbie Downer Mode>

Smoke Ring · Sep 6, 2012

This confirms what I have been saying about KCBS contests involving luck. Yes, to win you have to cook well and consistently but you also have to be lucky. Someone commented this solves the problem of guessing what went wrong, but unfortunately it doesn't solve the problem that something went wrong.

Jorge · Sep 6, 2012

Grizmt said:
As for the luck thing, there's always going to be luck involved when it gets into the tent, there's no real way to avoid it unless you clone all of the judges from one person but by shuffling everything I'd guess the luck factor (good table/bad table) could be reduced a bunch.

By shuffling boxes AND moving judges you actually increase the chance of one or more teams being judged by high/low scoring judges more than once.

Candy Sue · Sep 6, 2012

Why mix up judges on tables between categories? A better solution, IMO, is to make certain that boxes don't land on the same table in front of the same judges multiple times. How would you be certain that the judges that judged you in chicken wouldn't be judging you in ribs, pork and brisket, if you mix them up on tables too?

It's my understanding that there will be minor tweaks on the reports but the info on the reports above will be there when KCBScore replaces BQwin next year. Also, ranking change (approved last year) will happen for next year as well.

This has been a very long and frequently painful development process. I'm really looking forward to the release of KCBScore 1.0. As issues arise, they are being noted for inclusion in the next version release.

Slamdunkpro · Sep 6, 2012

Jorge said:
By shuffling boxes AND moving judges you actually increase the chance of one or more teams being judged by high/low scoring judges more than once.

+1

The distribution of entries process between the turn in table and entries hitting the judging tables works really well. It's rare that an entry goes to the same table twice.

tmcmaster · Sep 6, 2012

Carnivorous Endeavors BBQ said:
I wonder how this would work out if all certified judges scores were kept and averaged over a period of time or several contests. Judges could then be assigned to tables so that all tables have near the same cumulative average for scoring. If all tables had a close to the same average in scoring, it would seemingly(on the surface) start a contest with a more level playing field. This way you shouldn't get 1 table who likes to score all 9's for practically everything. Just a thought....

Very good idea! Almost assures the BOD won't even consider it.

thaberberger · Sep 6, 2012

Lake Dogs said:
Many of us with years and years of experience in cookoffs (not just BBQ) have known this and some of us have preached and tried to inform teams of this. There is a luck factor involved. Even if you train and train and get very well "educated" judges, group a few together with the same tendencies and same biases (we all have them; we're human) and the same types of preferences, and what you'll get are "good" tables and "bad" tables. It happens. I'm sure there's some scientific law here that applies.

For this reason, MANY sanctioning bodies force tables to change judges (mix up the mix) every category. You'll still end up with good and bad tables, but they've shifted and mixed.

There's still the luck of who else's BBQ landed on your table. Trust me, even if you judge each on their own merits, a judge, after tasting 3 or 4 fairly average or below average entries gets one good one, watch that score. It'll go WAY up. Deserved that high? Possibly; possibly not. Same on the other side, if yours is 4th in their tasting and they've had 3 in front of you that were off-the-chart awesome and yours is then very good too, the odds of getting that perfect score are fairly low.

It happens. We're dealing with humans.

For these reasons, many/most seasoned competitors will counsel new competitors to not read too much into any 1 competition scores, and dont change recipes, etc. until you've had 3 or 4 or 5 competitions saying the same thing. THEN and only then should you consider making changes.

Kudos to KCBS for showing this (like many others have done for a while).

100% agree.

Lake Dogs · Sep 6, 2012

Jorge said:
By shuffling boxes AND moving judges you actually increase the chance of one or more teams being judged by high/low scoring judges more than once.

You're right, but the "good" tables and the "bad" tables are shuffled. The odds increase that any 1 judge sees your Q does go up, but the odds are 0 that the same group of judges judges it. It's about mixing up the grouping and minimizing the effect of good/bad tables. It does this. It doesn't prevent one of the same judges judging your Q again.

Mind you, I applaud the new scoring results posting.

Lake Dogs · Sep 6, 2012

Smoke Ring said:
This confirms what I have been saying about KCBS contests involving luck. Yes, to win you have to cook well and consistently but you also have to be lucky. Someone commented this solves the problem of guessing what went wrong, but unfortunately it doesn't solve the problem that something went wrong.

This isn't a KCBS problem so much as it is a problem with every cookoff with multiple tables. It's a bit of the nature of the beast. You will have "hot" tables, and you will have "cold" tables. Your entry will land on a table with what should be the top 5 or 6 entries in the whole contest (and they aren't scored that way), and your entry will land on a table where everyone else around yours is aweful (so your good entry stands out just that much more). Your entry will also be tested after an entry that leaves a bad taste in the tasters mouth and yours will get a lesser score as a result. It happens, regardless of the sanctioning body, regardless of the type of cookoff.

Jorge · Sep 6, 2012

Lake Dogs said:
You're right, but the "good" tables and the "bad" tables are shuffled. The odds increase that any 1 judge sees your Q does go up, but the odds are 0 that the same group of judges judges it. It's about mixing up the grouping and minimizing the effect of good/bad tables. It does this. It doesn't prevent one of the same judges judging your Q again.

Mind you, I applaud the new scoring results posting.

That can be accomplished by sending boxes to different tables, and retaining the opportunity for each team to land on good/bad tables.

I want each team to have the same opportunity to land on the SAME table. Otherwise you just penalized the six teams that landed on a 'bad" table for chicken if you change the makeup for ribs. If you are randomly moving people from table to table there doesn't appear to be any safeguard to prevent the formation of another bad table, which one or more teams from the first category may land on again further hurting their chances. The converse is also true. You may create a judging pool where one or more teams gain an advantage in the event they land on more than one high scoring table.

mobow · Sep 6, 2012

Carnivorous Endeavors BBQ said:
I wonder how this would work out if all certified judges scores were kept and averaged over a period of time or several contests. Judges could then be assigned to tables so that all tables have near the same cumulative average for scoring. If all tables had a close to the same average in scoring, it would seemingly(on the surface) start a contest with a more level playing field. This way you shouldn't get 1 table who likes to score all 9's for practically everything. Just a thought....

I am not a statistician nor do I play one on TV. But several contest would not be enough to draw any meaningful conclusions. Most researchers consider 30 to be a minimum number of participants for a study to have any validity. 100 is better. Given the nature of this beast I think 30 or so is a good number to consider. It will then average out across the board for those times that really good or really bad food crosses the judging table.

And in my opinion the all 9's everytime judge is as much and maybe more of a problem than the lower scoring judge. keith

olewarthog · Sep 6, 2012

Jorge said:
That can be accomplished by sending boxes to different tables, and retaining the opportunity for each team to land on good/bad tables.

I want each team to have the same opportunity to land on the SAME table. Otherwise you just penalized the six teams that landed on a 'bad" table for chicken if you change the makeup for ribs. If you are randomly moving people from table to table there doesn't appear to be any safeguard to prevent the formation of another bad table, which one or more teams from the first category may land on again further hurting their chances. The converse is also true. You may create a judging pool where one or more teams gain an advantage in the event they land on more than one high scoring table.

But shouldn't the objective be NOT to have "good" & "bad" tables? Assigned seating means that a "good" or "bad" table remains as such throughout the contest. Random seating means that the same 5 or 6 "good" judges and the same 5 or 6 "bad" judges do not determine the winners & losers.

Smokin' Joe · Sep 6, 2012

Candy Sue said:
Also, ranking change (approved last year) will happen for next year as well.

Could someone please refresh my memory on what this change is?

Lake Dogs · Sep 6, 2012

Jorge said:
...
The converse is also true. You may create a judging pool where one or more teams gain an advantage in the event they land on more than one high scoring table.

But, they do now (in KCBS), or negative tables. In the test they had 6 tough tables (dont like the word bad) and they had 10 easy tables apparently. By keeping the tables as-is, there's a good chance that a few of the teams consistently landed on one of those 10 good/easy tables, putting the others at a distinct disadvantage. I'd personally prefer, for lack of a better word, busting the good and bad tables by shuffling judges. JMHO.

Basted moral support guy · Sep 6, 2012

Seeing those scans reminded me of how much we overcooked our chicken. I tried to block it out. :icon_blush:

stl-rich · Sep 6, 2012

Are the Judges given similar data? Back when I was judging I always wanted to know how myscores compared

Big Mike · Sep 6, 2012

stl-rich said:
Are the Judges given similar data? Back when I was judging I always wanted to know how myscores compared

Excellent idea. As a cook, I don't really care that I can tell who was on the table with me. What I like is seeing the judges avg score. I think that would be good for a judge to see as well

New KCBS Scoring Program Test

Got Wood.

Full Fledged Farker

Knows what a fatty is.

Babbling Farker

Smoke Ring

Guest

somebody shut me the fark up.

is one Smokin' Farker

Babbling Farker

is Blowin Smoke!

Knows what a fatty is.

Quintessential Chatty Farker

Quintessential Chatty Farker

somebody shut me the fark up.

is one Smokin' Farker

Full Fledged Farker

is one Smokin' Farker

Quintessential Chatty Farker

Knows what a fatty is.

is one Smokin' Farker

is one Smokin' Farker

Similar threads