New KCBS Scoring Program Test

[mobow]

Originally Posted by Big Mike

I like the Team Detail Report. I like the fact that I can see each judges avg score and who was at your table. The only thing that confuses me is the "Rank" column. What is that supposed to show. It is obviously not his rank at that table. Anybody have an idea

It is what place he came in over all. keith

[Carnivorous Endeavors BBQ]

I wonder how this would work out if all certified judges scores were kept and averaged over a period of time or several contests. Judges could then be assigned to tables so that all tables have near the same cumulative average for scoring. If all tables had a close to the same average in scoring, it would seemingly(on the surface) start a contest with a more level playing field. This way you shouldn't get 1 table who likes to score all 9's for practically everything. Just a thought....

[MAP]

Look at Judge 3 on table 2. Either he is a low scorer or the others are High scorers.

[ModelMaker]

Originally Posted by tmcmaster

The reason (well, ONE of) I am no longer a member of KCBS.


We all may joke about that sort of thing, but I really can NOT see anyone that I have EVER met in BBQ actually doing anything like that.

There is a commitee in charge of designing, debugging, testing, and implementing the new KCBSCORE program. It is headed by BOD member Mike Budai who when each step is accomplished then makes a monthly report to the entire BOD at the monthly meeting.
I don't find it suprising that the entire BOD was not aware of a beta test of the new scoring system as Mr. Budai was just doing his job getting the system up and running.
I'm very certain the board will be fully informed of progress at the next BOD meeting.
Don't be all hating at KCBS for doing what we ask of them.
Ed

[djqualls]

I cooked this contest and we were told at the cooks meeting that it was going to be used side by side the original system and then used as a comparison as well. They answered any questions by the teams and then added the results along with the original (Official) to our sheets at awards.

I really thought it was a good report and the awards came off early so I suppose the software worked goof. There were 44 teams in this contest soI'm guessing it was stable since we got a copy and awards were early. I did hear Mike thanking Dennis Polson who was there to judge for helping him with the data that day.

[Grizmt]

I found the info most interesting, really drills down to almost the bottom of the available data.
As always when getting this much detail depending on how it's used will determine whether it'll be a good thing or a bad thing in the long run.

Personally I've always liked the idea of mixing things up in the tent by moving TC's around as well as shuffling judges since it matters little if you're at the same table for any given turn. I've also been a big proponent of whomever is working the entry table shuffling the boxes after they've been renumbered before they're brought in the tent to the TC's to better assure truly blind judging.
As for the luck thing, there's always going to be luck involved when it gets into the tent, there's no real way to avoid it unless you clone all of the judges from one person but by shuffling everything I'd guess the luck factor (good table/bad table) could be reduced a bunch.

[dhuffjr]

Originally Posted by Big Mike

I like the Team Detail Report. I like the fact that I can see each judges avg score and who was at your table. The only thing that confuses me is the "Rank" column. What is that supposed to show. It is obviously not his rank at that table. Anybody have an idea

Mike that is the team in questions ranking overall as I'm reading the sheets.

I have been told that good reps make sure that each of your categories hit a different table so your not being judged by the same folks. Sounds about the same as moving judges around.

[Patrickkva]

Originally Posted by TooSaucedToPork

Love it. It solves the age old problem of guessing what went wrong. It let's us zero in on and focus on improving our product. By seeing who was against you, you can tell what judges like and don't like. That is...If this actually makes it out of beta without politics breaking it down into a useless program.

Could not say it any better than this.

[Slamdunkpro]

<Debbie Downer mode>
This reminds me of when casinos added history boards to the roulette tables. They gave the impression of being helpful but were statistically insignificant.
back to BBQ
Even if you know the table# and who was at that table you still have far too many variables to draw accurate conclusions based on that one sample.</Debbie Downer Mode>

[Smoke Ring]

This confirms what I have been saying about KCBS contests involving luck. Yes, to win you have to cook well and consistently but you also have to be lucky. Someone commented this solves the problem of guessing what went wrong, but unfortunately it doesn't solve the problem that something went wrong.

[Jorge]

Originally Posted by Grizmt

As for the luck thing, there's always going to be luck involved when it gets into the tent, there's no real way to avoid it unless you clone all of the judges from one person but by shuffling everything I'd guess the luck factor (good table/bad table) could be reduced a bunch.

By shuffling boxes AND moving judges you actually increase the chance of one or more teams being judged by high/low scoring judges more than once.

[Candy Sue]

Why mix up judges on tables between categories? A better solution, IMO, is to make certain that boxes don't land on the same table in front of the same judges multiple times. How would you be certain that the judges that judged you in chicken wouldn't be judging you in ribs, pork and brisket, if you mix them up on tables too?

It's my understanding that there will be minor tweaks on the reports but the info on the reports above will be there when KCBScore replaces BQwin next year. Also, ranking change (approved last year) will happen for next year as well.

This has been a very long and frequently painful development process. I'm really looking forward to the release of KCBScore 1.0. As issues arise, they are being noted for inclusion in the next version release.

[Slamdunkpro]

Originally Posted by Jorge

By shuffling boxes AND moving judges you actually increase the chance of one or more teams being judged by high/low scoring judges more than once.

+1

The distribution of entries process between the turn in table and entries hitting the judging tables works really well. It's rare that an entry goes to the same table twice.

[tmcmaster]

Originally Posted by Carnivorous Endeavors BBQ

I wonder how this would work out if all certified judges scores were kept and averaged over a period of time or several contests. Judges could then be assigned to tables so that all tables have near the same cumulative average for scoring. If all tables had a close to the same average in scoring, it would seemingly(on the surface) start a contest with a more level playing field. This way you shouldn't get 1 table who likes to score all 9's for practically everything. Just a thought....

Very good idea! Almost assures the BOD won't even consider it.

[thaberberger]

Originally Posted by Lake Dogs

Many of us with years and years of experience in cookoffs (not just BBQ) have known this and some of us have preached and tried to inform teams of this. There is a luck factor involved. Even if you train and train and get very well "educated" judges, group a few together with the same tendencies and same biases (we all have them; we're human) and the same types of preferences, and what you'll get are "good" tables and "bad" tables. It happens. I'm sure there's some scientific law here that applies.

For this reason, MANY sanctioning bodies force tables to change judges (mix up the mix) every category. You'll still end up with good and bad tables, but they've shifted and mixed.

There's still the luck of who else's BBQ landed on your table. Trust me, even if you judge each on their own merits, a judge, after tasting 3 or 4 fairly average or below average entries gets one good one, watch that score. It'll go WAY up. Deserved that high? Possibly; possibly not. Same on the other side, if yours is 4th in their tasting and they've had 3 in front of you that were off-the-chart awesome and yours is then very good too, the odds of getting that perfect score are fairly low.

It happens. We're dealing with humans.

For these reasons, many/most seasoned competitors will counsel new competitors to not read too much into any 1 competition scores, and dont change recipes, etc. until you've had 3 or 4 or 5 competitions saying the same thing. THEN and only then should you consider making changes.

Kudos to KCBS for showing this (like many others have done for a while).

100% agree.