-
working on DNS.. links may break temporarily.
New KCBS scoresheet
- Thread starter big matt
- Start date
musicmanryann
is Blowin Smoke!
I'm with Vince. At our contest this weekend there was clearly and bad table and a great table (from a cooks perspective ;-)). Fortunately for ours most, if not all, hit every table once. I have been saying for a couple of years now that sorting judges based on how they actually score, rather than on the number of contests they have competed would be a much more reliable method for equalizing the scoring across tables. You have judges that score high, score low, and score middle of the road no matter their experience level. Experience does matter a great deal, but there are still bad judges with a lot of experience and good judges with little experience.
It seems this new scoring system would open the door for tracking and sorting judges based on their variance from the mean. The question is whether or not the new system is currently keeping track of judges' scoring across multiple contests.
It seems this new scoring system would open the door for tracking and sorting judges based on their variance from the mean. The question is whether or not the new system is currently keeping track of judges' scoring across multiple contests.
Muzzlebrake
Babbling Farker
I'm going to use this data to help me make adjustments to things within my control. This past week for instance I know my ribs and pork tanked. I didn't really like either one but it didn't think they were as bad as they placed. Looking at the scores I can now see my ribs hit a low scoring table and the pork just tanked on its own merit.
I would have thought the judges scoring statistics would be very useful to them just as Tigerpaw mentioned.
I'm not sure that we need to seat judges according to their scoring averages to even out the tables though. In our contest this weekend, the top 10 overall I think 7 of the teams had hit the low scoring table at least once so it seems to have balanced out in the end. On the other hand, the GC didn't hit it once and won by 10 points, 2-4 all hit that table and were only separated by 5 points.
Who knows? I think more data is definitely need before any adjustments are made. The rest of the year should give you a good sample range to work with.
I would have thought the judges scoring statistics would be very useful to them just as Tigerpaw mentioned.
I'm not sure that we need to seat judges according to their scoring averages to even out the tables though. In our contest this weekend, the top 10 overall I think 7 of the teams had hit the low scoring table at least once so it seems to have balanced out in the end. On the other hand, the GC didn't hit it once and won by 10 points, 2-4 all hit that table and were only separated by 5 points.
Who knows? I think more data is definitely need before any adjustments are made. The rest of the year should give you a good sample range to work with.
mobow
is one Smokin' Farker
You also have to keep in mind that the low table may be low or it may be accurate. The high table may be accurate or it may be inflated. More information can be good but it can also increase the odds of a faulty conclusion. Keith
Outnumbered
is One Chatty Farker
- Joined
- Sep 18, 2011
- Messages
- 2,131
- Reaction score
- 1,699
- Points
- 0
- Location
- Smithville, Missouri, United States
I commend you for that and this is not a critique of how you handled. And thank you for handling it this way as it seems to be the best way to do it. Just using it to illustrate my point.
Assume that the non-CBJs score a point lower than the rest of the table for whatever reason. They throw the lowest out, so we can assume it's the low score, so you have two non-CBJ who score a point lower on across the board on taste from a purely arbitrary reason (at least theoretically here). You now have lowered the scores from that table by 5 points per category, per team. That's a 20 point swing for all 4 meats and can mean the difference between a GC and middle of the pack finish for overall.
Of course, the same can be said for the other extreme as well. Maybe you did a middle of the road cook, but you hit the right table and you mediocre stuff got GC.
But as gettin'basted says, this new system at least gives you the information to know if your entry was bad or if your luck was.
Thanks to KCBS for making this change. I think it's great.
Alexa RnQ
is One Chatty Farker
- Joined
- Sep 12, 2007
- Messages
- 2,063
- Reaction score
- 2,200
- Points
- 0
I think tracking across all categories and an accumulated body of several contests will reduce the faulty conclusion. A judge may land on a lucky table that gets all great entries, or on a poor table that gets all scary entries, but the odds of that happening in all four categories in contest after contest over time are pretty remote.
bignburlyman
is one Smokin' Farker
With all the talk of now being able to see more data and more of the tables scores, and how this may help balance the judges better, no one has mentioned the 800 pound gorilla in the room---the judges will most likely never see any of the score sheets. How will this help any judge to realize they are too harsh or too generous unless KCBS taps them on the shoulder 5 contests down the line and say "Your scores do not fit the mean average of the other judges"?
There are a lot of great judges out there, and there are a few total dooshbags, most of my contests judged are in Kansas, and I know a couple of judges that I prefer to not sit at the same table with them. Now expand that couple of judges nationwide and there really are quite a few that are not that great. Someone once told me to remember that even though medical doctors are mostly very well respected, half of them graduated in the bottom 50% of their class.
There are a lot of great judges out there, and there are a few total dooshbags, most of my contests judged are in Kansas, and I know a couple of judges that I prefer to not sit at the same table with them. Now expand that couple of judges nationwide and there really are quite a few that are not that great. Someone once told me to remember that even though medical doctors are mostly very well respected, half of them graduated in the bottom 50% of their class.
Slamdunkpro
Babbling Farker
I guess I'm in the minority who think that all this table info is as useful as the history boards on roulette wheels. It's fun to use to play "what if" but otherwise statistically irrelevant. Regardless of what the score sheet says your table history at any contest is always N=1. In other words your history starts from scratch every new contest. Different judges, different table combinations different tastes. This is especially true if the judge average score is also N=1 (just the history for that contest) or if this a combined score average across all 4 meats.
If the judge's average score is cumulative then what does that really tell you? A judge averages 30.2 and you got a 32.5 from them. OK so your flavor profile hits well with that judge, but you already knew that from awards since if you consistently get scores over the judge's mean average you should be getting walks.
If the judge's average score is cumulative then what does that really tell you? A judge averages 30.2 and you got a 32.5 from them. OK so your flavor profile hits well with that judge, but you already knew that from awards since if you consistently get scores over the judge's mean average you should be getting walks.
gettinbasted
is Blowin Smoke!
Agreed. Just another thing to discuss over a beer after awards.
Muzzlebrake
Babbling Farker
So you don't see any value in knowing that the product you turned in scored where it did due to it hitting either a good/hot table or bad/cold table. I think everyone has had that turn in they thought was great that tanked or sucked and they walked and wondered why. Granted I don't think there is much you can do about it but I see value in at least knowing that.
Slamdunkpro
Babbling Farker
There is no value because (1) you don't know the circumstances of those tables and (2) you'll most likely never encounter that set of variables (judges at that table, other entries at that table) again. Back to the roulette example. The odds of any one number coming up are 37-1; if 18 comes up 6 times in a row what are the odds it will come up 18 the next spin? 37-1. The odds always remain the same because they reset every spin. Your contest history within the aspect of judges and tables and all variables (N) is always N=1. It resets every event.
Balls Casten
is Blowin Smoke!
I would disagree ... I think it tells you what kind of food was at that table. All of these remarks are being made without seeing or tasting the entries.
.. and again I think the new scoring is good. I am just afraid of how it will be used to single out judges and influence their scoring.
jaestar
Full Fledged Farker
I would have left the Sam's Club contest last week wondering what went wrong with my brisket even though I cooked a good brisket. Based on past contest I figured a top 5 finish (my average in brisket this year is 168 over 14 contests) as this was one of the better briskets I have cooked this year, but finished 20th OA.
The new sheets showed that my brisket was at table 2. Table 2 had two total top 10 finishes. A 7th in ribs by Pig Skin and an 8th in pork. I talked to Scott before awards and he said he was happy with his ribs...which usually is a bad thing for the rest of us :razz:
At the brisket table I was with Tippy Canoe. Joe said that he cooked a good brisket too but finished 15th OA.
Table 4 had the following rankings (29 teams):
Chicken: 21, 22, 25, 26, 28, 29
Ribs: 7, 21, 23, 27, 29
Pork: 8, 15, 22, 26, 27, 28
Brisket: 15, 20, 26, 27, 28, 29
Now let's play this out another contest or two and say you have the bad luck of hitting the bad table. In the past, I would not have known that. I would probably have started to mess around with my recipe and then waste a couple of more contests trying to correct something that wasn't wrong in the first place.
The new sheets showed that my brisket was at table 2. Table 2 had two total top 10 finishes. A 7th in ribs by Pig Skin and an 8th in pork. I talked to Scott before awards and he said he was happy with his ribs...which usually is a bad thing for the rest of us :razz:
At the brisket table I was with Tippy Canoe. Joe said that he cooked a good brisket too but finished 15th OA.
Table 4 had the following rankings (29 teams):
Chicken: 21, 22, 25, 26, 28, 29
Ribs: 7, 21, 23, 27, 29
Pork: 8, 15, 22, 26, 27, 28
Brisket: 15, 20, 26, 27, 28, 29
Now let's play this out another contest or two and say you have the bad luck of hitting the bad table. In the past, I would not have known that. I would probably have started to mess around with my recipe and then waste a couple of more contests trying to correct something that wasn't wrong in the first place.
Smokin' Joe
is one Smokin' Farker
This is spot on in my opinion and the reason that I think this system is a huge improvement.
As jaestar pointed out at the Woodbury MN event of the bottom 4 placings in each category 12 of 16 came off of table 2. Thats either a trend or a very scary coincidence. (for the record, I'm not complaining about judges, I'm just saying I enjoy having a few additional facts to base any changes or feedback on)
Balls Casten
is Blowin Smoke!
Come on now ... would you really have "come away wondering what went wrong" "started to mess around with my recipe"? OR would you have chalked it up to a bad table and made plans for Rockford.
(poking the bear) You finished in 6th place do you think you should have finished higher?
Balls Casten
is Blowin Smoke!
I agree if thats all its used for.
But what if some of us dont want you guys having more info? Maybe this additional info should only go to teams that have not won a contest. :mrgreen:
ique
is One Chatty Farker
Sorry havent read the entire thread but...
The next thing to add for each judge is "scoring deviation".
Take the average score of the other 5 judges and compare it judge 6 score. Judge6 ScoreDev could be +7% or -5% for example. After say 10 contests judged we can identify judges that score much higher or much lower than average and try to address it. I know at least a couple judges in NEBS land with a ScoreDev of -35% :wink:
Or maybe even better write a judge seating program that mixes together judges so high scoring and low scoring judges are evenly distributed across the tables
The next thing to add for each judge is "scoring deviation".
Take the average score of the other 5 judges and compare it judge 6 score. Judge6 ScoreDev could be +7% or -5% for example. After say 10 contests judged we can identify judges that score much higher or much lower than average and try to address it. I know at least a couple judges in NEBS land with a ScoreDev of -35% :wink:
Or maybe even better write a judge seating program that mixes together judges so high scoring and low scoring judges are evenly distributed across the tables
Smokin' Joe
is one Smokin' Farker
This should happen!
Untraceable
Babbling Farker
would it be easier to handicap tables at the end?
nevermind. if every table was even, there would be even more .0002 spreads
nevermind. if every table was even, there would be even more .0002 spreads
musicmanryann
is Blowin Smoke!
My thinking as well. Once the last judge checks in the rep clicks a button and the computer assigns judges to each table virtually instantly. Again, I'm not sure if the new program is keeping track of scoring across multiple contests or not, but if it is this would seem relatively simple.
