PDA

View Full Version : Is the KCBS judgeing System Broken ?


YankeeBBQ
10-22-2008, 12:35 PM
Just wondering what the majority thinks.

Westexbbq
10-22-2008, 12:38 PM
I have cast my ballot.

The Anchorman
10-22-2008, 01:46 PM
works but could be improved, yes I answered in the poll.

Transformer BBQ
10-22-2008, 01:47 PM
Zero votes for "Yes toss it out and start over "... someone hasn't voted.

Bbq Bubba
10-22-2008, 01:49 PM
From whats been posted here lately, i'd say it's spot on! :icon_sleepy

Rickwieser
10-22-2008, 02:17 PM
this is a sore subject with me. yeah yeah cream rises to the top.or Did they just learn how to work the new system BUT the old system worked it just had too many ties and the kcbs platform was too antiquated to allow decimals to eliminate the ties. now they have a new platform(program) and it can do things that should have been done years ago. Starting at 9 and going down was a good way to do things for all judges. New judges cant always give points but they sure can take away. If it was a decimal judging system there would be virtually no ties. 8.3 9.0, 7.5 etc. Start at 9 and go down. Yes to answer the question its broke its been broke and will probably stay broke. Sad when a group of about 40 people changed the whole system for the masses. Should have been voted on by the whole membership. These are my opinions I realize not everyone will agree.

KC_Bobby
10-22-2008, 02:32 PM
If I vote "It works but could be improved" does that make it sound like a want a change?

I have nothing against the current system (since taking the judging class and having a year of comp q'ing behind me) but I believe most everything can be improved on from time to time and would be open to hearing ideas.

Smokin' Gnome BBQ
10-22-2008, 02:55 PM
everything can be improved. how is the real question.

ihbobry
10-22-2008, 03:18 PM
Option 2&3 are really the same hair just split.

rbinms33
10-22-2008, 03:19 PM
everything can be improved. how is the real question.

Agreed. The decimal idea makes the most sense to me. Even allowing just the .5 would make a big difference.

MilitantSquatter
10-22-2008, 06:49 PM
I like the idea of fractional scoring instead of just whole #'s...

Overall I think it works pretty good.

Buster Dog BBQ
10-22-2008, 08:44 PM
I would just like to see some consistency. I don't feel there should be a range of four to five points in any of the three scoring groups. Yes, I realize one of six chicken pieces can be bad but if you have all 9's and a 5, probably a judge issue.

Big George's BBQ
10-22-2008, 08:49 PM
There are a lot of things they can do to improve it- mentoring at a comp, comment cards to go with scoring, give something to the judges who do the event

Bentley
10-22-2008, 09:59 PM
I would just like to see some consistency. I don't feel there should be a range of four to five points in any of the three scoring groups. Yes, I realize one of six chicken pieces can be bad but if you have all 9's and a 5, probably a judge issue.


Naw...You really think...! KCBS just thinks those other 5 dont know BBQ!

HoDeDo
10-23-2008, 06:52 AM
Thanks Steve

Merl
10-23-2008, 07:44 AM
Consistency has been my favorite project for the past two years. In that direction, Minion and I proposed the definitions to be used with the numbers on the judging slips, which definitions are now on every judging slip.

Will that fix it? No, there will always be that "judge 5". Now I am not picking on judges by saying this. Reps are told to talk to "judge 5" when they see it.

When I talk to judges about their scores, sometimes they have a really valid reason, and I have to agree. Others, I try to suggest they may be inconsistent with the community standard.

I have seen this help some judges. I have seen other judges ignore it. I have turned in a few judges to the CBJ committee because they may be better utilized in another manner than judging.

We do not have a perfect system. I do not know what the perfect system is. When we began at 9, we had the same discussions about "judge 5". I am certain over time, there will be more changes and attempts to improve.

I think this discussion is good and often leads to new ideas.

Merl Whitebook

WannaBeBBQueen
10-23-2008, 08:12 AM
works for us, but there's always room for improvement I agree with Vinny, I think the decimal could work..

Juggy D Beerman
10-23-2008, 09:12 AM
Merl brings up a good point that people forget. The KCBS has always had those rogue judges whose scores were inconsitent with the other five judges at the table. This happened under the old system and still occurs under the present system. All of us hate that low scoring judge. One thing I have suggested over the years is that a score of three or less be defended with a short written comment by that low scoring judge. This way the team might learn from their mistake.

One last comment about those rogue judges. IMHO, these type of judges are not restricted to uncertified and untrained judges. They can even be CBJs.

One thing nobody seems to complain about is the inconsitent judge who judges too high. A main for the old system being discarded were the artificially high scores. The common mindset among judges was, if you could not find anything wrong with the entry, you gave it a score of nine.

I used to run KCBS appearance surveys on Ray Basso's forum. Shortly after the change in judging instructions, I reposted an entry we had judged under the old system, using the new system. The readers were unaware they had judged this entry before. Most of the judges who had previously given scores of nine were now deeming the entry with scores of sevens and eights. There even a few scores of six.

My observation and question at the time was, if it was a nine before, why wasn't it a nine now?

I like the descriptors that accompany each possible score. I think that this is a step in the right direction. As to the idea of using decimals, we already have eight possible scores, we don't need more. A score of seven or less is going to knock you out of the top any way.

One last suggestion I would make is to track the CBJs and see if their scores are inconsitent with the other five judges at the table on a regular basis. If this occurs, then these judges should either be retrained or given volunteer duties other than sitting at the judge's table.

Beers for thought,

Juggy

BBQ_Mayor
10-23-2008, 09:37 AM
When I talk to judges about their scores, sometimes they have a really valid reason, and I have to agree. Others, I try to suggest they may be inconsistent with the community standard.


I think this statement in it's self is a problem. Are we to say what a "community standard" is? And does that Rep know what a standard is?
So as cooks, we should cook to the reps standards and we will score better.

I've been through the judging class and have judged but mainly compete. The contests I have judge are by my standards, which is not right or wrong, but within the paramenters that were given to me in my judging class.

I don't think we will ever get rid of that "judge 5" but if we could get more consistant scores from one contest to the other regardless of Reps, that would be a start.

As far as the poll goes, I think the KCBS judging system is fine the way it is but could use a little tweaking.

Jorge
10-23-2008, 09:39 AM
One last suggestion I would make is to track the CBJs and see if their scores are inconsitent with the other five judges at the table on a regular basis. If this occurs, then these judges should either be retrained or given volunteer duties other than sitting at the judge's table.

Beers for thought,

Juggy

The software for that purpose would be relatively simple. The data required would be the scores, CBJ #, and table #. Depending on the current software being used it probably wouldn't be too difficult to add the new code. After each contest the Rep. could send a data file to KCBS. If the results were analyzed on a weekly basis a consistently high or low judge should appear fairly quickly when compared to their peers assuming they judge on a regular basis..... or demonstrate that there are very few consistent judges.

The difficult part in my opinion would be obtaining a consensus on how far out of the norm a judge would need to be, and how consistently for the software to alert whomever was responsible for dealing with the situation.

cmcadams
10-23-2008, 09:42 AM
I sat next to a judge the other day that gave almost every entry a 9 in appearance, saying later they all looked good. When I asked how one was as good as another, she said appearance didn't really count much anyway. Arghhhh!!! And she had judged several comps.

The reps said they don't want to tell judges how to judge.

I think there should be some sort of instruction each comp in some way, instead of just going over the rules. More detailed guidelines, etc.

I was glad to hear one of the judges make sure the reps said something about taking a piece of cracker and a drink between all entries, not just when they wanted.

Q Haven
10-23-2008, 09:56 AM
I am curious as to why scores are so much lower in the Northeast than in the rest of the country. In New England, a winning score is usually in the 630-640 range. That would be a 40th place score in contests in the rest of the country. Are New England judges being trained differently than in the rest of the country? Or are we just angry, bitter people in the Northeast and that reflects in the scoring?

stlgreg
10-23-2008, 02:30 PM
The problem i have with allowing decimals is you are just allowing too many numbers.

What is the real difference between a 7 and a 8? or an 8 or 9. A good deal right? What the difference between a 8.4 and a 8.7? Too many arbitrary numbers will be given out.

There is a group here in STL that has used a 100 point scale for scoring (0-15 app; 0-25 tenderness; 0-60 taste). They experienced about 34% ties. Sure you could keep the weights and make all categories 0-100 which is what decimals do but you will see other problems pop up.

Changes with the infrastructure would change. How about entry errors? Giving someone two characters (three if you include the decimal) to enter gives more than twice of a chance to have an error.

Sure you could build in the program to catch most things but you cant catch everything. By making the scale 2-9. There is only one digit to appear. You start allowing decimals then you start seeing these issues:
unlimited decimals 6.7 becomes 6.4 on accident.

Correct, If it is entered as 67 instead the program could beep and other double-checking would take care of that!

How much more time would it take to double-check it? More than a little. How much more time are willing to wait for the awards?

I am a numbers freak. After breaking down each possibility of the a KCBS scores, I can really tell how much thought and precision was put into the infrastructure of the system.

Sorry guys Ė math time

For those that do not know the KCBS system is broken down into 7ths.

1/7 is appearance
2/7 is tenderness
4/7 is taste
Then for some reason they take the result and multiply it by 4

The math on this is
14.5% of their score is appearance
28.5% is tenderness
57% is taste.

Or 14.28 Ė 28.57. Ė 57.14

There are 354 different possible combinations of scores a judge can assign to an entry. This is mainly due to the multipliers. By changing 28.57 to 28.58 and 14.28 then we have 729 different possibilities of scores. That will greatly reduce ties, if not eliminate them.

If you think there are too many ties then letís change the appearance or tenderness weight so that 2 appearance points exactly equals one tenderness point.


With all of that being said, things I want to see improved:
∑ On my scoresheet, I want to see the scores of the whole table i was at. I am also curious to see how a "high scoring table" really affects the places in general.
∑ On my scoresheet I want to know the table averages. If I can then go away with a thinking of well i just got the wrong table, then you know I feel better.
∑ I want judges scores to be tracked. If someone is off (letís say their score is dropped more than 40% (we can argue the number later) of the time then letís have someone at least talk to them, shadow them at their next contest, whatever.

Bentley
10-23-2008, 03:23 PM
I have turned in a few judges to the CBJ committee because they may be better utilized in another manner than judging.


As a non-KCBS member, but competitor who competes in KCBS events I would be intrested if you could elaborate on what might happen to the judge, once they have been reported to the CBJ committee?

Jorge
10-23-2008, 03:54 PM
this is a sore subject with me. yeah yeah cream rises to the top.or Did they just learn how to work the new system BUT the old system worked it just had too many ties and the kcbs platform was too antiquated to allow decimals to eliminate the ties. now they have a new platform(program) and it can do things that should have been done years ago. Starting at 9 and going down was a good way to do things for all judges. New judges cant always give points but they sure can take away. If it was a decimal judging system there would be virtually no ties. 8.3 9.0, 7.5 etc. Start at 9 and go down. Yes to answer the question its broke its been broke and will probably stay broke. Sad when a group of about 40 people changed the whole system for the masses. Should have been voted on by the whole membership. These are my opinions I realize not everyone will agree.

If the decimal system was instituted, do you think some judges would be more likely to score entries against each other?

I think they probably would, but that's only my opinion.

So many decisions, and so little time:wink:

Scottie
10-23-2008, 03:59 PM
As a non-KCBS member, but competitor who competes in KCBS events I would be intrested if you could elaborate on what might happen to the judge, once they have been reported to the CBJ committee?


I can tell you that KCBS takes this very seriously. I also know that they have banned and/or recinded CBJ's in the past for their actions. It holds everyone accountable. I think it's a good thing that KCBS does take action. We do not need a rogue judge thinking they can do things their own way and not follow the rules. I we as cooks are expected to follow rules, so should judges...

stlgreg
10-23-2008, 07:35 PM
If the decimal system was instituted, do you think some judges would be more likely to score entries against each other?

I think they probably would, but that's only my opinion.

So many decisions, and so little time:wink:
I do

Merl
10-23-2008, 09:51 PM
I think this statement in it's self is a problem. Are we to say what a "community standard" is? And does that Rep know what a standard is?
So as cooks, we should cook to the reps standards and we will score better.

By community standard, what the Reps are instructed to do is to place all 6 cards out. Look at the averages of the scores at the table. This is what I mean by community standard. Inconsistent judging is +/- 2 points from the mean. Juggy is correct when he says a judge who is 2 points above the mean average is the same as a judge who is 2 points below the mean average.

I do not look at each score. I look at the average of each entry and the overall card. That is when I talk to a CBJ.

I hope this helps to understand the roll of the Contest Rep.

Merl Whitebook

Bentley
10-23-2008, 11:01 PM
I can tell you that KCBS takes this very seriously. I also know that they have banned and/or recinded CBJ's in the past for their actions. It holds everyone accountable. I think it's a good thing that KCBS does take action. We do not need a rogue judge thinking they can do things their own way and not follow the rules. I we as cooks are expected to follow rules, so should judges...
Originally Posted by Merl http://www.bbq-brethren.com/forum/images/buttons/viewpost.gif (http://www.bbq-brethren.com/forum/showthread.php?p=764276#post764276)

I have turned in a few judges to the CBJ committee because they may be better utilized in another manner than judging.

I was hoping since Merl made the statement he might answer it.

I would be intrested if you could elaborate on what might happen to the judge, once they have been reported to the CBJ committee?

Merl
10-24-2008, 08:24 AM
Originally Posted by Merl http://www.bbq-brethren.com/forum/images/buttons/viewpost.gif (http://www.bbq-brethren.com/forum/showthread.php?p=764276#post764276)

I have turned in a few judges to the CBJ committee because they may be better utilized in another manner than judging.

I was hoping since Merl made the statement he might answer it.

I would be intrested if you could elaborate on what might happen to the judge, once they have been reported to the CBJ committee?

Bentley,

You know that is a human resource issues and cannot be discussed publicly. It would be absolutely inappropriate to discuss. I think the important issue is that KCBS addresses the issues.

Merl

Scottie
10-24-2008, 09:29 AM
I am curious as to why scores are so much lower in the Northeast than in the rest of the country. In New England, a winning score is usually in the 630-640 range. That would be a 40th place score in contests in the rest of the country. Are New England judges being trained differently than in the rest of the country? Or are we just angry, bitter people in the Northeast and that reflects in the scoring?


By my scores out there, I would have to say yes.... Yes to all... Maybe it's because of Brady and his knee? Or are people bitter still about Manny? :roll:

I, of course am kidding. I love going out to NEBS country. Need to figure out what I am going to do next year or if it's hitting Lake Placid again. Which I wouldn't mind doing either... As I was finally able to break into the top 10 this year... lol

Q Haven
10-24-2008, 09:52 AM
I'm a Yankees/Giants fan so those things don't bother me :-)

Hopefully we will be able to make it back to Lake Placid this year after missing it last year. That's a great town....and they treat the teams really well.

Bentley
10-24-2008, 10:39 AM
Bentley,

You know that is a human resource issues and cannot be discussed publicly. It would be absolutely inappropriate to discuss. I think the important issue is that KCBS addresses the issues.

Merl


No, I did not. I did not know CBJ's were employees of KCBS.

Thank you for the response.

Scottie
10-24-2008, 11:52 AM
I'm a Yankees/Giants fan so those things don't bother me :-)

Hopefully we will be able to make it back to Lake Placid this year after missing it last year. That's a great town....and they treat the teams really well.


Yankees and Giants? Do they know that at NEBS membership? :shock: I thought the Red Sox cap was part of the NEBS uniform...

As much as I would like to do a new venue, so I can get another state off my list... I do love going back to the oval... Dmitry and gang treat the cookers awesome and my kids just love the whole week/weekend...

Q Haven
10-24-2008, 12:29 PM
Yankees and Giants? Do they know that at NEBS membership? :shock: I thought the Red Sox cap was part of the NEBS uniform...

If it was part of the uniform I would renounce my membership lol....

CTSmokehouse
10-24-2008, 12:37 PM
If it was part of the uniform I would renounce my membership lol....


Same here...lol

Yours in BBQ,

Cliff

barbefunkoramaque
10-24-2008, 04:09 PM
Same here...lol

Yours in BBQ,

Cliff

Of course you could renounce your membership in the KCBS based on principle alone. :shock:

Rookie'48
10-24-2008, 07:59 PM
I agree with the "works but could be better" school of thought. The infamous "Judge #5" does need to explain his low / high scores and I feel that some CBJs let their own personal taste influence the scores too much. I know that "taste" is a very subjective thing but you are supposed to judge it on its own merits as it was presented by the cook.