KCBS judge seating changes

[Robert]

I've been told, by someone that was in a position to know, that at one time there was a proposal to do it via video having it produced by a college or university. The theory being that it would standardize instruction. It just needed board approval, which did not come.

Jorge,

I know that has been talked about by numerous outside of the board folks for years. It seems like the best way to eliminate any bias of any given instructor. Of course, after the video instruction you have the inevitable follow up questions. And the only way to address those questions would be with an FAQ database so that you could eliminate any moderator bias for those answers. In theory, the video method of instruction should make judge certification more generic with everybody being instructed the same way.

As for seating judges based on scores, I can see how it might even out the scoring somewhat, but I still feel you will have TOD and TOA regardless. If the goal is to even out the wild swings of TOD versus TOA will the end result be tighter scoring? Will tighter scoring lead to more ties that can only be decided by a computer coin flip? Just wait until that starts to be the norm.

If you recall, when they changed the 9 and down starting point, a 180 score became very rare which I think was the goal. Despite dire predictions, the cream still rose to the top. Somehow, scoring creep entered into the equation and the result is 180 scores have become more prevalent again. Surely not as many as before the change, but no longer rare today. How did that happen? Why did that happen?

So what does KCBS do? Will they ever reach scoring nirvana?

Robert

 

[Jorge]

Jorge,

I know that has been talked about by numerous outside of the board folks for years. It seems like the best way to eliminate any bias of any given instructor. Of course, after the video instruction you have the inevitable follow up questions. And the only way to address those questions would be with an FAQ database so that you could eliminate any moderator bias for those answers. In theory, the video method of instruction should make judge certification more generic with everybody being instructed the same way.

As for seating judges based on scores, I can see how it might even out the scoring somewhat, but I still feel you will have TOD and TOA regardless. If the goal is to even out the wild swings of TOD versus TOA will the end result be tighter scoring? Will tighter scoring lead to more ties that can only be decided by a computer coin flip? Just wait until that starts to be the norm.

If you recall, when they changed the 9 and down starting point, a 180 score became very rare which I think was the goal. Despite dire predictions, the cream still rose to the top. Somehow, scoring creep entered into the equation and the result is 180 scores have become more prevalent again. Surely not as many as before the change, but no longer rare today. How did that happen? Why did that happen?

So what does KCBS do? Will they ever reach scoring nirvana?

Robert

It's important for me to say that I don't know what KCBS has done so far. I don't KNOW what their new seating plan is. Just what I've heard. I don't know if they analyzed the data before developing that plan. I do know that I haven't seen the data and I'm just making some guesses.

I think that reliance on a scoring average has a greater likelihood of leading to additional issues. Scoring inflation is one but we are already close to the ceiling. Using a score average without considering standard deviation introduces the opportunity to create a TOA. If it happens by chance that's one thing, but creating it is another.

My personal opinion without the benefit of looking at multiple years of captured data, is that several things need to happen. CBJ instruction has to be standardized as much as possible. There needs to be a program to try to keep the various generations of judges on the same page. Most importantly, in my opinion, there needs to be an effort to get judges to use more of the scoring range available to them. I recognize that as a group cooks have gotten better. But, they haven't gotten that much better. I believe that the emphasis should be on identifying accuracy rather than consistency. A judge that uses the range and accurately scores each entry is of greater value to me than the judge that chooses between 8 and 9. If what I'd propose ever came to pass there would be a sizable percentage of cooks that would howl. Not because the standings changed dramatically, but because they were scoring lower even though their place in the field remained constant.

The problem(s) and solution(s) are all related to each other. Find out what the captured data says. Standardize instruction and implement changes as indicated by the analysis. Implement a program to keep judges on the same page. Implement a system to identify trends and act on them before they become a problem. Use the data to measure success or lack thereoff. This sort of thing is the norm today using data science. All of us may not use it individually, but it touches everyone daily.

I'm just guessing. I don't know. I don't know if anyone does. The first person to analyze the data will be the first person to have a clue.

 

[rorkin]

The scores below 6 have become pretty useless..
Judges know that a 6 will kill an entry.. So many judges are reluctant to use 3,4,5.
Instead 6 becomes "I did not like it" rather than average.
I find that if I use a 5 on an entry that deserves it ,my scores are lower than table average often.. The entry is scored low by entire table but it looks like 6 is the the mark of disaproval

 

[ModelMaker]

It's important for me to say that I don't know what KCBS has done so far. I don't KNOW what their new seating plan is. Just what I've heard. I don't know if they analyzed the data before developing that plan. I do know that I haven't seen the data and I'm just making some guesses.

I think that reliance on a scoring average has a greater likelihood of leading to additional issues. Scoring inflation is one but we are already close to the ceiling. Using a score average without considering standard deviation introduces the opportunity to create a TOA. If it happens by chance that's one thing, but creating it is another.

My personal opinion without the benefit of looking at multiple years of captured data, is that several things need to happen. CBJ instruction has to be standardized as much as possible. There needs to be a program to try to keep the various generations of judges on the same page. Most importantly, in my opinion, there needs to be an effort to get judges to use more of the scoring range available to them. I recognize that as a group cooks have gotten better. But, they haven't gotten that much better. I believe that the emphasis should be on identifying accuracy rather than consistency. A judge that uses the range and accurately scores each entry is of greater value to me than the judge that chooses between 8 and 9. If what I'd propose ever came to pass there would be a sizable percentage of cooks that would howl. Not because the standings changed dramatically, but because they were scoring lower even though their place in the field remained constant.

The problem(s) and solution(s) are all related to each other. Find out what the captured data says. Standardize instruction and implement changes as indicated by the analysis. Implement a program to keep judges on the same page. Implement a system to identify trends and act on them before they become a problem. Use the data to measure success or lack thereoff. This sort of thing is the norm today using data science. All of us may not use it individually, but it touches everyone daily.

I'm just guessing. I don't know. I don't know if anyone does. The first person to analyze the data will be the first person to have a clue.

When I was still on the CBJ Committee and they were just starting to keep judges scoring data and the cooks were clapping their hands thinking this data influx was going to be a Godsend to fix the "judging problem".
The chair, CEO, and anybody else who had a voice repeatedly said this data capture was just for information, not to be published or used to identify out of sync judges. Now we have judges reports, TOA's and TOD's.
Must have changed their minds???
Ed

 

[thirdeye]

The CBJ 50 question continuing education review is good tool, and it just took less than 20 minutes for me to complete it. I'm curious why the PowerPoint presentation used in CBJ classes is not available for viewing by CBJ's as another continuing education tool?

Obviously it would require some sign-in credentials (as the 50 question review does), and would be protected from downloading. Since there would be no Q&A interaction like a typical judging class has, maybe a FAQ section could follow, and updated annually with additional information.

 

[Slamdunkpro]

I'm curious why the PowerPoint presentation used in CBJ classes is not available for viewing by CBJ's as another continuing education tool?

KCBS probably doesn't own it.:becky:

 

[rorkin]

KCBS probably doesn't own it.:becky:

Too funny.. But stranger things have happened