Tracking Judges

http://www.bbq-brethren.com/forum/showthread.php?t=94021 :-D

This was approved in this months meeting to be started this last (10/15) weekend if I recall...

I have no problem with judges being tracked. If nothing else, hopefully it moves to a computerized system where results get tabulated faster and cooks know why they got all 9's and one 5. If it is because the one piece was bad fine. but hate not knowing what that judge thought to score 4 points lower then everyone else. I just hope judges don't feel they have to score a certain way to not be punished per se.

Just my 2 cents.

So. If I give you a 777 I am safe for any score between 9 and 5. Give me a break. keith

While I also don't have a problem with the tracking of judges, I still have a problem with poor penmanship (IE a '9' being read as a '4').

We may find that we have judges complaining like we do about the scores...

What I would like to see is that judges are given a print out of there tables scores. That would be very educational to me to see how I compare to my fellow judges. Or let us all see scores before we turn them in so we can talk about why we scored things the way we did in real time while the samples are still in front of us. If Joe gave a nine on taste and I gave a 2 I can let Joe have a bite of the piece I got to confirm or tell me I am crazy. Tracking is not the answer real time real school education would help greatly.

mobow said:

So. If I give you a 777 I am safe for any score between 9 and 5. Give me a break. keith

Click to expand...

Not really, if the rest of your table is scoring the entry with 4s and 5s, you'd be more than 2 points outside the table average.

Why would a judge feel threatened by seeing how their scores compare to the table average? Why would a judge be concerned for the sanctioning body to see how their scores compare to the rest of the table? If all you're wanting to do is write down a number so your score falls close to the table average, why judge?

The best way to be "safe" is to take what you do seriously, and to properly judge an entry as you were trained to do. If you do that, you shouldn't have anything to worry about.:thumb:

Smokedelic said:

Not really, if the rest of your table is scoring the entry with 4s and 5s, you'd be more than 2 points outside the table average.

Why would a judge feel threatened by seeing how their scores compare to the table average? Why would a judge be concerned for the sanctioning body to see how their scores compare to the rest of the table? If all you're wanting to do is write down a number so your score falls close to the table average, why judge?

The best way to be "safe" is to take what you do seriously, and to properly judge an entry as you were trained to do. If you do that, you shouldn't have anything to worry about.:thumb:

Click to expand...

Not really if 4 judges give a 5 one judge gives a 4 and I give a 7 the average is 5.166. If they all give 4's the chances of me giving a 7 are pretty slim. My point is this will not do anything to help with the problem. A 4 point spread is way to high. What is needed if we want to see improvement in consistency or at least understanding of differences in score is immediate feedback while the samples are in front everyone. I want my score to be the most accurate it can be. I have judged 25 contest and think I am getting pretty good at it but I don't really know as I never see my scores compared to others unless I do double duty as table captain. I do like the idea of giving me a copy of the tables scores so I can see how I am doing. I just think the tracking and having a KCBS person talk to you is not very helpful.keith

I don't see a problem with the scores being tracked or a rep sitting down and talking to a CBJ who had really off scores. All of suggestions brought up in the BOD will really help and are a good idea. However I still think they are focusing on the result of a problem and not really the problem itself. I still think the focus should be on updating the training class to prevent the problem in the first place instead of trying to correct the problem after the fact.

I personally could setup an appearnce test online in a matter of hours that would include pics of comp boxes and require new cbjs' to give each box a score and why they gave it the score BEFORE they judged a competition. Then you could focus on the really hard part taste and texture. Bottom line is the training class is probably a 6 at most.

KCBS is trying to address an issue that I guess members have brought to their attention...To that I say Bravo...

I judged the Autumn Blaze BBQ in Garnett, KS. last weekend and they informed us of this new tracking policy. I have mixed feelings about this and I would like to know more about why this change was needed. If someone starts giving fewer 8's and 9's, I believe we will be short changing the truly good teams out there. Kind of dumbing down to the average. I have one more contest this year in Butler, MO. and I plan on getting there early to talk to some of the other judges about this.

At an Iowa contest this year I sat next to Ed (Modelmaker) at the judges' table. There was an entry that was mostly pulled pork in the center with some chunks on the edges. We each gave it 8s & 9s while another CBJ gave this same entry 5s & 6s. He was questioned by the Rep about his scoring and he (the judge) said it was one of the worst pork entries that he had ever eaten.

Go figure.

Rookie'48 said:

At an Iowa contest this year I sat next to Ed (Modelmaker) at the judges' table. There was an entry that was mostly pulled pork in the center with some chunks on the edges. We each gave it 8s & 9s while another CBJ gave this same entry 5s & 6s. He was questioned by the Rep about his scoring and he (the judge) said it was one of the worst pork entries that he had ever eaten.

Go figure.

Click to expand...

My first question for that judge would be "if it was that bad why didn't you give it a 2 or 3? Is the worst BBQ you've ever eaten really "average"?"

How would this pertain to a chicken score?

If judge X scored an 8 2 2 (for what that judge believed was raw chicken) would they be flagged for it? If the other 5 judges scored the chicken's taste and tenderness in the 7, 8, 9's should judge X really be flagged?

This definitely opens up a serious discussion and IMO isn't exactly fair to Judge X.

I would very much like to see my scoring pattern over a period of time and also see how I deviate from average or statistical mean for each table/contest/etc.

If I am too far above or below the statistical trend I'd like somebody to call it to my attention.

There is a well known tendency to be nice in scoring inferior entries. A 6 is often given out of kindness, an experienced judge knowing that even a very good cook can have a bad day. Keeping long-term statistics and using a testing tool on low scores might work to "even out" the scoring range, overall.

Finally, I think judges don't reach a comfort and consistency level until about ten contests. New judges could get more feedback and perhaps come up to speed more quickly.

Bring on the statistics!

I find it odd that tracking cbj scores could lead to a disscussion by the rep, or in the end being made to retake a judging class, when a lot of contests have VIP judges that may have had a 10 min. class. The cbj's pay their hotel and other travel expenses.

I have several proposals to bring up at the banquet in jan. Judging is one of the areas. Throughout this season I brought up to KCBS "skewed judging scores" Im amazed that its actually being looked at by the board. My first was at Wesmont this spring, was told that there were 100%CBJ, 3 seasoned judges per table, and 3 judges per table that were judging for the first time. We all know when we turn in sub par entries, but we thought our rib box was spot on. 3 judges gave 999 the other 3 were something like 776 765 655. This is what I brought up to KCBS, as a cook there is no way I can adjust to balance out these scores. I see the problem stemming from the influx of new judges from the PitMaster show.
My suggestion to this is to change the judging certification, Dont make a judge cook with a team to be a master judge, make them cook with a team BEFORE their allowed to judge a contest. Let them see firsthand the time, effort and money spent to get to the final product. They get hands on experience, trying product after turn ins, and we can explain to them why were happy with our turn ins, or not. Every professional competition, be it sports, dancing, singing, has one thing in common... their judges have experience in the field, or some type of backround of the event their judging. BBQ.. anyone with a free Saturday and $75 can judge my product. Their getting their experience on the cooks dime. This is just a suggestion, feel free to comment.

DocStl said:

My first was at Wesmont this spring, was told that there were 100%CBJ, 3 seasoned judges per table, and 3 judges per table that were judging for the first time. We all know when we turn in sub par entries, but we thought our rib box was spot on. 3 judges gave 999 the other 3 were something like 776 765 655.

Click to expand...

I bet I can guess what 3 judges gave you 9's. Bottom line is the KCBS needs to do more in the training class. They spent a lot of time on correct garnish, but not once say here is what a 9 box should look like, an 8 box, a 7 box, a 6 box etc etc.

I still think they should take pics of good and bad boxes put them all online allow the new CBJ's to score at least 10 boxes in each category and give them the results to see how close or how far they are from what other seasoned judges are giving. It's just a little time and effort that will really go a long way. That or show a quick slideshow in the training class.

DocStl said:

I have several proposals to bring up at the banquet in jan. Judging is one of the areas. Throughout this season I brought up to KCBS "skewed judging scores" Im amazed that its actually being looked at by the board. My first was at Wesmont this spring, was told that there were 100%CBJ, 3 seasoned judges per table, and 3 judges per table that were judging for the first time. We all know when we turn in sub par entries, but we thought our rib box was spot on. 3 judges gave 999 the other 3 were something like 776 765 655. This is what I brought up to KCBS, as a cook there is no way I can adjust to balance out these scores. I see the problem stemming from the influx of new judges from the PitMaster show.
My suggestion to this is to change the judging certification, Dont make a judge cook with a team to be a master judge, make them cook with a team BEFORE their allowed to judge a contest. Let them see firsthand the time, effort and money spent to get to the final product. They get hands on experience, trying product after turn ins, and we can explain to them why were happy with our turn ins, or not. Every professional competition, be it sports, dancing, singing, has one thing in common... their judges have experience in the field, or some type of backround of the event their judging. BBQ.. anyone with a free Saturday and $75 can judge my product. Their getting their experience on the cooks dime. This is just a suggestion, feel free to comment.

Click to expand...

I can only speak for Westmont, because I am on the BBQ Committee for the event. There were alot of new CBJ's for the contest. They have had a few judging classes up in these parts, which means new judges. Does it suck? Yes it does. but I guess the way I look at it. It effects everyone. Sure can't argue with Quau winning the event. So he must of been doing something right with the new judges. Did I have some abnormal scoring? Yes i did. I am all charged up about it after seeing my score sheet, but I have forgetten about it by the time I get home. Sorta like golf. Don't worry about your last shot and only worry about your next shot.

Hope all is well Doc.