Tracking Judges

[Munkee's Pit]

Just noticed this in the BOD minutes, and was wondering what others think about it.

Judges Tracking and Education program:
Merl Whitebook made a motion that beginning the weekend of October 15, 2010 that the judges tracking toggle on the scoring system be implemented.

That Contest Reps be required to input the CBJ number in the scoring program.

That all contest results will be evaluated utilizing the existing KCBS judges tracking program by the KCBS office and kept confidential.
That the results of each contest be evaluated, records maintained and the results be kept confidential, by KCBS.. That when it appears that a CBJ statistically is inconsistent in scoring (+/- 2 from the mean of the overall contest results,) at two or more contest, in a 12 month period, the CBJ will be mentored by the CBJ Chairperson. If inconsistent judging continues, a third time the CBJ will be suspended until completing a CBJ class at no cost. Should the problems continue then the matter shall be brought to the Board for removal or further action by the board. The motion was seconded by Ed Roith.

Following discussion Merl Whitebook called for the vote.

The motion was seconded by Don Harwell.
Vote:
10 yes, 0 no 0 abstentions

[goodsmokebbq]

http://www.bbq-brethren.com/forum/showthread.php?t=94021 :-D

[Divemaster]

This was approved in this months meeting to be started this last (10/15) weekend if I recall...

[Buster Dog BBQ]

I have no problem with judges being tracked. If nothing else, hopefully it moves to a computerized system where results get tabulated faster and cooks know why they got all 9's and one 5. If it is because the one piece was bad fine. but hate not knowing what that judge thought to score 4 points lower then everyone else. I just hope judges don't feel they have to score a certain way to not be punished per se.

Just my 2 cents.

[mobow]

So. If I give you a 777 I am safe for any score between 9 and 5. Give me a break. keith

[Divemaster]

While I also don't have a problem with the tracking of judges, I still have a problem with poor penmanship (IE a '9' being read as a '4').

We may find that we have judges complaining like we do about the scores...

[mobow]

What I would like to see is that judges are given a print out of there tables scores. That would be very educational to me to see how I compare to my fellow judges. Or let us all see scores before we turn them in so we can talk about why we scored things the way we did in real time while the samples are still in front of us. If Joe gave a nine on taste and I gave a 2 I can let Joe have a bite of the piece I got to confirm or tell me I am crazy. Tracking is not the answer real time real school education would help greatly.

[Smokedelic]

So. If I give you a 777 I am safe for any score between 9 and 5. Give me a break. keith

Not really, if the rest of your table is scoring the entry with 4s and 5s, you'd be more than 2 points outside the table average.

Why would a judge feel threatened by seeing how their scores compare to the table average? Why would a judge be concerned for the sanctioning body to see how their scores compare to the rest of the table? If all you're wanting to do is write down a number so your score falls close to the table average, why judge?

The best way to be "safe" is to take what you do seriously, and to properly judge an entry as you were trained to do. If you do that, you shouldn't have anything to worry about.:thumb:

[mobow]

Not really, if the rest of your table is scoring the entry with 4s and 5s, you'd be more than 2 points outside the table average.

Why would a judge feel threatened by seeing how their scores compare to the table average? Why would a judge be concerned for the sanctioning body to see how their scores compare to the rest of the table? If all you're wanting to do is write down a number so your score falls close to the table average, why judge?

The best way to be "safe" is to take what you do seriously, and to properly judge an entry as you were trained to do. If you do that, you shouldn't have anything to worry about.:thumb:
Not really if 4 judges give a 5 one judge gives a 4 and I give a 7 the average is 5.166. If they all give 4's the chances of me giving a 7 are pretty slim. My point is this will not do anything to help with the problem. A 4 point spread is way to high. What is needed if we want to see improvement in consistency or at least understanding of differences in score is immediate feedback while the samples are in front everyone. I want my score to be the most accurate it can be. I have judged 25 contest and think I am getting pretty good at it but I don't really know as I never see my scores compared to others unless I do double duty as table captain. I do like the idea of giving me a copy of the tables scores so I can see how I am doing. I just think the tracking and having a KCBS person talk to you is not very helpful.keith

[LGHT]

I don't see a problem with the scores being tracked or a rep sitting down and talking to a CBJ who had really off scores. All of suggestions brought up in the BOD will really help and are a good idea. However I still think they are focusing on the result of a problem and not really the problem itself. I still think the focus should be on updating the training class to prevent the problem in the first place instead of trying to correct the problem after the fact.

I personally could setup an appearnce test online in a matter of hours that would include pics of comp boxes and require new cbjs' to give each box a score and why they gave it the score BEFORE they judged a competition. Then you could focus on the really hard part taste and texture. Bottom line is the training class is probably a 6 at most.

[Bentley]

KCBS is trying to address an issue that I guess members have brought to their attention...To that I say Bravo...

[davidh9946]

I judged the Autumn Blaze BBQ in Garnett, KS. last weekend and they informed us of this new tracking policy. I have mixed feelings about this and I would like to know more about why this change was needed. If someone starts giving fewer 8's and 9's, I believe we will be short changing the truly good teams out there. Kind of dumbing down to the average. I have one more contest this year in Butler, MO. and I plan on getting there early to talk to some of the other judges about this.

[Rookie'48]

At an Iowa contest this year I sat next to Ed (Modelmaker) at the judges' table. There was an entry that was mostly pulled pork in the center with some chunks on the edges. We each gave it 8s & 9s while another CBJ gave this same entry 5s & 6s. He was questioned by the Rep about his scoring and he (the judge) said it was one of the worst pork entries that he had ever eaten.

Go figure.

[Slamdunkpro]

At an Iowa contest this year I sat next to Ed (Modelmaker) at the judges' table. There was an entry that was mostly pulled pork in the center with some chunks on the edges. We each gave it 8s & 9s while another CBJ gave this same entry 5s & 6s. He was questioned by the Rep about his scoring and he (the judge) said it was one of the worst pork entries that he had ever eaten.

Go figure.
My first question for that judge would be "if it was that bad why didn't you give it a 2 or 3? Is the worst BBQ you've ever eaten really "average"?"

[Crash]

How would this pertain to a chicken score?

If judge X scored an 8 2 2 (for what that judge believed was raw chicken) would they be flagged for it? If the other 5 judges scored the chicken's taste and tenderness in the 7, 8, 9's should judge X really be flagged?

This definitely opens up a serious discussion and IMO isn't exactly fair to Judge X.

[Hub]

I would very much like to see my scoring pattern over a period of time and also see how I deviate from average or statistical mean for each table/contest/etc.

If I am too far above or below the statistical trend I'd like somebody to call it to my attention.

There is a well known tendency to be nice in scoring inferior entries. A 6 is often given out of kindness, an experienced judge knowing that even a very good cook can have a bad day. Keeping long-term statistics and using a testing tool on low scores might work to "even out" the scoring range, overall.

Finally, I think judges don't reach a comfort and consistency level until about ten contests. New judges could get more feedback and perhaps come up to speed more quickly.

Bring on the statistics!

[New Pal Frank]

I find it odd that tracking cbj scores could lead to a disscussion by the rep, or in the end being made to retake a judging class, when a lot of contests have VIP judges that may have had a 10 min. class. The cbj's pay their hotel and other travel expenses.:confused:

[DocStl]

I have several proposals to bring up at the banquet in jan. Judging is one of the areas. Throughout this season I brought up to KCBS "skewed judging scores" Im amazed that its actually being looked at by the board. My first was at Wesmont this spring, was told that there were 100%CBJ, 3 seasoned judges per table, and 3 judges per table that were judging for the first time. We all know when we turn in sub par entries, but we thought our rib box was spot on. 3 judges gave 999 the other 3 were something like 776 765 655. This is what I brought up to KCBS, as a cook there is no way I can adjust to balance out these scores. I see the problem stemming from the influx of new judges from the PitMaster show.
My suggestion to this is to change the judging certification, Dont make a judge cook with a team to be a master judge, make them cook with a team BEFORE their allowed to judge a contest. Let them see firsthand the time, effort and money spent to get to the final product. They get hands on experience, trying product after turn ins, and we can explain to them why were happy with our turn ins, or not. Every professional competition, be it sports, dancing, singing, has one thing in common... their judges have experience in the field, or some type of backround of the event their judging. BBQ.. anyone with a free Saturday and $75 can judge my product. Their getting their experience on the cooks dime. This is just a suggestion, feel free to comment.

[LGHT]

My first was at Wesmont this spring, was told that there were 100%CBJ, 3 seasoned judges per table, and 3 judges per table that were judging for the first time. We all know when we turn in sub par entries, but we thought our rib box was spot on. 3 judges gave 999 the other 3 were something like 776 765 655.

I bet I can guess what 3 judges gave you 9's. Bottom line is the KCBS needs to do more in the training class. They spent a lot of time on correct garnish, but not once say here is what a 9 box should look like, an 8 box, a 7 box, a 6 box etc etc.

I still think they should take pics of good and bad boxes put them all online allow the new CBJ's to score at least 10 boxes in each category and give them the results to see how close or how far they are from what other seasoned judges are giving. It's just a little time and effort that will really go a long way. That or show a quick slideshow in the training class.

[Scottie]

I have several proposals to bring up at the banquet in jan. Judging is one of the areas. Throughout this season I brought up to KCBS "skewed judging scores" Im amazed that its actually being looked at by the board. My first was at Wesmont this spring, was told that there were 100%CBJ, 3 seasoned judges per table, and 3 judges per table that were judging for the first time. We all know when we turn in sub par entries, but we thought our rib box was spot on. 3 judges gave 999 the other 3 were something like 776 765 655. This is what I brought up to KCBS, as a cook there is no way I can adjust to balance out these scores. I see the problem stemming from the influx of new judges from the PitMaster show.
My suggestion to this is to change the judging certification, Dont make a judge cook with a team to be a master judge, make them cook with a team BEFORE their allowed to judge a contest. Let them see firsthand the time, effort and money spent to get to the final product. They get hands on experience, trying product after turn ins, and we can explain to them why were happy with our turn ins, or not. Every professional competition, be it sports, dancing, singing, has one thing in common... their judges have experience in the field, or some type of backround of the event their judging. BBQ.. anyone with a free Saturday and $75 can judge my product. Their getting their experience on the cooks dime. This is just a suggestion, feel free to comment.


I can only speak for Westmont, because I am on the BBQ Committee for the event. There were alot of new CBJ's for the contest. They have had a few judging classes up in these parts, which means new judges. Does it suck? Yes it does. but I guess the way I look at it. It effects everyone. Sure can't argue with Quau winning the event. So he must of been doing something right with the new judges. Did I have some abnormal scoring? Yes i did. I am all charged up about it after seeing my score sheet, but I have forgetten about it by the time I get home. Sorta like golf. Don't worry about your last shot and only worry about your next shot.

Hope all is well Doc.

[Richtee]

KCBS is trying to address an issue that I guess members have brought to their attention...To that I say Bravo...

Bingo. I have experienced 8-9-6's. A hear-say (I know the witness) story:

Lady at a judging class says "I don't like pork, do I have to judge that?"

TRUE story.

[paydabill]

My question if you got a bad piece of chicken or the rib that was at the end of a slab. How many of us had to use two slabs to finish out a box. You know the top 4 are the ones you want to eat, but the second slab is over done. Because I am judge 6, I have to go through all that.

Sounds like crap to me.

[Richtee]

My question if you got a bad piece of chicken or the rib that was at the end of a slab. How many of us had to use two slabs to finish out a box. You know the top 4 are the ones you want to eat, but the second slab is over done. Because I am judge 6, I have to go through all that.

Sounds like crap to me.
Well...at least you'll know WHY then, eh.

[DocStl]

I can only speak for Westmont, because I am on the BBQ Committee for the event. There were alot of new CBJ's for the contest. They have had a few judging classes up in these parts, which means new judges. Does it suck? Yes it does. but I guess the way I look at it. It effects everyone. Sure can't argue with Quau winning the event. So he must of been doing something right with the new judges. Did I have some abnormal scoring? Yes i did. I am all charged up about it after seeing my score sheet, but I have forgetten about it by the time I get home. Sorta like golf. Don't worry about your last shot and only worry about your next shot.

Hope all is well Doc.

Thanks for the well wishes Scottie! I dont have a case of sour grapes, I knew going in. Skewed judging didnt stop us from competing 21 times this year, and yes its fair for all. But, now it seems that there is more luck of the draw on which table your box lands on. I have different theories on why some of us have all over the board scores, and some are far more consistant (Scottie, Quau, Pellet Envy) I believe that certain cooks have the right sauce/spice recipie that satisfies a larger number of pallets. Too sweet scores well on some tables, heat on others, but the ones who have a flavor that satisfies the masses will fair better. I cooked with Mike (Quau) a few times this year, and some of the scoring got him too. couple weeks back got something like 39th out of 40 in pork or something like that. It happens. Anyway best to you and your cause! Much respect.

[The_Kapn]

Here is my "generic" opinion.

All during my career in aviation, I was constantly reviewed/evaluated or reviewing/evaluating others.
Never bothered me a bit.
It just kept me on my toes to know the right stuff and keep the right skills.
Now, we were paid professionals, so I understand the difference to volenteer judges.

I even did a couple of years as a Gymnastic Judge for the USGF.
Rigorous training and constant evaluation.
All training and such was at our expense.
When judging, we were only compensated for a small part of expenses.
Volenteers just like BBQ Judges.

USGF judges have to be "in-range" on each set of scores or a conference is called with the head judge to explain and compromise on the proper score.
Constant evaluation and feedback and it never hurt anyones (almost anyones) feelings.

If KCBS or FBA wants to track my scoring, I am fine with that.
And I would actually like to see how I and my fellow judges agree or disagree (on a macro scale).

JMHO

BTW-- I am one who sees "grading creep" in scoring. That is totally normal. Periodically, the governing body has to adjust the "norm" in an attempt to get back to real grading.
That is a conversation for another day.

TIM

[DocStl]

My question if you got a bad piece of chicken or the rib that was at the end of a slab. How many of us had to use two slabs to finish out a box. You know the top 4 are the ones you want to eat, but the second slab is over done. Because I am judge 6, I have to go through all that.

Sounds like crap to me.

I dont use the end bone off any slabs, and I agree with chicken, one piece can vary from the other easier than the other 3 meats. Judge 6, if he got a bad piece and scored lower, would be thrown out. Moot point. but too many times there are 3 perfect scores at the same table that there are 3 below average scores. Thats the issue at hand

[Balls Casten]

It would be interesting to see what Quau (or any of the top teams) thought about this. In 32 cooks (aside from Little Rock & Royal, 227 & 490 teams) The worst they finished over all was 13th. I have only met the man a couple times but I'm guessing he doesnt think he should have had all top 5 finishes. Heck may he does. :?:

[Gadragonfly]

First I want to say that I hate it when my big fat fingers hit the wrong key and I delete a really nice reply that states exactly what I want to say :mad2::rant:</whine>

My suggestion to this is to change the judging certification, Dont make a judge cook with a team to be a master judge, make them cook with a team BEFORE their allowed to judge a contest. Let them see firsthand the time, effort and money spent to get to the final product.

I think that this idea is one that has led to “scoring creep”. I do not believe that the idea that “time and money spent” should have any effect on the scores that I give to the products presented to me for judging. I have heard this comment in more judging tents than not – “time and money” and “any score below a 7 eliminates a team from a respectable overall finish”. I believe it is my responsibility to give each team my most carefully considered and honest opinion of the sample presented to me. I do believe that a judge should cook with a team sometime within the first 10 contests judged but to give a judge a perspective of different cooking styles, flavor profiles and preparation methods. I would hope that the three judges that gave you low scores did so because that was what they honestly believed your product deserved based on the knowledge that they had at that moment. (Keeping in mind that I’ve seen quite a few very critical “seasoned” judges) At most contests I’ve judged (50+) the judges’ sign-in takes place between 9 – 9:30 with the judges’ meeting at 10 or 10:30. After the meeting there is anywhere between 30 minutes to 1 hour before the first turn-in and I would suggest that we use that time to have organized, open discussion about contests we’ve judged recently and product presented. A time to answer questions that new judges might have or to discuss controversial ideas – “should sauce dribbled in a v-shape over chicken be considered marking?”. I would also suggest that it is very difficult for a judge with less than 10 contests under their belt to know how to find a team that would allow them to hang out for the weekend. If teams want judges to have cooking experience early on then there should be a method of posting teams open to the idea. I would like to see a place on the KCBS website that judges can go to and find a team and that the pit boss can post restrictions – i.e. drinking allowed, encouraged, not allowed during completion; must like animals and/or babies and teenagers….

But this is just my opinion and that and 10 cents won’t buy you a cup of coffee anymore, but until anyone comes up with a better idea I plan to continue to give scores based on my honest, considerate but objectively subjective opinion.

[DocStl]

[QUOTE=Gadragonfly;1436832]First I want to say that I hate it when my big fat fingers hit the wrong key and I delete a really nice reply that states exactly what I want to say :mad2::rant:</whine>



I think that this idea is one that has led to “scoring creep”. I do not believe that the idea that “time and money spent” should have any effect on the scores that I give to the products presented to me for judging. I have heard this comment in more judging tents than not – “time and money” and “any score below a 7 eliminates a team from a respectable overall finish”. I believe it is my responsibility to give each team my most carefully considered and honest opinion of the sample presented to me. I do believe that a judge should cook with a team sometime within the first 10 contests judged but to give a judge a perspective of different cooking styles, flavor profiles and preparation methods. I would hope that the three judges that gave you low scores did so because that was what they honestly believed your product deserved based on the knowledge that they had at that moment. (Keeping in mind that I’ve seen quite a few very critical “seasoned” judges) At most contests I’ve judged (50+) the judges’ sign-in takes place between 9 – 9:30 with the judges’ meeting at 10 or 10:30. After the meeting there is anywhere between 30 minutes to 1 hour before the first turn-in and I would suggest that we use that time to have organized, open discussion about contests we’ve judged recently and product presented. A time to answer questions that new judges might have or to discuss controversial ideas – “should sauce dribbled in a v-shape over chicken be considered marking?”. I would also suggest that it is very difficult for a judge with less than 10 contests under their belt to know how to find a team that would allow them to hang out for the weekend. If teams want judges to have cooking experience early on then there should be a method of posting teams open to the idea. I would like to see a place on the KCBS website that judges can go to and find a team and that the pit boss can post restrictions

First, I applaud you for being the type of judge we all want to see judging contests. kudos to you :clap2:
As for this type of idea being a cause for scoring creep, I disagree, point being is to give new judges the overall feel, experience of comps. So, maybe I could have worded it differently :roll: And yes as a cook we know anything below a 7 will wreck your finish, have I ever turned in a product that deserved a 6? Absolutly! It happens. But I know when I close the box for turn in if its below average.
I judge, table captain, organize, and compete, I see it come from all sides. Im not saying all judges need to be retrained, educated different, etc. Just pointing out different ways for new judges to get more exposure to the sport. The cooking with a team is not to show them how much "time and money" we spend on comps, but gives them a peek into our world up front and let them see how serious we are about contests. So take your time and be serious as a judge.
As a judge myself, Ive seen judges score down 2 points because the sauce was sweet, or spicey. Ive heard judges say they never score more than a 7 on anything because their husbands bbq is better than any they ever judged. These are the ones that should be re-educated. KCBS must be seeing a trend in skewed judging also, or they wouldnt be looking at other options and ideas.

[RedPig]

KCBS is trying to address an issue that I guess members have brought to their attention...To that I say Bravo...
What he said.

[Ford]

but too many times there are 3 perfect scores at the same table that there are 3 below average scores. Thats the issue at hand
I guess you want all 6 judges to be reviewed in this case. A 50-50 split indicates there could be a problem with all 6 judges. Maybe the entry should have been scored in the 6-8 range. We see 9's and say that's what I deserve and what's wrong with those other judges.

I personally like tracking variances from the average. If at that contest it was just one judge that was way off then maybe they need some counseling but if 3 then it shouldn't be a problem. And the system says repeated deviations from the average not just one contest. We all know that there are reasons for one low score that make the judge right.

[mobow]

I guess you want all 6 judges to be reviewed in this case. A 50-50 split indicates there could be a problem with all 6 judges. Maybe the entry should have been scored in the 6-8 range. We see 9's and say that's what I deserve and what's wrong with those other judges.

I personally like tracking variances from the average. If at that contest it was just one judge that was way off then maybe they need some counseling but if 3 then it shouldn't be a problem. And the system says repeated deviations from the average not just one contest. We all know that there are reasons for one low score that make the judge right.

That is what worries me about the proposed system. There is no way to take this into account. I judge 15 to 20 contest a year. Two of these low score but it is right situations and I am in trouble. keith