New KCBS Scoring Program Test

[Outnumbered]

Like it a lot. Not sure I see the value in knowing the number vs. the blind number however. Seems like overkill to me. What value does that really provide?

[Grizmt]

Originally Posted by Slamdunkpro

Interesting, I've never seen it done at the TC level. That would seem to be a lot of box handling with the potential for dropped boxes. Here the boxes are always shuffled and checked for repeat tables before the TC's get their trays.

The procedure I've seen is this with a couple of minor variations:

1. Turn in hits the turn in table and is placed in the tray along with whatever number of clams that table will be judging which could be anywhere from 5 to as high as 8 (that I've seen) depending on how many judges are on site.

2.The numbers are changed from team number to KCBS number using whatever "factor" was generated for that contest.

3.Most times it is double checked for accuracy.

4.The tray is brought inside or set on a TC table right inside the tent depending on the configuration of the tent and how many TC's there are.

5.The TC's then check the box #'s for boxes that have hit their table before and swap out with other TC's if there are any repeats.

6. TC's then carry the tray to the table and start the judging process.

[Outnumbered]

Originally Posted by ModelMaker

There is a commitee in charge of designing, debugging, testing, and implementing the new KCBSCORE program. It is headed by BOD member Mike Budai who when each step is accomplished then makes a monthly report to the entire BOD at the monthly meeting.
I don't find it suprising that the entire BOD was not aware of a beta test of the new scoring system as Mr. Budai was just doing his job getting the system up and running.
I'm very certain the board will be fully informed of progress at the next BOD meeting.
Don't be all hating at KCBS for doing what we ask of them.
Ed

Sounds to me like, and I'm totally uneducated in the BOD history here, but it is operating as a BOD should. Boards set strategy, not tactics.

The smaller group is testing a new system from a tactical standpoint after the BOD set the strategy that they may need to look at improving the scoring system.

If it works, they can bring it the entire BOD for approval and full adoption then. If there is no beta test, how can the BOD have any data to support their decision either way.

[Jorge]

Originally Posted by olewarthog

But shouldn't the objective be NOT to have "good" & "bad" tables? Assigned seating means that a "good" or "bad" table remains as such throughout the contest. Random seating means that the same 5 or 6 "good" judges and the same 5 or 6 "bad" judges do not determine the winners & losers.

It absolutely should be. My point has been that if you begin moving judges around, from table to table, you risk creating additional tables that score high or low. I haven't seen any safeguards mentioned to prevent that from happening.

If a system that moves judges based on how they score a category, in comparison to others at the same table, you are looking at additional issues. Once you start singling judges out based on their scores you are likely to see a change in the way they score, and there is no guarantee that it will be a reasonable correction. If a judge has been a point low, how does anyone know that they won't start scoring two points higher to avoid being singled out and moved to another table in front of their peers?

The new software can become a very powerful tool. With power comes responsibility. It will take some time to really find out what the data says. Some trends are likely to become very evident early on, and some of those trends may change dramatically over time as the data gathered grows to proved a more complete and accurate view of what is actually happening.

There are also risks associated with telling a judge how far outside the norm they are without sufficient data. Being an outlier one day may indicate that 5 other people were off that day, if over time that outlier is more consistent than those other 5 judges. If we create a system where judges adjust their scoring from week to week, the data and system will suffer.

My personal opinion is that the responsible thing to do is take a look at a reasonable data set, see what can be learned from it, and then assess the options available to improve the system.

My .02

[mobow]

Originally Posted by Carnivorous Endeavors BBQ

Mobow,

I write this with all due respect but and I understand what you are saying but it was more about the general concept than trying to get caught up in the details. If the new system won't be effective until 30-100 contests judged then it would seemingly be useless as it takes most folks are really long time to achieve that.

I do not disagree with you at all. The points I am wanting to add to this conversation are that good decisions require accurate conclusions and accurate conclusions can only occur if we have good data. Good data takes time. Yes, maybe a year or so. That does not make it useless. Just not as fast as we might like it. keith

[Gowan]

Originally Posted by Grizmt

The procedure I've seen is this with a couple of minor variations:

1. Turn in hits the turn in table and is placed in the tray along with whatever number of clams that table will be judging which could be anywhere from 5 to as high as 8 (that I've seen) depending on how many judges are on site.

2.The numbers are changed from team number to KCBS number using whatever "factor" was generated for that contest.

3.Most times it is double checked for accuracy.

4.The tray is brought inside or set on a TC table right inside the tent depending on the configuration of the tent and how many TC's there are.

5.The TC's then check the box #'s for boxes that have hit their table before and swap out with other TC's if there are any repeats.

6. TC's then carry the tray to the table and start the judging process.

Close, but that's not the procedure we use in Georgia.

1) Numbers are changed first. Usually multiple people are involved to avoid delays and double check for errors.

2) Turn-in personnel sort the boxes into the trays, referring to the checkoff list the Table Captain brought back after the last category to avoid placing the any box number on a table twice. (If it's the first category, boxes are sorted randomly.)

3) TC retrieves their tray along with the checkoff list. They cross off each box in the tray from the list. If any repeats are encountered, they confer with the Rep to see if a swap is warranted. (sometimes due to low team counts, etc. a repeat is unavoidable)

4) TC presents the boxes to judges for scoring.

5) When all score cards are turned in to the Rep, TC returns the tray with the checkoff list to the turn-in table for the next category.

[Gowan]

Almost forgot:

I do have one big problem with the beta scoring sheets. The alternate number should never be listed, as this breaks the blind. If you cross reference the alternate number to team names like that, then you will have teams and judges comparing notes on entries between judging and awards.

If you are going to do that, you may as well forget renumbering completely.

[Uomograsso]

Originally Posted by CivilWarBBQ

Close, but that's not the procedure we use in Georgia.

1) Numbers are changed first. Usually multiple people are involved to avoid delays and double check for errors.

2) Turn-in personnel sort the boxes into the trays, referring to the checkoff list the Table Captain brought back after the last category to avoid placing the any box number on a table twice. (If it's the first category, boxes are sorted randomly.)

3) TC retrieves their tray along with the checkoff list. They cross off each box in the tray from the list. If any repeats are encountered, they confer with the Rep to see if a swap is warranted. (sometimes due to low team counts, etc. a repeat is unavoidable)

4) TC presents the boxes to judges for scoring.

5) When all score cards are turned in to the Rep, TC returns the tray with the checkoff list to the turn-in table for the next category.

In all the contests I have judged or TC'ed the TC's receive the boxes and check to make sure that their table has not received a box from that team before. I have only heard of one pair of reps that do the duplicate checking/re-ordering.

I usually do one or two contests a month. I will admit that even though I try to maintain consistent standards, my judging can vary from contest to contest. Some contests I may judge "harder" than others depending on my mood. Judging for appearance and tenderness allow for the most objectiveness. Does a box look good, is the meat presented neatly, does the rib meat come off the bone cleanly, can you pull the brisket apart or is it a rubber band, is the pork mushy, etc. etc. Taste is almost purely subjective. How can you train someone what good BBQ should taste like? If you can come up with a consensus about what good BBQ should taste like I'll do my best to apply it when I judge.

[pahutchens]

As a Judge and Table Captain I've seen the luck and had a few discussions with other judges/TC. Sometimes you have a hot a table because your table 'lucked' out and got a string of contenders. One contest this summer had the best string of ribs that I could recall and another contest ribs were politely described as dismal.

Perhaps a better example. Sam's club we had 5 of 6 pork just money muscle only. Two other judges (different tables) I remarked about it too said there's were more 'standard' pulled pork and little to no money muscle. Not saying Miron Mixon is right that you need money muscle to win. Butt I thinks some cooks have really bought in to it.

As a TC I was told I'm the Rep's rep. If my table has wide swings in scores I bring it to their attention and the judges are monitored a little closer to see it a trend is there that is not supported by the rest of the table. They don't tell them what to score but remind them of HOW to score.

[smoke-n-my-i's]

Originally Posted by pahutchens

As a Judge and Table Captain I've seen the luck and had a few discussions with other judges/TC. Sometimes you have a hot a table because your table 'lucked' out and got a string of contenders. One contest this summer had the best string of ribs that I could recall and another contest ribs were politely described as dismal.

Perhaps a better example. Sam's club we had 5 of 6 pork just money muscle only. Two other judges (different tables) I remarked about it too said there's were more 'standard' pulled pork and little to no money muscle. Not saying Miron Mixon is right that you need money muscle to win. Butt I thinks some cooks have really bought in to it.

As a TC I was told I'm the Rep's rep. If my table has wide swings in scores I bring it to their attention and the judges are monitored a little closer to see it a trend is there that is not supported by the rest of the table. They don't tell them what to score but remind them of HOW to score.

I totally do not agree with you on this one.... my last contest, 30 teams, my pork was overdone and the MM was pretty much mush... I used larger than normal pulled, and got a 5th place..... so, with NO money muscle at all, how would you explain your theory of you have to have MM to win?

[Jorge]

Originally Posted by smoke-n-my-i's

I totally do not agree with you on this one.... my last contest, 30 teams, my pork was overdone and the MM was pretty much mush... I used larger than normal pulled, and got a 5th place..... so, with NO money muscle at all, how would you explain your theory of you have to have MM to win?

I think his point was that many teams seem to have bought into that theory, not that it was accurate.

[pahutchens]

Originally Posted by Jorge

I think his point was that many teams seem to have bought into that theory, not that it was accurate.

Yeah that's what I thought I said too

[crd26a]

In theory (okay, the theory in my mind on this one), the issue will be addressed in the seating of judges at the beginning of a contest. If they are able to track high judges, low judges, etc., for the experienced judges at a contest, each table should be "balanced" prior to seating the judges. This will take away allowing a judge to sit where they please, however, it would put the tables on an even keel for the known judges. To me, this will become where the evening out is.

This will still allow for the new or inexperienced judges to be wild cards. But if we can get 4 of the 6 on average down to a known balance, we "should" see more consistent judging across the tables going forward. I'd much rather compete knowing that the average scores on most of the judges is balanced and taking a shot at 1-2 wild cards vs. a table of who knows what.

And as a side note, I'm still willing to bet we'll see the historically strong teams still be strong. It just could help level out some of the playing field.

[Grizmt]

It'll be interesting to see how the data plays out over time, especially when up until recently there were some (what I consider) wide swings in what judges were told about scoring when in training class and at some events.

This brings up a question for y'all, is a judge who's considered a hard or low scorer incorrect or following what he/she was taught just like the easy/high scoring judge is doing.
The reason I ask this is as many people know for a while (at least out here in the west) judges were instructed to basically not score below a 5/6 unless it's completely and totally inedible. The reason given was "it would be an insult after all the hard work and money that the teams put in to get to the judges tent with an entry".

Given that instruction wouldn't or couldn't that cause a certain section of judges to automatically score higher and the judges that had not heard that instruction actually use the full 2-9 system meaning they could be scoring consistently lower?
I'd suggest that regional data sampling would probably be a good idea. I'd almost bet that you'd see the data swing from one region to another and most likely from one category to another like chicken being scored harder than let's say beef in TX or beef harder than pork in TN. Just thinking about regional tastes is all and no, I will NOT give an example of how the West Coast would score since KCBS doesn't have a "Tofurkey" category.

[Rooter-Tooter BBQ]

ultimately the only way to resolve this in which i have no idea how to do it and im sure nobody else does is: when our boxes go to a table we are competing against those six boxes. To show who is truly the best is to let each judge taste 40 pieces of chicken, ribs, pork, and brisket. Then you will really know who is the best. But dang, i would get full plus i would forget what the other 39 pieces tasted like, but hey that could be a good thing. Who knows, i just like being able to be there even if we don't win. I mean there is nothing like having stories to tell after a comp...like just by chance your trailer flips and the cooker door is open and 3 of your 4 slabs of ribs fall on the ground or maybe just maybe fall into the soapy sink. Im just saying, not that happened or nothing. LOL