Originally Posted by olewarthog
But shouldn't the objective be NOT to have "good" & "bad" tables? Assigned seating means that a "good" or "bad" table remains as such throughout the contest. Random seating means that the same 5 or 6 "good" judges and the same 5 or 6 "bad" judges do not determine the winners & losers.
It absolutely should be. My point has been that if you begin moving judges around, from table to table, you risk creating additional tables that score high or low. I haven't seen any safeguards mentioned to prevent that from happening.
If a system that moves judges based on how they score a category, in comparison to others at the same table, you are looking at additional issues. Once you start singling judges out based on their scores you are likely to see a change in the way they score, and there is no guarantee that it will be a reasonable correction. If a judge has been a point low, how does anyone know that they won't start scoring two points higher to avoid being singled out and moved to another table in front of their peers?
The new software can become a very powerful tool. With power comes responsibility. It will take some time to really find out what the data says. Some trends are likely to become very evident early on, and some of those trends may change dramatically over time as the data gathered grows to proved a more complete and accurate view of what is actually happening.
There are also risks associated with telling a judge how far outside the norm they are without sufficient data. Being an outlier one day may indicate that 5 other people were off that day, if over time that outlier is more consistent than those other 5 judges. If we create a system where judges adjust their scoring from week to week, the data and system will suffer.
My personal opinion is that the responsible thing to do is take a look at a reasonable data set, see what can be learned from it, and then assess the options available to improve the system.