Automated Scoring

[motoeric]

I know this isn't the standard topic for the comp forum, but it's not an idle question.


I was talking with someone this evening that is actually technically adept (which I am not). We were discussing the future of scoring BBQ events and how the scoring could be expedited.


What flaws do you see in the following:


Each Table Captain has a netbook or other low-end, low cost device. As the judges are done with each category, they use the device to enter their unique judge id and their scores. The device requires the judge to verify that each score is correct on a separate screen for accuracy. Any scores below a certain threshold need to receive an authorization by the table captain (to avoid someone giving all threes or fours).


Each device (from each Table Captain) sends the info to the Reps computer that crunches the numbers and spits out the results.


Table Captains collect the original scoring sheets to use as spot checks against the data entered.


This would make data entry quicker, allow for results to be printed quicker, allow for emailing or texting of individual scores, eliminate the problem of Reps interpreting poor handwriting (is that a 7 or a 9?) and allow for monitoring of unreasonable variance of individual judges scoring over a period of time.


Thoughts?


Eric

 

[Jorge]

Too many variables, too expensive, doesn't address keying errors....

A scantron reader and new cards address the keying error issue. One card reader issued to each rep reduces the hardware needs and costs. A basic reader can process 30+ cards per minute. You eliminate network security issues.

 

[timzcardz]

Yeah, what Jorge said.

 

[Hub]

I agree with Jorge. Technology will advance, but slowly. A scan of #2 pencil lead input will be a logical and affordable next step and won't require great expense or administrative change. The deployment of devices to Table Captains may also be viable but brings up a lot more issues.

 

[musicmanryann]

I am for anything that can streamline the scoring process and minimize probablity of human error. I have discussed something like this with several people in the past, and I like the scantron idea--something I hadn't thought of.

 

[Jorge]

I am for anything that can streamline the scoring process and minimize probablity of human error. I have discussed something like this with several people in the past, and I like the scantron idea--something I hadn't thought of.

It seems to make sense so far, and I'm supposed to talk to some folks about the devices next week. There's a cost involved, and the way I look at it without hard numbers at this point is that one would probably be issued to each Rep. We don't want to be shipping them all over the country as needed because they'd get lost or damaged and not do the job we purchased them for. It streamlines the process and eliminates some of the human element and keying errors. It's worlds faster than what we have now. 50 teams, four categoreies, 200 cards total. We are waiting on the printer for awards to start.

Eric, I'm sorry for the short reply this morning but my dogs had me up and were demanding attention before I could get back to sleep. There are multiple issues that I see with what you've proposed. First, the TC should be concerned with monitoring his/her table and taking care of issues that arise before anything else. Second they should be focused on getting the trays to the table in a timely manner before they cool off any more than they have to, without doing data entry. Those are the two most important tasks a TC has, and to distract from that doesn't benefit the judges or teams in the end in my opinion.

There are also other issues with the technology you are looking at. It's possible, but comes with some major hurdles. How many Reps are capable of setting up such a network, and what's the cost? Do we only accept TCs that have approved devices? Who's responsible for testing and software development? If we purchase the devices, how many have to be fielded on any given weekend? See where I'm going? Beyond that, there are some major security issues. Current encryption, if used is probably good enough for this proposed use. The concern is the security of the network. You'd be surprised how many geeks cook comp BBQ 99.9% could be trusted to not.....play around and the .1% would almost always be caught and identified. My question is why make the system more complicated and expensive than it needs to be when more reliable, secure, accurate, and affordable tech option are available?:becky:

 

[Slamdunkpro]

I've been looking at the scoring / reporting issue for a while now. The biggest issue I see with any type of OMR system is that the score cards aren't in the best environment for OMR. Sauce, finger prints, & moisture just to name a few are detrimental to OMR technology. Couple that with there is no easy verification of scores and I suspect the process might actually be slower.

From having been in the back of house for a number of contests, I've found reproduction and hand collation of score sheets to be a more prevalent bottleneck than the actual data entry portion of the scoring process.

In my view, the quickest way to speed up the awards calculation process is 1) eliminate the need to hand collate the results. Currently, unless there is a collating copier available, the rep has to print out X number of Overall results, X number of ch9icken, etc, then the raw scores, then hand assemble each package. 2) Reduce the number of pages that have to be printed. There is no reason to print 5-7 pages per team for a 25 team contest.

 

[BBQ Bandit]

Too many variables, too expensive, doesn't address keying errors....

A scantron reader and new cards address the keying error issue. One card reader issued to each rep reduces the hardware needs and costs. A basic reader can process 30+ cards per minute. You eliminate network security issues.

Trying to envision that; thinking out loud...
[I used to handle/process scantron sheets in mass quantities]

Judges receive a blank scantron sheet and filling in the spots for tray number and dots between 1 thru 9 for each judging aspect?

Would that might lead to entry error?

Keeping a judging score sheet free from getting juiced/sauced up is hard enough for me. Can see the scanning/reading heads getting gunked up.

 

[Arlin_MacRae]

I hadn't thought of the effect of sauce on a scanned card. The idea is sound, but that does need to be figured into it. It's not rocket science, though.

 

[lcbateman3]

The issues with using tablets and such is they all have to be interconnected. Even if you dock them to get the data from all T.C. there is still computer issues there. You would almost have to send an IT person to every event. What happens if one fails? Or worse, one just dies when getting the data off of it? Those scores are lost.

I do think there is a better way, and the solutions are out there, we just have to find them.

 

[CBQ]

I have visions of Harvard 2010 where the generator running the judge's tent kept stalling, and the laptop battery was bad, so every time the power went out, the reps had to start over with the scores. After a few tries, they ended up plugging the judge's tent into my nearby RV so they could get the scores out.

I'm a gadget guy by nature, but you know what comps can be like - high winds, rain, floods. Having too many devices could be trouble.

I don't understand why we don't have score cards with pre-printed numbers on them, and the judges just circle the one they want. Simple, low tech, but less risk of nines that look like fours, etc.

 

[Smokenstein & monster crew]

eric the madman...

 

[Spydermike72]

The concern is the security of the network. You'd be surprised how many geeks cook comp BBQ 99.9% could be trusted to not.....play around and the .1% would almost always be caught and identified. My question is why make the system more complicated and expensive than it needs to be when more reliable, secure, accurate, and affordable tech option are available?:becky:

Hey I resemble that remark!! :becky::becky:

I agree with Jorge, the security of the network would be an issue. There are those of us out there that do this for a living and I can tell you I could hack it and you would never know about it... Not that I would ever do that, but there are some that arent as ethical as others...

As some one said it isn't rocket science and it is a good topic for discussion. With today's technology we should be able to figure something out reliable and secure with much faster results than hand tabulation...

 

[Muzzlebrake]

From having been in the back of house for a number of contests, I've found reproduction and hand collation of score sheets to be a more prevalent bottleneck than the actual data entry portion of the scoring process.

In my view, the quickest way to speed up the awards calculation process is 1) eliminate the need to hand collate the results. Currently, unless there is a collating copier available, the rep has to print out X number of Overall results, X number of ch9icken, etc, then the raw scores, then hand assemble each package. 2) Reduce the number of pages that have to be printed. There is no reason to print 5-7 pages per team for a 25 team contest.

That is an interesting observation. I remember a couple of years ago @ New Paltz the reps tried to deliver the scores via email in an effort to "go green". I was actually amazed at the amounts of paper required, I never even thought about the collation process. It has to be a daunting ( and mind numbing) task.
I would not be against the distribution of scores via email, I think the flaw in New Paltz if I remember, was we had to wait until Monday to get the scores. Mobile broadband has changed a lot recently, may be more feasible now.

 

[Jorge]

From having been in the back of house for a number of contests, I've found reproduction and hand collation of score sheets to be a more prevalent bottleneck than the actual data entry portion of the scoring process.

In my view, the quickest way to speed up the awards calculation process is 1) eliminate the need to hand collate the results. Currently, unless there is a collating copier available, the rep has to print out X number of Overall results, X number of ch9icken, etc, then the raw scores, then hand assemble each package. 2) Reduce the number of pages that have to be printed. There is no reason to print 5-7 pages per team for a 25 team contest.

Fair point and one that I'd hope is addressed by the new software. It would be simple code to have everything printed as a package for each team.

 

[dmprantz]

I had a question about this at awards yesterday, and I was wondering: Currently does the KCBS software use double entry to confirm score reporting? Does any one know if it's part of the RFP for it to be included in the new software? Certainly that affects time, which I know part of this thread is trying to eliminate, but it would improve any issues with accuracy.

dmp

 

[iampaulb]

Assumptions:
KCBS Events, I guess it could be other org, I am just not that familiar with their processes
I know NOTHING about current KCBS backend procedures for scoring and such
Avg Comp 9Tables
Up to 54 Competitors
KCBS Reps already have laptops to "Manage" this.
Setups are used 50% of the time, 2 out of 4 weekends a year.


HARD COSTS: (I will ignore the back-end costs assuming KCBS has sever space, email capabilities, etc)
Each Table needs a Netbook ($300) for the TC. You will need one for each of the 9 tables, plus a spare. So that's $3000. You will need at least 1 WIFI RTR and a backup so $200 ($100 ea). Mobile Internet Access to KCBS Servers $50 ($100/mo, lets say these are used 1/2 the month). You will also need a secure/safe way to store the equipment (Rolling padded stage boxes) at about $400. For 9 Tables, up to 54 teams, you are looking at $3650 to get this started ($3600 CAPEX; $50+ OPEX).

Now multiply this by the number of events KCBS puts on a month and divide by 2. Lets say KCBS sponsors an average of 25 events a mo.

$3650 x 25/2 = $45k +/-
Just an off the cuff estimate, I am sure I am missing a couple things here. I have a feeling KCBS sponsors way more than 25 events a month!


SECURITY:
With tens of thousands of $$$ on the line, KCBS will really need to beef up the security in order to help ensure the integrity of the contests. I am not a hacker, but I would surmise it would be easier to hack the KCBS Mobile WIFI network than get a winning score at a comp!!

You would need to really look at the potential vulnerabilities of all the systems involved here. I am a biased bastard, but MS Windows has too many holes, so I would suggest Linux, but this is not the most USER friendly OS. Linux is not a perfect solution, as with any OS there are security issues, so the OS would need to be kept up to date.

What about passwords? (WIFI, Laptop, Netbook, tracking application)
They would need to be "strong" in order to deter attacks. But if they are difficult to remember, that will work against the contest.
How do we track change control so that cooks do not hack into laptops and change scores after a meat is judged?



LOGISTICS:
How do you ship the equipment to/from contests?
How do you track inventory?
What happens if something is lost in transit?
Where do we store all this equipment during the off season?
What about increased costs to insure electronics from fire/theft?
Who is going to setup and test this BEFORE an EVENT? In what time, most events are controlled chaos with the reps running around just trying to make the event happen.
Where is the TC Netbook supposed to be kept during the comp? Judging Table square footage is at a premium at every event I have ever been to!


ADMINISTRATION:
Whom is going to design & test the proof of concept?
Whom is going to test vulnerabilities?
Whom is responsible to setup the standards?
Whom is responsible to administer the machines?
Whom is responsible for user & admin documentation?
Whom is going to provide onsite support when things go wrong? (and they will)
Whom is going to continually update the hardware, applications, and documentation?


Summary:
If KCBS is looking to hire someone to run the project, let me know, and I will FW my resume. Combing computers and BBQ, that is a yummy project to be involved in!!

Practically, there are just too many variables here, I just hit the 50,000 foot view in my post here. That is not to say this could not happen, but it would probably take about 6 months and a couple hundred $$$ to really figure out the best solution. By adding in countermeasures (SSH, STRONG Passwords, Intrusion Detection, Firewalls, VLANS, ACLs, L2 Security, L3 Security, etc) you could mitigate most the risks, but never ELIMINATE ALL the risks.

I guess the next question would be, how much does KCBS value the ability to:
+Provide Instant Scoring Feedback to Cooks
+Improve Accountability in Scoring
+Improve customer support to cooks & event organizers
+Help the KCBS Reps do their job more efficiently

Just my 2 cents...
Paul

 

[The_Kapn]

As a very part time judge for KCBS and FBA, I would like to see "fill in the dots" scorecards.
They are scanned into the system by the REPs.

Obvious advantage is that there is no human input once the Judge fills in the dots. Less chance for error and much faster.
Software could catch omissons and out of range scoring.

I think this would be the lowest cost system based on what little I know.

But, it is time to come out of the stone ages on scoring input.

Just a thought.

TIM

 

[BBQ Bandit]

Another aspect on the Scantron/fill-in-the-dots...

How will the scores be adjusted for a DQ at the judging table if the sheets are already filled in?
(less than 6/foreign matter/illegal garnish, etc.)

One way on that line of thought.... BEFORE the dot is filled in... write down the score given... and if there are no further discussions/red flags by the Table Captain... have the dot filled in.

Or... have a second set of dots for... "Is this a DQ" Yes or No

 

[carlyle]

Interesting conversation with many good points illustrating problems with trying to improve what we deal with now.

From what I know, KCBS is coming out with new computer software. It has been in development a long time, and I have no idea when it rolls out or what it is capable of doing.

I would like to see the program come out and be used to determine strength and weaknesses - then let's see if we need further change.

As one who is in the back and sees the process at work I can see how carefully the processes are applied. Works well most of the time. Occasionally in spite of everyone's best efforts, errors can occur.

So I am all for improving what we have right now. But I have not seen anything so far in this thread that makes me enthusiastic.

So, let's get the new program up and running and see what it brings.