Great essay!

At least that’s what the computer thinks

I read an article on techcrunch the other day about a group giving a prize to anyone who can create an automatic scoring algorithm that will grade a student’s essay. Whoa there people! I can tell already that you are drawing your battle lines and digging in for this one. Let em give you some more info and then my take on this whole thing and then let me have it in the comments. Agree, disagree but just click past the link to get to my thoughts and more importantly, the place where you can leave a comment and weigh in on your own.

Kaggle who? and the basics

kaggle.com is a website that allows companies who have statistical/analytical problems and want them outsources. Instead of pairing them up with a group they make it a competition to see who comes up with the best solution. Many companies (Ford, Wikipedia, etc.) use Kaggle to help them with some solutions. In this case, the Hewlett foundation is sponsoring the Automated Student Assessment Prize (ASAP) to find an algorithm that will grade a student’s essay. You can read the Hewlett foundations’s reasons for this here.

So you see Kaggle is just hosting the competition. According to Kaggle the competition has 176 “players” and over 1300 entries (at the time of writing this). That’s a lot of people working on it. The competition started in January and is coming to a close on April 30th. You can read all the details here.

The winner will receive $60,000 for first place, $30,000 for second and third place receives $10,000. The goal here is to see if automated scoring is just as effective as human scoring.

The Good

Being a teacher and keeping up with marking student work is tough. Other subjects have it OK like math and IT as it can be pretty objective. English, in my opinion, has it the worse. Grading essays middle school is difficult, time consuming and just plain tiring. A high school teacher I know said he would give his students an essay (2-4 pages typed) for their final grade. It was due 3 weeks before school ended and he then dedicated the rest of his time grading those 80-100 essays. That is a long time to be grading one assignment. Obviously an automated scoring system will greatly speed this process up. It will take far less time (even if it’s just looking for grammar) than doing it all by hand. For that merit alone, a lot of teachers are smiling ear to ear after reading this.

There is also the consistency factor. As teachers grade, sometimes they notice mistakes they’ve overlooked. This could be anything from an grammar mistake to stating an incorrect fact that seemed OK at the time. This situation may mean having to revisit papers that have already been marked to see if this mistake (that the teacher just noticed) was present there too. It doesn’t happen too often but I don’t know a teacher who hasn’t experienced this situation. Sometimes teachers just decide to continue plowing through papers, hoping students don’t catch the grading inconsistency. If they go back it obviously adds more time to the already huge amount required and then there is still no guarantee that small inconsistencies in the grading haven’t been made.

Programs may miss detailed items or be confused by those items, but it will do so consistently. The program won’t get tired, won’t carelessly miss a detail it caught on the first 50 papers. It will be undeviating in its task at hand. Obviously this is all dependent on the quality of that mystical algorithm that ASAP is looking for, but if it does work (somewhat well) it will be worth its price.

Students can also use this program to get more comfortable with revising and scrutinizing their own work. A lot of students can write an essay and then leave it. Going back over the essay looking for spelling, grammar, punctuation and making sure everything makes sense and flows nicely is something that doesn’t happen as much as it should. Heck, I know I’m guilty of that today! Having a program that can at least get them started with that revising process is a good thing. If students can see that revising is not just an important step but a vital step is crucial. A lesson I need to learn myself.

The Bad

When a student writes an essay, they are writing that essay for an audience. Even if it is just the teacher, there is an intended person on the reading end. I know a number of English teachers and they do find some good pieces of writing (yes, even in the middle school) and take pleasure reading and sharing that student’s work. With a program, this discovery will be greatly diminished. The software may find those with the highest score but miss those pieces of work with a point that no one else may have considered or had a great metaphor to drive home a positing they have taken. These individual moments could be overlooked and lost. That’s a shame.

Also, we are talking about a program. I have yet to use a program that is perfect or never fails (and I use a lot). NEVER! This program will be no different. It will have it’s faults. It will have its bugs. It will screw up and score someone’s essay who typed the alphabet over and over again as the next American classic (well maybe that last one is a bit of an exaggeration) but I think you get my drift. The software will always be a “work in progress.” This will no doubt lead to frustrated teachers, students, administrators, and parents as scores are given that don’t add up.

Also, there are those teachers who will abuse this software as well. Many of us have had the misfortune of knowing those teachers who don’t care all that much about education. Who want to take the quick and easy way out. Who want to do as little work with as little effort as possible. Teachers like that will probably shell out their own money to buy this software and use it to exclusively grade their students’ work in a few hours as opposed to a week. I am happy to say I’ve met less than 3 teachers in my 12 years in education that I can recall who match this description. Regardless, there will be those who will use this program to do all their essay grading and never think twice about it.

My last negative point I can think of, is that students will figure out what the program is looking for and will change their writing style to accommodate for that. This does not sit too well with me. I think writing is probably the best way to truly express oneself. Whether it is a report, thesis or technical writing and if they write for a program that is looking for very specific items in an essay, than I think that will greatly limit their ability to have their voice truly heard. I always liked writing, it gave me an unparalleled platform to reach out and connect with someone. This is why I really like blogging.

In Conclusion

This technology is coming, let’s not kid ourself. It may be a decade or more away but it is definitely coming. This particular prize is probably for a standardized testing program (on a large scale), not individual schools or teachers. I like the idea of using this type of technology. I think it will be a great time saver for the normal English teacher and can be a great tool for students (if they are given access to it). There are some definite negatives here, but I am trying to see through everything to the end game. I think this program (when it comes to fruition) will lead to more students revising their essays.

When it first comes out it will probably be inadequate and very buggy, but the demand will hopefully swell and more companies will jump in the game. More companies, more products, more choices and more competition will lead to better and better products. I am not suggesting that it will replace a teacher, nor am I suggesting that teachers wait around for this program to come out. I just think that as tools go, this one will greatly help the way children (or people in general) approach revising their writing. A step that is often skipped by people (ahem-yours truly).

As always, this isn’t going to replace a teacher. It’s a tool, a resource and should be used as such. I think I’m write (ha ha I love my puns) but I could be wrong. At any rate, what do you have to say about this? What is your opinion? We at IT Babble are dying to hear what you have to say.