And... we're back!

Friday, Oct 26th, 2pm:
EssayTagger.com is back up and seems to be responding normally. See my earlier post and its live updates during the morning's downtime.


What happened?
Google App Engine suffered a worldwide outage around 9:30am (CST) Friday after which they slowly restored services. The outage knocked out major sites like Dropbox, Instagram, Khan Academy, and anyone else running on Google's infrastructure. During most of this outage EssayTagger.com was either inaccessible or experienced excruciatingly slow load times. The site reached stability around 2pm.

Is this normal?
Nope. An outage of this scale is unprecedented. Most tech folks view Google's infrastructure as being as robust and as close to invulnerable as you can get and their track history had borne that out, until this morning.

Are you going to drop Google App Engine now?
For the moment, no. This was an aberration. The realities of website hosting are that downtime happens, no matter which infrastructure you're running on. And, to be honest, I have much more faith in Google's engineers than I do in anyone else -- including myself. Yes, their system failed this morning, but in the brief 11 months of EssayTagger's life, Google App Engine has been remarkably stable and much more reliable than anything else out there.


Read Google's mea culpa, their analysis of what happened, and the new preventative measures they've put in place:
http://googleappengine.blogspot.com/2012/10/about-todays-app-engine-outage.html

"We know you rely on App Engine to create applications that are easy to develop and manage without having to worry about downtime. App Engine is not supposed to go down, and our engineers work diligently to ensure that it doesn’t. [...] We know that hundreds of thousands of developers rely on App Engine to provide a stable, scalable infrastructure for their applications, and we will continue to improve our systems and processes to live up to this expectation."

Site down: Google's servers experiencing problems

Friday, October 26th, 9:53am: 
The site is currently down. We run on Google's "App Engine" (GAE) infrastructure and they are currently experiencing problems that render our site -- and other prominent App Engine sites like KhanAcademy.org -- inaccessible.

Google App Engine status can normally be viewed here, but even the status page is failing to respond.

Needless to say, this is inconvenient but also rare; Google's infrastructure is among the best in the world and they rarely ever see interruptions of their App Engine service.

Interestingly, google.com search service is still functioning (as is blogger.com -- as evidenced by this post being publishable!). Not surprising that they'd have a separate set of servers for their core business.

Follow the latest updates on #GAE via Twitter:
https://twitter.com/search?q=%23GAE&src=hash

And my own Twitter account:
https://twitter.com/KeithMukai


UPDATE 11:10am
EssayTagger.com has begun to respond again, but service is intermittent. Google App Engine is not yet stable.


UPDATE 11:35am
From Google's Max Ross: "At approximately 7:30am Pacific time this morning, Google began experiencing slow performance and dropped connections from one of the components of App Engine.  The symptoms that service users would experience include slow response and an inability to connect to services.  We currently show that a majority of App Engine users and services are affected.  Google engineering teams are investigating a number of options for restoring service as quickly as possible, and we will provide another update as information changes, or within 60 minutes."


UPDATE 12:51pm
From Google's Christina Ilvento: "We are continuing work to correct the ongoing issues with App Engine.  Operation has been restored for some services, while others continue to see slow response times and elevated error rates.  The malfunction appears to be limited to a single component which routes requests from users to the application instance they are using, and does not affect the application instances themselves.

We’ll post another status update as more information becomes available, and/or no later than one hour from now."

EssayTagger.com is now responding more consistently. Cautiously optimistic that we're through the worst of it.


UPDATE 1:45pm
The App Engine status board is looking better. The error spike is returning to more sane levels but the system is still in an "elevated" problem state.

EssayTagger.com performance is still a little unpredictable with intermittent reports of documents that couldn't be uploaded to the system. We rely on Google Docs under the hood to process incoming documents so even if our site is working, this integration point with Google might still see issues.


Update 2:07pm

From Google's Christina Ilvento: "At this point, we have stabilized service to App Engine applications. App Engine is now successfully serving at our normal daily traffic level, and we are closely monitoring the situation and working to prevent recurrence of this incident.

This morning around 7:30AM US/Pacific time, a large percentage of App Engine’s load balancing infrastructure began failing. As the system recovered, individual jobs became overloaded with backed-up traffic, resulting in cascading failures. Affected applications experienced increased latencies and error rates. Once we confirmed this cycle, we temporarily shut down all traffic and then slowly ramped it back up to avoid overloading the load balancing infrastructure as it recovered. This restored normal serving behavior for all applications. 

We’ll be posting a more detailed analysis of this incident once we have fully investigated and analyzed the root cause."

So in theory EssayTagger.com and all other affected websites should be back to full power.

Latest Update: New and improved free trial!

Let's jump straight to the good stuff:

The new free trial will allow instructors to grade up to 99 essays in one assignment!


Our original free trial allowed for only a single class set of essays (~25). The original idea was to encourage teachers to start small. As a classroom teacher I advocated for the small trial approach to minimize any possible classroom disasters.

Don't get me wrong, I think EssayTagger is a remarkably robust system, but anytime a new workflow or new technology is involved, there's always the risk of things going wrong. And as a frazzled teacher I'd rather see only one section crash and burn than all three!


But then came the feedback
That's all nice in theory, but in reality teachers were more unhappy about creating inconsistencies between their sections by singling out just one of them to try EssayTagger. It's too much hassle to have different sections in different places or systems; second period is over here, but fourth and sixth periods are over there.


So we changed it
Our new approach now lets the instructors decide if they want to have all of their sections switch to EssayTagger or just experiment with one or two sections.

The only new limitation is that free trial users can only have one assignment in the system at a time. And even if you delete and start over with a new assignment, the 99 essay limit continues to count down. I think that's a reasonable compromise.


Existing free trial users
Your free trial accounts have been automatically upgraded to the 99 essay limit. If you have more than one assignment in the system, don't worry, you get to keep them; they've been "grandfathered" in so you don't lose your existing data. You just won't be able to create any new assignments as a free trial user.


Your thoughts?
Being a classroom teacher myself, I try to bias all company decisions toward what's best for teachers and their students. I hope this improvement is further proof of that commitment. As always, I'm eager to hear your thoughts and suggestions.

Signs of the Grammar Apocalypse: #1

Screencap from Sunday night's Chargers-Saints game after Saints quarterback Drew Brees broke Johnny Unitas' record of 47 consecutive games with a TD pass.

Just indescribably horrible. We are doomed.

Announcing: Free Common Core Rubric Creation Tool!

We're super-proud to announce the release of our new tool that helps teachers create Common Core-aligned rubrics! Open to the public, totally free.

EssayTagger's Common Core Rubric Creation Tool

http://www.essaytagger.com/commoncore

You are ahead of the curve and are working hard to align your curriculum to Common Core. But assessing and tracking your students' progress within Common Core is difficult -- and nigh impossible to do for essays.

I spent the whole dang summer wrestling with the standards, trying to figure out how to incorporate them into real-world, practical writing rubrics.

My initial approach was to try to coax the actual text of the standards into a more rubric-friendly format. But teachers shouldn't have to waste their time adapting the W.8.1a text just to be able to include "Thesis" on their rubrics.

Instead just evaluate "Thesis" like you normally would but add, "Oh, and by the way, 'Thesis' is part of W.8.1a." This is where the tool comes in to help you.

On teacher accountability, pt1: The trouble with bad data

In Part 1 I lay out the case against teacher accountability measures via "value-added" analysis of standardized test score data. In part 2 I offer practical compromises.

Here in the Chicagoland area we are in the fourth day of the Chicago Teachers Union (CTU) strike that is making national headlines.

I did my Master of Education and teacher certification program at the University of Illinois-Chicago. Not surprisingly, a lot of my former classmates are current Chicago Public School (CPS) teachers. I spoke with them last night as they returned from a day out on the picket lines.

They made it clear that this was about fighting a flawed teacher evaluation system that puts undo emphasis on their students' standardized test scores. They also have serious concerns about the push to privatize the public school system. Then are the more tangible things they're fighting for like reduced class sizes (raise your hand if you think 38 teenagers in one room can be productive at anything).

The media and the average Joe on the street think this is about money or benefits or the teachers stubbornly refusing any form of accountability. This is incorrect.


Let's talk about accountability. It's important.
Accountability matters. Teachers should be held to high standards and should be judged by the quality of their work.

Understand that teachers aren't fighting accountability; they're fighting a particular form of accountability that is of dubious value and may indeed be deeply flawed.

On teacher accountability, pt2: Possible compromises

In part 1 I laid out the case against the current method of teacher accountability via value-added analysis. Here I offer what I think are reasonable compromises.

This focus on quantifiable standardized test scores is not going to go away. Some form of accountability linked to test scores is unavoidable. Period. I leave it to the statisticians to refine the analysis and reduce that 53% margin of error.

But here are some practical solutions to incorporate this data while controlling for its flaws: