And... we're back!

Friday, Oct 26th, 2pm:
EssayTagger.com is back up and seems to be responding normally. See my earlier post and its live updates during the morning's downtime.


What happened?
Google App Engine suffered a worldwide outage around 9:30am (CST) Friday after which they slowly restored services. The outage knocked out major sites like Dropbox, Instagram, Khan Academy, and anyone else running on Google's infrastructure. During most of this outage EssayTagger.com was either inaccessible or experienced excruciatingly slow load times. The site reached stability around 2pm.

Is this normal?
Nope. An outage of this scale is unprecedented. Most tech folks view Google's infrastructure as being as robust and as close to invulnerable as you can get and their track history had borne that out, until this morning.

Are you going to drop Google App Engine now?
For the moment, no. This was an aberration. The realities of website hosting are that downtime happens, no matter which infrastructure you're running on. And, to be honest, I have much more faith in Google's engineers than I do in anyone else -- including myself. Yes, their system failed this morning, but in the brief 11 months of EssayTagger's life, Google App Engine has been remarkably stable and much more reliable than anything else out there.


Read Google's mea culpa, their analysis of what happened, and the new preventative measures they've put in place:
http://googleappengine.blogspot.com/2012/10/about-todays-app-engine-outage.html

"We know you rely on App Engine to create applications that are easy to develop and manage without having to worry about downtime. App Engine is not supposed to go down, and our engineers work diligently to ensure that it doesn’t. [...] We know that hundreds of thousands of developers rely on App Engine to provide a stable, scalable infrastructure for their applications, and we will continue to improve our systems and processes to live up to this expectation."

Site down: Google's servers experiencing problems

Friday, October 26th, 9:53am: 
The site is currently down. We run on Google's "App Engine" (GAE) infrastructure and they are currently experiencing problems that render our site -- and other prominent App Engine sites like KhanAcademy.org -- inaccessible.

Google App Engine status can normally be viewed here, but even the status page is failing to respond.

Needless to say, this is inconvenient but also rare; Google's infrastructure is among the best in the world and they rarely ever see interruptions of their App Engine service.

Interestingly, google.com search service is still functioning (as is blogger.com -- as evidenced by this post being publishable!). Not surprising that they'd have a separate set of servers for their core business.

Follow the latest updates on #GAE via Twitter:
https://twitter.com/search?q=%23GAE&src=hash

And my own Twitter account:
https://twitter.com/KeithMukai


UPDATE 11:10am
EssayTagger.com has begun to respond again, but service is intermittent. Google App Engine is not yet stable.


UPDATE 11:35am
From Google's Max Ross: "At approximately 7:30am Pacific time this morning, Google began experiencing slow performance and dropped connections from one of the components of App Engine.  The symptoms that service users would experience include slow response and an inability to connect to services.  We currently show that a majority of App Engine users and services are affected.  Google engineering teams are investigating a number of options for restoring service as quickly as possible, and we will provide another update as information changes, or within 60 minutes."


UPDATE 12:51pm
From Google's Christina Ilvento: "We are continuing work to correct the ongoing issues with App Engine.  Operation has been restored for some services, while others continue to see slow response times and elevated error rates.  The malfunction appears to be limited to a single component which routes requests from users to the application instance they are using, and does not affect the application instances themselves.

We’ll post another status update as more information becomes available, and/or no later than one hour from now."

EssayTagger.com is now responding more consistently. Cautiously optimistic that we're through the worst of it.


UPDATE 1:45pm
The App Engine status board is looking better. The error spike is returning to more sane levels but the system is still in an "elevated" problem state.

EssayTagger.com performance is still a little unpredictable with intermittent reports of documents that couldn't be uploaded to the system. We rely on Google Docs under the hood to process incoming documents so even if our site is working, this integration point with Google might still see issues.


Update 2:07pm

From Google's Christina Ilvento: "At this point, we have stabilized service to App Engine applications. App Engine is now successfully serving at our normal daily traffic level, and we are closely monitoring the situation and working to prevent recurrence of this incident.

This morning around 7:30AM US/Pacific time, a large percentage of App Engine’s load balancing infrastructure began failing. As the system recovered, individual jobs became overloaded with backed-up traffic, resulting in cascading failures. Affected applications experienced increased latencies and error rates. Once we confirmed this cycle, we temporarily shut down all traffic and then slowly ramped it back up to avoid overloading the load balancing infrastructure as it recovered. This restored normal serving behavior for all applications. 

We’ll be posting a more detailed analysis of this incident once we have fully investigated and analyzed the root cause."

So in theory EssayTagger.com and all other affected websites should be back to full power.

Latest Update: New and improved free trial!

Let's jump straight to the good stuff:

The new free trial will allow instructors to grade up to 99 essays in one assignment!


Our original free trial allowed for only a single class set of essays (~25). The original idea was to encourage teachers to start small. As a classroom teacher I advocated for the small trial approach to minimize any possible classroom disasters.

Don't get me wrong, I think EssayTagger is a remarkably robust system, but anytime a new workflow or new technology is involved, there's always the risk of things going wrong. And as a frazzled teacher I'd rather see only one section crash and burn than all three!


But then came the feedback
That's all nice in theory, but in reality teachers were more unhappy about creating inconsistencies between their sections by singling out just one of them to try EssayTagger. It's too much hassle to have different sections in different places or systems; second period is over here, but fourth and sixth periods are over there.


So we changed it
Our new approach now lets the instructors decide if they want to have all of their sections switch to EssayTagger or just experiment with one or two sections.

The only new limitation is that free trial users can only have one assignment in the system at a time. And even if you delete and start over with a new assignment, the 99 essay limit continues to count down. I think that's a reasonable compromise.


Existing free trial users
Your free trial accounts have been automatically upgraded to the 99 essay limit. If you have more than one assignment in the system, don't worry, you get to keep them; they've been "grandfathered" in so you don't lose your existing data. You just won't be able to create any new assignments as a free trial user.


Your thoughts?
Being a classroom teacher myself, I try to bias all company decisions toward what's best for teachers and their students. I hope this improvement is further proof of that commitment. As always, I'm eager to hear your thoughts and suggestions.

Signs of the Grammar Apocalypse: #1

Screencap from Sunday night's Chargers-Saints game after Saints quarterback Drew Brees broke Johnny Unitas' record of 47 consecutive games with a TD pass.

Just indescribably horrible. We are doomed.