And... we're back!

Friday, Oct 26th, 2pm:
EssayTagger.com is back up and seems to be responding normally. See my earlier post and its live updates during the morning's downtime.


What happened?
Google App Engine suffered a worldwide outage around 9:30am (CST) Friday after which they slowly restored services. The outage knocked out major sites like Dropbox, Instagram, Khan Academy, and anyone else running on Google's infrastructure. During most of this outage EssayTagger.com was either inaccessible or experienced excruciatingly slow load times. The site reached stability around 2pm.

Is this normal?
Nope. An outage of this scale is unprecedented. Most tech folks view Google's infrastructure as being as robust and as close to invulnerable as you can get and their track history had borne that out, until this morning.

Are you going to drop Google App Engine now?
For the moment, no. This was an aberration. The realities of website hosting are that downtime happens, no matter which infrastructure you're running on. And, to be honest, I have much more faith in Google's engineers than I do in anyone else -- including myself. Yes, their system failed this morning, but in the brief 11 months of EssayTagger's life, Google App Engine has been remarkably stable and much more reliable than anything else out there.


Read Google's mea culpa, their analysis of what happened, and the new preventative measures they've put in place:
http://googleappengine.blogspot.com/2012/10/about-todays-app-engine-outage.html

"We know you rely on App Engine to create applications that are easy to develop and manage without having to worry about downtime. App Engine is not supposed to go down, and our engineers work diligently to ensure that it doesn’t. [...] We know that hundreds of thousands of developers rely on App Engine to provide a stable, scalable infrastructure for their applications, and we will continue to improve our systems and processes to live up to this expectation."

Site down: Google's servers experiencing problems

Friday, October 26th, 9:53am: 
The site is currently down. We run on Google's "App Engine" (GAE) infrastructure and they are currently experiencing problems that render our site -- and other prominent App Engine sites like KhanAcademy.org -- inaccessible.

Google App Engine status can normally be viewed here, but even the status page is failing to respond.

Needless to say, this is inconvenient but also rare; Google's infrastructure is among the best in the world and they rarely ever see interruptions of their App Engine service.

Interestingly, google.com search service is still functioning (as is blogger.com -- as evidenced by this post being publishable!). Not surprising that they'd have a separate set of servers for their core business.

Follow the latest updates on #GAE via Twitter:
https://twitter.com/search?q=%23GAE&src=hash

And my own Twitter account:
https://twitter.com/KeithMukai


UPDATE 11:10am
EssayTagger.com has begun to respond again, but service is intermittent. Google App Engine is not yet stable.


UPDATE 11:35am
From Google's Max Ross: "At approximately 7:30am Pacific time this morning, Google began experiencing slow performance and dropped connections from one of the components of App Engine.  The symptoms that service users would experience include slow response and an inability to connect to services.  We currently show that a majority of App Engine users and services are affected.  Google engineering teams are investigating a number of options for restoring service as quickly as possible, and we will provide another update as information changes, or within 60 minutes."


UPDATE 12:51pm
From Google's Christina Ilvento: "We are continuing work to correct the ongoing issues with App Engine.  Operation has been restored for some services, while others continue to see slow response times and elevated error rates.  The malfunction appears to be limited to a single component which routes requests from users to the application instance they are using, and does not affect the application instances themselves.

We’ll post another status update as more information becomes available, and/or no later than one hour from now."

EssayTagger.com is now responding more consistently. Cautiously optimistic that we're through the worst of it.


UPDATE 1:45pm
The App Engine status board is looking better. The error spike is returning to more sane levels but the system is still in an "elevated" problem state.

EssayTagger.com performance is still a little unpredictable with intermittent reports of documents that couldn't be uploaded to the system. We rely on Google Docs under the hood to process incoming documents so even if our site is working, this integration point with Google might still see issues.


Update 2:07pm

From Google's Christina Ilvento: "At this point, we have stabilized service to App Engine applications. App Engine is now successfully serving at our normal daily traffic level, and we are closely monitoring the situation and working to prevent recurrence of this incident.

This morning around 7:30AM US/Pacific time, a large percentage of App Engine’s load balancing infrastructure began failing. As the system recovered, individual jobs became overloaded with backed-up traffic, resulting in cascading failures. Affected applications experienced increased latencies and error rates. Once we confirmed this cycle, we temporarily shut down all traffic and then slowly ramped it back up to avoid overloading the load balancing infrastructure as it recovered. This restored normal serving behavior for all applications. 

We’ll be posting a more detailed analysis of this incident once we have fully investigated and analyzed the root cause."

So in theory EssayTagger.com and all other affected websites should be back to full power.

Avoiding Session conflicts on save() due to Hibernate lazy loading

With Java Reflection we can force Hibernate to instantiate lazily-loaded (aka proxy) member objects before we attempt to save the containing entity.

Yesterday I posted my Spring + Hibernate + Cloud SQL sample project code to help other developers get up and running with Hibernate and App Engine.

Today I had to make some DB changes and slightly rearchitected my code. That uncovered a problem. I'm using Spring's OpenSessionInViewFilter to support lazy loading in the view layer. It's dang convenient. And it's currently configured for singleSession=false. That means that it'll use a read-only Session for gets but a separate save/update Session for writes.

Calling getHibernateTemplate().save() had been working fine until my code reshuffling exposed a problem. If an entity has any member objects that have not yet been initialized (due to lazy loading), the save() call will throw a session conflict exception. Specifically:

org.springframework.orm.hibernate3.HibernateSystemException: illegally attempted to associate a proxy with two open Sessions

Here's a simple example:

The problem is that the read-only Session created the lazy-loading proxy Owner within oldCat but hasn't been asked to initialize the proxy yet. Then the save/update Session gets the newCat and tries to access its Owner and now both Sessions are trying to handle the non-initialized proxy, thus the "two open Sessions" complaint.

So folks will generally do something lame like this to get around it:

I'm not judging. I've done it in the past too. But it's clearly a crappy solution. If that chunk of code doesn't need to access any fields in Owner, it shouldn't have to just to make the DAO/ORM layers happy.

So here's my new solution: programmatically search for non-initialized proxies and call getHibernateTemplate().initialize() on them before we try to save.

This requires Reflection to inspect the Cat and look for fields that might need initialization.

Unfortunately my DAO doesn't have permission to access Cat's private member variables. So instead we scan for the public getters and look at their return types. If the return type is a domain object that is mapped by Hibernate, then we "get" it and initialize as needed.

How do we know it's mapped by Hibernate? I'm just filtering on the return type's package name; I only care about DTOs from com.essaytagger.model. The proxies that need initialization are easy to spot--they return true to (obj instanceof HibernateProxy) tests.

Here's the new version of my _GenericDaoHibernateImpl's save() method (my GenericDAO approach is kind of complicated. Track back through the Spring+Hibernate+CloudSQL post first and then see the more in-depth discussion of the original Objectify-based version that was adapted from David Chandler):

This GenericDAO work is getting me more and more into Java Reflection. It's pretty dang cool! But I'm not completely confident in this approach just yet. We'll see if any other odd use cases popup before we call this a win.

What do you think? See any problems down the road?

Spring + Hibernate on Google App Engine's new Cloud SQL!

Full project code for a Spring + Hibernate version of Google's "guestbook" sample webapp that accesses Cloud SQL!

Download the project code!

UPDATE: See the new version of my GenericDAO's save() method to avoid Session conflicts due to lazily-loaded member objects.

I was super-thrilled that Google approved my application to be part of the Cloud SQL limited preview. In gratitude I wanted to share the sample code that I used to do my proof-of-concept tests for accessing a SQL DB in App Engine.

I'm accustomed to using Hibernate with annotations for my ORM layer and was excited to be able to go back to that world in App Engine. But annotations won't work with App Engine as long as they limit the JPA spec to v1.x (I tried and failed to get around the limitation). So that means we're back to using Hibernate Tools and reveng.xml to generate the Hibernate *.hbm.xml mapping files. It's a pain and it feels like it's 2003 again, but whatever. Once you dust off that part of your memory it's not so bad.

How to programatically upload a file to Google Docs when you can't create a java.io.File in App Engine

Google's Data API (gdata) makes accessing their suite of tools and sites a lot easier than making direct HTTP calls to the Web-based service interfaces. But there don't seem to be too many code snippets out there--or at least not enough that do what I want to do.

I have a document that was uploaded through a Servlet and is now sitting in a byte[] array. Now I want to upload that file to my app's Google Docs account. Here's what Google's documentation offers:


Fairly straightforward, but there's one catch: The setFile() method is expecting a java.io.File. Normally that isn't a problem, but if you're developing in Google App Engine (GAE), then you know that you can't create filesystem files.

Digging through the DocumentListEntry code (thank you, Google, for making gdata open source!!) I was able to figure out how to avoid this File creation entirely:

Elegant coding with Objectify, Java generics, and interfaces

David Chandler posted code for an ObjectifyGenericDao on his blog. It takes care of all of the typical Objectify datastore operations you might need to perform on any kind of entity type.

It works great as-is. Here's a code snippet of the generic DAO in action:

Pretty simple. You define the generic type as you directly instantiate the DAO and you get instant access to ObjectifyGenericDao's suite of basic datastore operations.

But directly instantiating any service layer should set off an alarm in your head: "Wait! I'm supposed to code to interfaces!"