FINAL UPDATE 5:31pm
The last lingering effects of the Google App Engine server problems seem to have been cleared up and our own testing now shows full functionality restored.
This sort of downtime is frustrating, but it's still only the second Google App Engine outage since we launched EssayTagger 14 months ago. All sites suffer some downtime but we still believe in Google's reliability and ability to react and recover faster and more robustly than we could if we were managing our own server hardware.
UPDATE 1:23pm
Message from Google:
"We are still working to resolve the issue related to Google App Engine serving. At this point error error rates for affected applications should be declining. We will provide another status update by 11:30 AM PST."
EssayTagger is responding again, albeit slowly. Grading app is still severely impacted.
UPDATE 12:07pm
Google App Engine servers continue to see problems and it seems to have spread beyond the backend task queue. They've rescinded their earlier resolution note and are now only saying "We will update this message shortly when the incident has been resolved."
These server issues are now affecting the main EssayTagger site.
UPDATE 10:35am
Message from Google:
"This morning some Google App Engine applications reported elevated error rates and increased latency. This issue should be resolved as of 8:10 AM US/Pacific time. We apologize for the inconvenience and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better."
There still seems to be some intermittent slowdowns for backend processes (uploading an essay, marking an essay as graded). Though Google says the issue is resolved, I still recommend caution.
ORIGINAL POST 1/15 10:28am
At approximately 9am CST Google's App Engine servers suffered a problem with their "task queue" service which EssayTagger uses for behind-the-scenes processing. The majority of the site has not been affected.
Specific interruptions occurred on the student upload page which relies on the task queue to process incoming essay files. The other major impact was that the grading app was unable to process essays when they were marked as graded. Normal grading activity (evaluating rubric elements, adding comments, etc) should not have been affected.
The task queue service is currently intermittent. Take care if using the grading app and keep an eye on the "synced" or "syncing..." message at the top right. If it remains on "syncing..." for more than a few seconds, pause before doing any more work. Your work is only guaranteed to be saved when you see the "synced" message.
Updates will be posted as new information arises.
The last lingering effects of the Google App Engine server problems seem to have been cleared up and our own testing now shows full functionality restored.
This sort of downtime is frustrating, but it's still only the second Google App Engine outage since we launched EssayTagger 14 months ago. All sites suffer some downtime but we still believe in Google's reliability and ability to react and recover faster and more robustly than we could if we were managing our own server hardware.
UPDATE 1:23pm
Message from Google:
"We are still working to resolve the issue related to Google App Engine serving. At this point error error rates for affected applications should be declining. We will provide another status update by 11:30 AM PST."
EssayTagger is responding again, albeit slowly. Grading app is still severely impacted.
UPDATE 12:07pm
Google App Engine servers continue to see problems and it seems to have spread beyond the backend task queue. They've rescinded their earlier resolution note and are now only saying "We will update this message shortly when the incident has been resolved."
These server issues are now affecting the main EssayTagger site.
UPDATE 10:35am
Message from Google:
"This morning some Google App Engine applications reported elevated error rates and increased latency. This issue should be resolved as of 8:10 AM US/Pacific time. We apologize for the inconvenience and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better."
There still seems to be some intermittent slowdowns for backend processes (uploading an essay, marking an essay as graded). Though Google says the issue is resolved, I still recommend caution.
ORIGINAL POST 1/15 10:28am
At approximately 9am CST Google's App Engine servers suffered a problem with their "task queue" service which EssayTagger uses for behind-the-scenes processing. The majority of the site has not been affected.
Specific interruptions occurred on the student upload page which relies on the task queue to process incoming essay files. The other major impact was that the grading app was unable to process essays when they were marked as graded. Normal grading activity (evaluating rubric elements, adding comments, etc) should not have been affected.
The task queue service is currently intermittent. Take care if using the grading app and keep an eye on the "synced" or "syncing..." message at the top right. If it remains on "syncing..." for more than a few seconds, pause before doing any more work. Your work is only guaranteed to be saved when you see the "synced" message.
Updates will be posted as new information arises.
