EZID maintenance complete: Sunday Sept 30, 2018 08:15 PT

The EZID maintenance has been successfully completed.

Advertisements

EZID maintenance Sunday Sept. 30 08:00 PT

We’re planning for EZID maintenance during the regular EZID maintenance period from 08:00 until 09:00 PT on Sunday Sept 30. There may be a brief outage lasting only a few minutes, or if things don’t go as planned, for a longer part of the hour. If things don’t go at all as planned, we’ll restore the system and have it working again by 09:00 PT.

AWS issues resolved

The problems have been resolved and EZID is operating normally again.

AWS network issues

While it appears the EZID server is running, some identifier operations are failing.  Beginning at approximately 5am Pacific time, CDL began encountering networking issues in its AWS environment.  Many services are impacted.  The support group is investigating.

EZID database restart

EZID’s database went down and was automatically and immediately restarted at approximately 19:42 PDT earlier this evening.  All appears to be running fine now.  We’re investigating the cause.  The system was processing an extremely large number of identical searches that had been submitted by one user simultaneously, but it’s unclear whether the database had problems and the searches built up, or the searches caused the database to go down.

Database hiccup

At approximately 7:30am PDT, EZID’s database in the Amazon cloud infrastructure suffered a brief outage.  3 operations were affected.  The database has been operately normally since then.

Server hiccup

The server hung on a routine software update and had to be restarted, resulting in an outage of a few seconds.  There were a few requests being processed at the time, and those likely failed.

Excessive usage encountered

EZID suffered a brief interruption when it was restarted to ban a particular user.  We’re unsure whether the user was malicious, or just being over-enthusiastic in their stress testing.

N2T resolver backlog

Due to heavy activity, there is a backlog in registering ARK identifiers with the N2T resolver.  The backlog should clear in about a day.

The backlog has cleared. DOI registrations will be reflected in the Handle System with their normal delay, a few seconds to a few minutes.