Database upgrade completed

The database upgrade ended up taking about 60min, but is completed.

Database upgrade

During this morning’s maintenance window we’re going to be upgrading EZID’s database.  Based on upgrades of similar systems, we expect EZID to be down for approximately 20 minutes.

Higher-than-normal DOI activity

EZID is encountering higher-than-normal DOI activity.  It may take up to a day for a newly-registered DOI to appear in the Handle System.  We expect the activity to subside within the next couple days.  In the meantime, we’re working with DataCite to see if there’s a way we can push requests through faster.

N2T problem resolved

The N2T database problem has been resolved and EZID and N2T are again connected.  New ARK identifiers will resolve.  ARK identifiers created during the outage are being back-registered into N2T now; that process will complete in about 4 hours.

N2T binder/resolver down

At 9:34pm PDT last night the N2T binder/resolver suffered a database failure.  At 7:00am PDT this morning EZID was temporarily disconnected from N2T, meaning that EZID identifier operations will now succeed normally, with no further internal server errors.  The only effect is that newly-registered ARK identifiers will not resolve.  When N2T is back online, any ARK identifiers created during the outage will automatically be registered with N2T.  DOI identifiers are unaffected.

Intermittent errors noticed, addressed

An identifier subsystem that EZID talks to has been returning intermittent errors for the last day or so, which to EZID clients look like internal server errors.  Because these errors are intermittent, we’ve added some extra retry logic in EZID to shield EZID users from them.  In the meantime, we’re continuing to investigate the problems with the subsystem.

DataCite Metadata Store problems fixed

DateCite traced its problems registering DOIs to disk space issues and has corrected the situation.  New DOI registrations via EZID again appear to be moving through normally and the backlog of pending registrations has been completely cleared.

Errors registering DOIs with DataCite Metadata Store

DataCite is seeing errors with 500 status code for registrations in the MDS. While this always happens occasionally for a number of different reasons, the number of 500 errors has increased in the last 24 hours. The 500 status code means that DOI registration was not successful. They are looking into the issue and will post an update as soon as possible.

Momentary database outage

EZID’s backend database suffered a momentary outage at 12:24pm Pacific time.  We are investigating why the high availability/replication mechanism did not shield EZID from this outage.  But all appears to be running normally now.

DOI registrations delayed a few hours

The last few days some large-scale processing has been taking place in EZID, with the result that DOI registrations have been delayed up to 8 hours.  We have adjusted the procesing to impact end users no more than a few minutes.

As a reminder, DOI registration backlogs may occur at any time.  This is due to the twin facts that EZID queues DOI operations for background processing, and that EZID operates much faster than the DataCite and Handle systems it interacts with.