Migration Post-Mortem

PINES is circulating books. Patrons are finding stuff. Catalogers are cataloging. PINES central staff members are still in-country and not in fear for their lives. We’ve done it, but we didn’t get here without a lot of hard work and lessons learned. For posterity’s sake, herein lies the tome of the Evergreen migration. The good, the bad, and the ugly.

What we did right

• We sent our patron data to Unique Management Services for clean up. This made the job of importing the patron data oh-so-much easier. Rob Klaus, you rock!
• We performed multiple dry-run migrations. This was useful in ensuring our data migration plan was sound from A to Z.
• We asked some library staff around the state to look at the dry run data, and to double-check us. This proved helpful in identifying translation problems before they became a problem in the real system.

What we did wrong

• The biggest one: lack of sleep. On Tuesday morning, when bugs and other issues started popping up, the development team was running on no sleep. We wanted things to be perfect on day 1, and I think we pushed too hard, when we should have left good enough alone, and gotten some sleep. So, when our troops hit the beachhead on Tuesday morning, we were not sharp at all.
• We recently lost the PINES/GPLS Helpdesk Manager, and this position was not filled by go-live. (Our new Helpdesk Manager just started last week). I’m not sure if this is something we did “wrong”, but the lack of a Helpdesk Manager had a significant negative impact on the PINES support structure.
• We didn’t write up any “what to do if…” documents for the libraries prior to go-live. There was confusion about which way to go, and email is not a good medium for transporting urgent messages on go-live morning.
• We didn’t anticipate the immediate increase of OPAC usage. When a tool isn’t painful to use, people seem to use it more. (Imagine that.) It also helps when the catalog kind of pretty to look at.
• In order to troubleshoot issues, we had to turn logging levels up on the production system. The sheer amount of log traffic swamped the central logger servers, and in turn, made finding pertinent log entries extremely difficult. We have learned we need much more robust central logging servers.
• We didn’t have a dedicated training cluster. This is a scarce resources issue, and it hurt us. Given the resources, we should have trained PINES library staff on a training cluster with real data instead of relying on the demo system (and demo data) for training.
• Related to the no sleep bullet above, we waited too long before pulling non-essential services. In hindsight it seems obvious, but at the time it wasn’t on any of our radars for some reason.

That’s a lot more bullets in the negative column than the positive, and I don’t want the folks to get the wrong idea. No ILS migration is easy. I think most librarians had rather subject themselves to a million papercuts than migrate ILS’s. However, I think as far as migrations go, this has been a roaring success, not even close to worthy of papercut torture. I think this is especially true when the scope of the PINES system is taken into consideration. We have taken an enormous leap forward, at warp speed. The proof is in the pudding.

So, who’s ready for Evergreen v2? 🙂