Trying to get your driver’s license at the revenue office this morning, but couldn’t? Tried to reserve a cabin at Petit Jean State Park online, but couldn’t? Trying to use the state’s website to find out how to contact the director of the Department of Information Systems to ask why, but couldn’t?
That’s because from about 2 a.m. to 10 a.m. this morning, the state’s data system was suffering a temporary conk-out, thanks to a perfect storm that started with Sunday’s deluge, which compromised Entergy’s electrical feed to Big Mac and other state buildings, which required the state to use its back-up system, which itself failed, shutting down the data center. No email, no interagency communication, no public access to online services.
It was Murphy’s Law in action, DIS Director Claire Bailey said, requiring between 60 and 100 people to work around the clock to keep Arkansas from losing its cyber mind.
The state switched to its fuel generator back-up system at 5 p.m. yesterday so Entergy could make repairs to its feed to Big Mac. Repairs were to take seven hours. Nine hours in, at 2 a.m., the repairs were still ongoing when the generator’s fuel pump had a “catastrophic failure,” Bailey said, shutting the generator down. Batteries kept the servers up long enough for DIS staff to properly and systematically shut down systems.
No data was lost, Bailey said. DIS chief operating officer Jeff Dean said the main frame started to show signs of life about 7:30 to 8 a.m. today and the servers were operating at 90 percent at about noon. The website of the state Supreme Court remained down a bit longer, since the Supreme Court operates on a separate server.
A new state data center in West Little Rock that is to come online soon and act as a backup should prevent such a failure in the future.