The French government’s accounts payable system, Chorus, is back online after a four-day outage, the French State Financial Computing Agency (AIFE) announced Monday.
An accident at a data center operated by French servers and services company Bull on Wednesday affected Chorus’s storage systems hosted there. That incident took the core of Chorus, an SAP system with 25,000 users, offline, although another application, Chorus forms, continued to serve its 30,000 users.
The server room’s fire extinguishing system was accidentally triggered following an error by one of Bull’s subcontractors, resulting in simultaneous damage to several major components of a storage bay holding Chorus data, the agency said.
Bull had little to say about the accident.
“We can confirm that last Wednesday there was an incident affecting one of the rooms in one of our data centers,” said Bull media relations manager Aurélie Negro via email. “By putting the necessary emergency plan into action, we were able to resume normal operating conditions within an hour. However, the degraded operating conditions during that hour may have had an impact on some of our customers,” she wrote.
For Chorus, the impact was severe: Even though the disks were arranged in a RAID 6 pattern to provide redundancy, it was not possible to recover the data or to restore the coherence of the disk system, the agency said.
After working all Wednesday night in its attempt to recover data, the agency decided Thursday morning to restore a backup of the data made before the incident. That, and the subsequent verification of the restored data, took most of the weekend. Service was finally restored at midnight Sunday, the agency said.
Despite the problems, a system handling the exchange of files continued working, allowing payments to suppliers to be queued up for processing. On Monday morning, the agency sent orders to the Bank of France to make 13,400 delayed payments totalling €181 million (US$232 million).
“Chorus has been in operation since 2008, and this is the first major incident,” said AIFE director Régine Diyani in a statement. The system is intended to be available 22 hours a day, five days a week, with an average availability during those hours of 99.8 percent since its creation, and 100 percent in the first five months of this year, she said.