Mislabeled wires responsible for two-day IT outage in South Dakota, officials say

A wiring issue at the South Dakota State Data Center was responsible for a two-day power outage that interrupted the state’s IT systems in August, two state officials told lawmakers on Thursday.

The state’s chief information officer, the Bureau of Information and Telecommunications Commissioner Mark Wixon, testified before the state legislature’s Government Operations and Audit Committee on Thursday about the outage, saying that it stemmed from the replacement of an uninterruptible power supply. The piece of hardware failed, tripping the main breaker at the data center multiple times. The loss of electricity shuttered several state systems, including those at the state’s Department of Revenue, the Department of Public Safety and the state attorney general’s office.

Darin Seeley, the state Bureau of Human Resources and Administration commissioner, told the legislature that resetting the tripped breaker caused damage, requiring the breaker to be replaced. But even when the breaker was replaced, he said, issues persisted because of mislabeling.

“In the flow of the electricity … those [switches] were energized, unbeknownst to the electricians and being hit by significant amperage when the system was being turned on, because there is a mislabeling of the electrical lines in that facility,” Seeley said.

He continued on to say that the facility had been updated numerous times over the past fifty years.

“When you shut off the main, you believe that everything behind that main is off,” he said. “Unfortunately, those were labeled to say that they are connected to that main, and they are not connected to that main. They continued to stay hot and the electricity was delivered to the switches, which ultimately then, they failed, and that’s what caused the actual network outage then.”

Wixon told legislators that even though the system had built-in layers of redundancy, the wire mislabeling caused the switches to also malfunction. He added that, in total, replacements for damaged parts totaled about $80,000. However, he said that the response of the state’s technology staff, that he had been appointed to lead, was “a very proud moment.”

”It definitely was a team effort,” he said.