How Texas used its disaster playbook after a huge ransomware attack

Many details of the ransomware attack that struck 23 local governments across Texas in August remain either unknown or under wraps as part of an ongoing federal investigation. But the incident also allowed the state to flex its ability to marshal a disaster-level response following a cyberattack, two of Texas’ top IT officials said Monday.

Texas CIO Todd Kimbriel, speaking at the National Association of State Chief Information Officers annual conference in Nashville, Tennessee, said the first municipality to detect something was wrong with its systems called its managed service provider in the early morning of Aug. 16. By 8:46 a.m., the Department of Information Resources had been alerted that several local governments around the state had been hit with ransomware, with more reports pouring in. By noon, the state operations center in Austin was up and running, coordinating several different agencies to begin responding to the attack, he said.

But the speedy coordination between DIR, the Texas Department of Emergency Management (TDEM), the state National Guard, Texas A&M University and several other statewide organizations was only possible because of a 2017 law that extended the governor’s power to issue disaster declarations to cover cyberattacks, Kimbriel said. Without that legal tweak, it would’ve been much harder to assist the dozens of small towns hit in ransomware event.

“When the governor declares a disaster, that means the SOC can be activated,” he said. “But if you don’t have that disaster declaration in place, you can’t get that dispatch.”

Kimbriel also said learning the ransomware was affecting a city’s water utility prompted the call for a disaster declaration.

“When it hit the water system then we engaged with the governor’s office,” he said.

In declaring an emergency over a cyberattack, Texas Gov. Greg Abbott added his state to a club that already included Colorado and Louisiana, both of which have also used a disaster playbook to respond to ransomware. Getting the declaration wasn’t simply an internal decision, Kimbriel said, but it required going through a few bureaucratic channels, including filing what’s known as a State of Texas Assistance Request — or STAR request — that brings state-level aid to local governments in need.

Executing the plan

Texas also had a cyberattack response plan on the books, a result of work done several years ago by the team led by Nancy Rainosek, the state’s chief information security officer. A report earlier this year from the National Governors Association urged states to create cyber response plans because they can make things run relatively smoothly.

“We pulled it off the shelf and executed the plan,” Kimbriel said.

Speaking alongside Kimbriel, Rainosek also praised the resource and coordination abilities offered by TDEM, which frequently activates during hurricanes and other natural disasters. The emergency management agency brought in applications like a secure chat app that allowed every responder to get updates in real time, and WebEOC, a crisis-management tool that organizes emergency workflows. TDEM was also skilled in the less-technical aspects of the ransomware response, Rainosek said.

“Another benefit about working with TDEM is that they fed us, oh my gosh,” she said. She compared this with the experience of Colorado IT workers who subsisted on pizza and soda for about 10 days during the initial response to a ransomware attack last year against that state’s transportation department before a statewide emergency declaration was issued.

Many of the communities impacted by the August ransomware attack are now rebuilding or replacing their damaged systems, some with the help of a deep discount in the state’s bulk-purchasing agreement with Dell, Kimbriel said.

A growing concern

Still, many of the details of the attack remain unknown to the public, including the full list of communities affected. Both Kimbriel and Rainosek attributed their reticence to the FBI, which is leading the criminal investigation into the incident.

What is known is that the attack — likely using malware called Sodinobiki or, alternatively, REvil — was directed at a managed service provider, and transmitted to 23 of its customers. But Kimbriel said the ransomware succeeded because of bad practices at the impacted government organizations, not the provider itself.

“These organizations were impacted because they did not follow good cyber hygiene,” he said, reminding the audience that IT organizations need to install security patches and maintain strong password policies. “This was not 100 percent of the service provider’s customers.”

Kimbriel and Rainosek told StateScoop later that ransomware attacks against service providers — which can affect a large number of governments — is a growing worry, but also one they have little control over.

“If I’m a bad actor, that’s a pretty darn good threat,” Kimbriel said.

But Rainosek noted that she has no authority over the IT decisions local governments make, though she said she plans to offer advice where she can.

“It’s not just service providers, it’s local governments in general,” she said. “We’re going to document best practices and share those.”