Ransomware recovery lessons from an Indiana county: Budgets matter, people matter

Googling “I didn’t think it could happen to me” returns stories from pregnant teenagers, skin cancer victims and people with spinal cord injuries. A ransomware breach should be tame by comparison, but that’s not the way it sounded in Madison County, Indiana.

The Nov. 4 incident, which took hold of 600 county computers and 75 servers, forced a local jail to revert to pen and paper and had police officers calling other agencies for criminal record look-ups. The county paid a $21,000 ransom following the advice of its insurance carrier, Travelers Insurance, and commissioners subsequently approved three U.S. Signal contracts for defense and offsite storage totaling almost $200,000.

The trojan that infected county networks – CryptoLocker – is one of the most common variants of ransomware found in the wild. When the malware infects a computer system it will encrypt data and create a personal key that is stored by the attackers. Victims will be asked to pay a ransom to unlock their files.

The county is now clear of the incident, said Lisa Cannon, its IT director. Data is encrypted. Links between data and software are repaired. Users are operational. Now’s the time to look back and answer the big question: What happened?

Cannon, who had been IT director just two months when the incident hit, traced the incident to three factors:

An ongoing legal investigation prevented Cannon from revealing who the vendor was, or exactly what or where the vulnerability was. The bottom line, she said, is that the county is now watching its vendors more closely.

“The timing of it was a total surprise,” Cannon said. “It was something that we were aware was out there and you don’t ever think it’s going to happen to you. We thought we had safeguards in place, but we have to be right 100 percent of the time. They only have to be right once. What I learned was you need to know what your vendors are doing.”

Before the incident, Cannon said she was in the process of procuring offsite backup so the county would adhere to the 3-2-1 backup rule – three backups in two forms of media, with one of them offsite.

“When this hit, we were like, ‘Cool! We’ll just turn on our backups and we’ll be OK.’ CryptoLocker changed the file extension on our backup. It locked out our backup. And we did not have the offsite. The backups were in a different building, but they were still on the same network.”

The problem began four years ago, when the county council slashed the IT budget, Cannon said. There was “so much going on with politics at that time,” she said, and implementation of the tighter budget was hasty and uncoordinated. A vote on a Tuesday determined the fate of five county workers on that Thursday.

“Those people were given two days’ notice that they were losing their job, and even though we had contracts and had projects in the making, we lost our funding literally overnight,” Cannon said. “I don’t know what their rationale was. What we ultimately felt like is that they were trying to outsource our department. The commissioners are the ones who would ultimately make that decision and they were not for it, so even though they cut us down by half of our staff and more than half of our budget, we continued to operate under those conditions.”

Cannon’s six-person team is responsible for 12 buildings, 600 computers, 600 phones, 75 servers, 300 mobile dispatch terminals, and 24-hour operations for public safety, including everything from a police force to a volunteer fire department.

“When they cut us like that, they were going on the prospect it would be cheaper to outsource,” Cannon said. “Well, I don’t believe that at all, and I know that the users, especially when you’re talking public safety and the judges and courts, that they are not going to be willing to wait two or three days for someone to pick up a printer or get a monitor swapped out or find out why their phone is disconnecting. That was the argument against the outsourcing, but the ransomware was even more glaring.”

‘We made it’

Cannon is now fighting to get funding back — the approximately $200,000 now being used for offsite backup and remediation is just a third of what was cut.

“We bought professional services to come in and verify that we have best practices. We had someone come in and audit our firewall to verify that we had no more holes in the firewall anywhere, anything that left us vulnerable,” she said.

The county is also purchasing network switches for the local courthouse, which is the entity most frequently accessed for citizen services.

“As an extra precaution, we have placed switches that will monitor port traffic and if the port shows anything that’s not normal traffic coming through or a greater amount of traffic, it shuts it down at the port level and notifies us that that it has shut it down and then we will audit, go up and figure out what’s going on with that PC and then determine whether to turn on that port back on or not,” she said.

The county had antivirus software running before the breach, but it did not detect the malware. New software was installed and remnants of CryptoLocker were removed, Cannon said.

When asked for advice to counties hoping to avoid going through what Madison County went through, Cannon named awareness of public interfaces and building a dedicated team.

“I had a great team,” Cannon said. “One of my guys worked 155 hours in less than 10 days. Another guy worked 150 in 10 days. I worked 144 hours in 10 days and was still off because I had a family member who had a very critical surgery right in the middle of all of that, but we still had to do what we had to do. But we made it. And that’s what any IT director needs to make sure they do is surround themselves with good, passionate people for the technology and the organization.”