How Sparks, Nevada, is rethinking security after ransomware

The ransomware epidemic isn’t going anywhere, and state, county and local governments are all in the crosshairs. There have been roughly 100 reported ransomware attacks against public sector institutions in the U.S. since the beginning of the year. Some pay the ransom so they can get back up and running, while others may not give in, but then face a long, painful road back to full operational capacity. 

Cyberattacks can happen to anyone. When it happened to the city of Sparks, Nevada, it led us to take a different approach to detecting and mitigating threats. 

A ransomware attack hit us in August 2015 through a legitimate website used by our police department to access training videos, which was triggered when a detective clicked on a compromised video on one of our training sites. Instead of starting the training video, the link began an attack. The detective’s PC was subsequently encrypted and the ransomware then propagated to any drive attached to the network and spread to the other devices on the shared employee drive.

The first indication that something was amiss was when someone tried to copy a file from a shared drive and discovered that it was encrypted. Then our geographic information system administrator said all of the “flyover” pictures of our city had been encrypted. GIS is the lifeblood of any city’s operations, and this caused considerable alarm. 

We narrowed in on the origin of the attack, unhooked the affected machines from the network and got to work addressing the issues in order to get our systems back online. Our immediate priority was restoring essential services — everyone focused on that goal as we raced to get our systems back online. Fortunately, we were well prepared with comprehensive backups, so critical systems, such as public safety, were restored within hours. Still, restoring some data took much longer. 

At the time of the attack, our backup tape library used LTO3 tapes, storing about 700 gigabytes each. We had to get all of the necessary tapes from an off-site vendor in order to feed them into the tape library one by one — a painfully slow task. It was two weeks before our GIS data was completely back online, as huge image files needed to be restored from small, slow tape backups.

We needed a method to keep this from happening again.

Monitoring options

We considered multiple approaches to solving the monitoring and alerting challenge. A dedicated security staff was not an option because the budget simply wasn’t there. Plus, we would have needed to purchase, integrate and manage a suite of security information and event-management tools that would have needed to be regularly refreshed and taken considerable time to get ramped up — and we needed to act quickly.

Instead, security-operations-center-as-a-service made the most sense for us. A managed detection and response, or MDR, approach provided us with dedicated resources to identify issues without our needing to expand our staff. MDR is a relatively new type of cybersecurity solution that the advisory firm Gartner predicts will grow in adoption by organizations from 5 percent today to 25 percent by 2024.

Numerous demos showed us what we didn’t want. One vendor sent alerts by email, which didn’t reflect the urgency of the situation. Another required us to have an employee dedicated to monitoring and reviewing logs.

The MDR approach we chose provides an assigned security engineer who remotely watches network traffic and looks for anomalies that require action. If the engineer spots something, he or she immediately calls to alert us and provide recommendations.

We were alerted to an issue within five minutes of installing a demo of this MDR solution. A city employee had opened a legitimate website, but his password was being transmitted in clear text. We were immediately alerted and were able to shut that connection down right away.

In addition to our day-to-day interactions, our vendor holds quarterly meetings with us to discuss concerns and issues, as well as potential enhancements to the service. This relationship allows us to continually improve our security game and plan for emerging risks that might otherwise bite us. 

Retaining accountability

MDR monitors everything from our network to endpoints to cloud services like Office 365, giving us a holistic view. It has also reduced the number of false positive alerts we’ve had to deal with. 

But that doesn’t mean we can be complacent. While the City of Sparks can delegate the responsibility of cybersecurity monitoring, we retain accountability for security and strategy.

We still need our own security infrastructure and personnel to manage vendor relationships and ensure that all of our protection tools are configured and maintained properly. We also need people to act on any alerts we receive or threats that might slip through our defenses.

Although I have confidence in MDR, I can’t overstate the need for comprehensive backups. We were prepared for the ransomware attack, as our backups were complete and current. But there’s still a disruption to city services and internal workflows while everything gets restored. 

There are also other threats besides ransomware, such as natural disasters or physical damage to systems from fire, weather or even a deliberate act. Being prepared for any eventuality demands a smart backup plan.

We now back up to the cloud as well as to tape. We can restore virtual machines via the cloud, and we use higher-capacity tapes to make it quicker and easier to restore data. In fact, I now believe our critical server infrastructure can be up and running within an hour.

Taking control

With a managed approach we can deploy our resources much more efficiently. It lets us focus on the city’s priorities, especially around public safety. Rather than staffing a team to provide 24/7 monitoring, our MDR service provides a more cost-effective solution. This enables us to optimize hiring to accelerate projects that provide better services to our residents while maintaining appropriate security.  

Ransomware attacks will continue until they no longer work for the bad guys. Having been hit with ransomware and recovered from the episode, we have learned (or relearned) some lessons. A good backup strategy is a must. Also, while firewalls and anti-virus tools are important, they aren’t the whole story, so whether you do it yourself or use a service, you need to monitor for what inevitably slips through. Don’t get too comfortable, either; we regularly review our technology and protocols to make sure we’re ready for the next threat. You also need a good incident-response plan that includes all constituents, from IT to legal to PR.

In security, nothing is 100 percent. We still see attacks all the time, but have been able to stop them from taking hold in our network. Cyberattacks against government agencies are much more likely to happen than not. The good news is that the tools and strategies to improve your security and your response capabilities already exist. Being proactive is the best defense.

Steve Davidek has been working in information technology since 1981 and began working as a computer operator for the City of Sparks, Nevada, in 1984. He’s led initiatives that include storage area network designs and enhancements, virtualization of more than 95 percent of the city’s servers, desktop virtualization, networking upgrades, and high-speed Wi-Fi. He’s been Sparks’ IT manager since 2014 and plans to retire in October 2019.