Expect all of your technology to fail
Redundant systems are a response to the idea that anything can break. But even with backups in place, overconfidence in technology can be just as harmful as a lack of planning, warned Jay Bowden, an emergency planner with Newport News, Virginia. Bowden has worked in emergency management for about 35 years “in some form or fashion” he said, and he’s watched as technology has both eased the lives of those who use it and simultaneously eroded skills that are potentially critical during times of emergency.
“I’m not sure my grandson would survive in a shelter once the battery died on his iPad,” Bowden said. “Everything now relies on electricity, right down to the water supply, sewer. You name it, it’s got a plug on it.”
Back in 1993, when he was working as deputy director for the Mayor’s Office of Emergency Management in Nashville, Tennessee, the city faced what Bowden called a “unique” ice storm that reminded everyone that technology might not always be working as expected or planned. The storm came just weeks after the city had launched a new emergency response program.
“We had [plan] notebooks that hadn’t even been passed out when it happened,” Bowden said.
The storm was unique, he said, because the region lost lots of power lines and transformers, but the roads were “totally fine as far as ice goes.” Bowden said he and his team had been in the emergency operations center for three days responding to the storm before they received an unwelcome surprise.
“We learned there was an entire community up in one of the rural parts of our county that was completely without power,” Bowden said. “They had lost telephone service, they had lost all communications, so they had no way of letting anybody know there was a problem.”
Police cars were supposed to be patrolling that area, and there was a fire department there, but no one radioed for help, and Bowden said he never quite found out why.
“It was one of those miscommunications where somebody thought that somebody thought that somebody else had done something else,” he said. “But the bottom line was that the public couldn’t call for help, couldn’t call for 911, they didn’t have any power.”
Bowden said he learned unforgettable lessons that day. One was to never expect that something is working right just because nobody says its actually broken. Back then, the way to check that a region had power was to call a place that had an answering machine or a fax machine like a school and see if it picked up, but because there was no specific plan outlined to check on that community, no one did.
That day also reinforced in Bowden the idea that people who enjoy the comforts of city life tend to lose the self-reliance and survival skills that often accompany life in a rural area. Nothing disastrous happened, he said the people who lived in that area knew how to build fires, heat up food outside and make do. But not everyone does, and those skills are becoming increasingly rare.
“That’s the whole thing about technology,” Bowden said. “We’re running hard into it because it makes our lives so much easier and so much quicker and so much everything. But in the course of doing that, we lose track of what we might be losing, too.”
Dont fall victim to information paralysis
As a meteorologist and not a trained emergency management specialist, Derek Arndt says he was not prepared for the onslaught of information confronting him several years ago during a field study for a new type of radar system.
While leading a weather data training and provisioning crash course in Oklahoma called OK-First, Arndt was asked to help in testing a short-wavelength radar system to fill in the gaps for traditional and expensive long-wavelength systems.
As a supporting researcher, Arndt decided he would play a game. He pretended to be an emergency dispatcher as he monitored the data coming in from new radar system. But there turned out to be an “intense” characteristic of the new system he said he was not prepared for. While the traditional radar system delivered updated data every four or five minutes, the new system delivered data every 60 seconds.
“When the updates are 60 seconds, there’s this odd paralysis that happens,” Arndt said. “You think, ‘OK, the next update’s only 30 or 40 seconds away, so I can afford to wait for that one to make my decisions. And then you get that one and you’re like, ‘I can afford to wait for the next one. It’s right around the corner.’ I was paralyzed by this rapid onset, this overload of information.”
In an actual event, Arndt said the way he handled the fast-churning data would have been “disastrous.”
James Hocker, the current program manager of OK-First said information paralysis is definitely something he hears about, particularly for newer emergency managers.
And there’s no simple cure for it, Hocker said, save good old-fashioned experience.
Trust your threat intel
As extended protests and riots took the city of Ferguson, Missouri, to the brink in the summer of 2014, Missouri Chief Information Security Officer Michael Roling knew he would need help in fortifying the states cybersecurity and patrolling the internet.
A united effort from local, state and federal agencies to gather threat intelligence the data collected by public safety and law enforcement to get a bead on potential incidents before they happen provided Roling with the support he needed during an unprecedented crisis. Roling said that collecting threat intelligence from open internet channels like Twitter enabled law enforcement to invest their time and resources wisely.
That intel allowed agencies to respond effectively to online actors threatening to “dox” law enforcement and inflict damage on the states digital services a plan that was thwarted, he said, because the the government side knew what to look for.
We were able to identify key threat actors communicating on open channels like Twitter, and understand their tactics, techniques and procedures, Roling said.
Roling, whos served as Missouris CISO since 2009, told StateScoop that the Ferguson unrest was the first time hed seen a coordinated intelligence campaign effort between all three levels of government actually come together. The act of gathering threat intelligence, he said, mostly revolved around homegrown tools on the state and local level combing social media, online forums and message boards for mentions of Missouri or other trigger words related to the protests.
Support from federal groups in particular gave the team a chance to stay one step ahead of threats made online, he said.
MS-ISAAC, DHS, FBI we were all huddled in various channels talking to each other as events were transpiring, Roling said.
The [online] threat actors, they dont differentiate necessarily between different forms of government, Roling said. They just call it government and go after it. So we had to unify our front, state, local and federal when it comes to these styles of attacks, and going back to Ferguson, that was the first big event that I can recall where I got to see that in action and be a part of it and it worked.
The intelligence Roling and his team gathered helped back up law enforcement on the ground as well as the governments online services.
[Threat intel] should shape everything from network endpoint all the way to the human, because the human is the number one attack target and will continue to be in perpetuity, Roling said. Understanding how theyre being attacked is absolutely critical because you can educate your employees as to what to look for.
Standardize your data
The wildfires that tore through Californias wine country last October gave residents little time evacuate or check in on their loved ones. And for the most vulnerable residents hospital patients, residents of mental health and long-term-care facilities, and kids enrolled in daycare centers the fires were even more dangerous.
Inside the California Health and Human Services Agency, which oversees the states hospitals and other care centers, state workers scrambled to account for the facilities within range of the blaze.
But it was chaotic at first.
Marko Mijic, the agencys assistant secretary for program and fiscal affairs, recalls that the early effort to determine which facilities were safe was chaotic for a rather simple, but easy-to-avoid reason: agency offices contacting affected facilities were collecting data differently from each other. That left CHHS with an incomplete picture of which of the 3,000 facilities it regulates in Northern California were out of the fires path and which needed to be evacuated.
How people were collecting addresses across departments was completely different, Mijic told StateScoop. We had no way to aggregate the data we had and no way to give a holistic look at the data.
Compiling all the data into a usable format took nearly two weeks, during which CHHS had to send employees into the field to verify the status of facilities that werent accounted for. In the meantime, CHHS couldnt always help panicked residents who called about the well-being of their relatives receiving treatment at unverified facilities.
We had the public calling us, wanting to know where their family members were, Mijic said. We needed a better way to communicate.
The response to the Northern California fires wasnt a complete debacle, though. Mijic and CHHSs then-deputy secretary, Michael Wilkening, established a task force to secure medical and care facilities out of the state operations center, which helped the agency figure out which sites needed to be evacuated, and when they could be repopulated. (Wilkening was nominated in May to take over as secretary of health and human services.)
In an ideal emergency response, Mijic said, CHHS would have uniform datasets that could be loaded into a publicly accessible dashboard-like tool on which residents could get updates in real time.
When more fires broke out in Los Angeles and San Diego counties last December, Mijic and his team had learned its lesson about data collection. A full-blown public dashboard wasnt ready yet, but CHHS had set up a tool just an Excel sheet, really, he said with standard data fields. Within 48 hours of the first outbreak on Dec. 4, Mijic said, CHHS had aggregated the status of 2,500 facilities into a GIS-based map that helped the agency determine which needed to be evacuated first.