Measuring the impact of data programs is about more than just dollars
As the role of the chief data officer has risen in state and local government, their bosses — governors, mayors, cabinet members — have enjoyed being able to cite figures like savings found from fraudulent and wasteful spending or the number of charts and graphs hosted on open-data dashboards. But for a practice driven by cold, statistical analysis, actually quantifying the returns states get on their investments in data can be nebulous territory.
Carlos Rivero estimates that in the three years he served as Virginia’s chief data officer, the commonwealth spent less than $6 million on funding his operation. But Rivero told StateScoop recently that was money well spent, pointing to several programs his office spearheaded to reduce overdose deaths, track COVID-19 metrics, tamp down on unemployment fraud and help displaced workers find new jobs.
And while all of those efforts showed tangible benefits to Virginia, Rivero said it’s difficult to give them a precise financial value.
“It’s very difficult to measure, to attach any value,” said Rivero, who’s now a vice president at the IT management firm GCOM. “Just the number of data sets doesn’t tell you much of a story. It’s how much of an impact it’s having.”
A 2018 study from the consulting firm McKinsey estimated that governments worldwide could reap a collective $1 trillion from data-analytics investments, especially if data programs are focused directly on cutting down fraud, waste and abuse. But McKinsey also concluded that among governments that have created data programs, few actually have the capability to measure their financial impacts.
‘The people who don’t get sick’
Jane Wiseman, a senior fellow at Harvard University’s Ash Center for Democratic Governance and Innovation, said there are three ways for a government to measure the impact of its data-analytics work. The first is purely financial: “The ROI can be the money you get back from figuring out a more efficient way to do a tax or recoup fraud,” she said.
The lines start blurring on Wiseman’s second category. She pointed to one of Chicago’s data-analytics efforts, a round of restaurant inspections that considered if establishments met certain characteristics, including previous violations, whether or not they possessed a liquor license and proximity to sanitation complaints. During a two-month trial period in 2014, Chicago’s food inspectors found that 55% of restaurants with health violations met at least one of the predictive categories.
That survey led to the creation of a predictive analytics tool that Chicago has used in the years since, with officials claiming to now spot violations 25% sooner. But, Wiseman said, it’s tougher to celebrate than a big cost savings.
“It’s not like there’s going to be an above-the-fold story about all the people who didn’t get sick,” she said.
And the “most swishy” of ways to measure impact is faith in government, Wiseman said. She has some experience in that area: As Massachusetts’ assistant public safety secretary in the early 2000s, Wiseman said she oversaw the collection of data showing who wears their seatbelt. She said Black and Latino males between 18 and 24 were least likely to use their seatbelts, leading to a marketing campaign aimed at getting those groups to buckle up.
“What’s the ROI?” she asked. “How can you possibly measure it?”
‘Things that are tangible’
Part of the difficulty in measuring data programs’ value comes from the fact that there’s no clear-cut model for how they’re organized or funded. Tyler Kleykamp, a former Connecticut chief data officer, said he started out as an “office of one” with a “pretty meager budget,” most of which was spent on getting the state’s open-data platform up and running — a common beginning for data officers.
“A lot of [governments] we’ve seen are not necessarily up front contemplating what financial investments they might make beyond hiring a chief data officer,” he said.
Kleykamp, who now leads the State Chief Data Officers Network at Georgetown University’s Beeck Center for Social Impact and Innovation, said his office gradually expanded to include a few other employees, including Connecticut’s geospatial information systems analysts. But nationally, state data offices vary greatly in scope and impact.
“It doesn’t have to be a huge, massive, up-front investment,” he said.
Rivero, the former Virginia CDO, said his office was able to enhance its effectiveness by pivoting its Framework for Addiction Analysis and Community Transformation, or FAACT — a platform originally created to track opioid-related deaths — to respond to issues raised by the COVID-19 pandemic.
That conversion allowed Virginia health officials to track COVID statistics and determine which parts of the commonwealth could ease up on their restrictions in the early months of the crisis: In July 2020, for instance, much of Virginia started allowing restaurants to reopen for limited indoor dining, though heightened caseloads in the Hampton Roads region led to a few weeks of additional restrictions.
“That allowed the state to continue operating from an economic perspective while having one region locked down,” Rivero said. “It ultimately resulted in a budget surplus of $2 billion. These are things that are tangible.”
Big spenders
While Kleykamp and others said there’s no universal model for how states should invest in data programs, some are bigger spenders than others. Indiana’s Management and Performance Hub, a governmentwide analytics shop, will get about $7.4 million in 2022 and 2023 under the state’s biennial budget.
And North Carolina’s Government Data Analytics Center in 2020 spent about $36.3 million to provide data services to state agencies, and another $23.9 million to support the state’s health information exchange, according to an annual report from the North Carolina Department of Information Technology.
There are a variety of funding sources: The GDAC is run as a public-private operation between the state and the analytics multinational SAS, which contributed $15.4 million to the center’s budget. The health information exchange was supported by another $9 million from SAS and a federal grant of $3.75 million.
“Ten years from now, data shops will be a standard line item” in state and local budgets, Wiseman said.
Indiana has claimed a 4-to-1 return on the Management Performance Hub. Ohio Lt. Gov. Jon Husted said last year that his InnovateOhio program led to $2.3 million in savings. Still, despite the savings government leaders herald from their data programs, Kleykamp said the benefits of data analytics are measured far beyond raw numbers.
“What are the changes that are happening as a result of better leveraging data?” he said. “Saving 5 or 10% in data hosting costs is far less than savings from reducing the prison population by 1%. I think that’s how we should measure these efforts.”
This story was featured in StateScoop Special Report: Data and Analytics (2022)
