Software-defined, cloud-based and virtualized: Louisiana’s new data center does it all
At an AWS re:Invent keynote speech last year, AWS CEO Andy Jassy and VMware CEO Pat Gelsinger did something surprising.
Alongside giant private-sector companies like Sysco, Liberty Mutual and McDonald’s, they mentioned a humble government organization: the State of Louisiana. Gelsinger recalled how in 2014 the state formed its Office of Technology Services (OTS) to consolidate some 16 scattered IT shops, and as part of that restructuring, the office began virtualizing its infrastructure, a project that has grown to include VMware cloud on AWS. A central technology office with a staff of 850 became the technology coordinator for 32,000 employees across the state’s executive branch.
Then through contracts with software companies Nutanix and VMware, the state took things a step further and reworked its architecture using hyper-converged infrastructure and software-defined networking, a technology just a few years old that creates a new layer of abstraction for increased automation and control. For the uninitiated, SDN is like the networking equivalent of telling Siri to schedule an appointment versus manually creating the calendar event from the command line.
SDN architectures are hard to find in government — of VMware’s half-million-plus customers across all sectors, just a couple thousand are using the same kind of technology now found at OTS.
Louisiana technology leaders told StateScoop they enjoyed the brief onstage recognition because it was an uncommon celebration of cutting-edge technology in state government, and — even better — recognition of what they hope will prove a successful venture into a technology space few others in government have attempted.
A walk back in time
“You walk into a state, it’s like going back in time — it’s like a time machine,” Louisiana Chief Technology Officer Michael Allison said. “The things that you’re doing today in state government they’ve done private and commercial for a decade. We want to stay on the forefront and we want to embrace the changes that are coming in, taking a more agile approach and being able to stay relevant in technology rather than being so far behind.”
As a member of AWS and VMware’s lighthouse program — a partnership program for organizations interested adopting in young technologies — the state is creating a seamless enterprise cloud that allows services to be hosted on premise, moved to the public cloud and then moved back down as needed, while SDN is simplifying network administration and eliminating possible configuration mistakes.
Joe Skorupa, lead SDN and network virtualization researcher at Gartner, said these technologies are in the early-adopter phase, and it’s rare to see state and local governments using them.
“It’s just a lot of things have to come together to make it all work and part of the challenge is simply getting all of the teams to work together,” Skorupa said.
When different teams can’t agree on which technology to use, the discussion will frequently devolve into “arm wrestling in the conference room,” he said.
“People hate change, people are threatened by change, people are comfortable with what they’ve always used, people like incrementalism,” Skorupa said. “In spite of the fact that senior management says we need to be agile, we need to be responsive — we reward the staff for no failures. If you tell people, ‘Your bonus is based on never having an outage,’ you’re going to do everything you can to make sure you never introduce anything new.”
Owning it
The infrastructure portion of the state’s project is complete — in February 2016, the agency awarded a contract and the SDN implementation went live in December.
Because the new system is being used for Medicaid, the state pays just 10 percent of the $2.5 million project costs, while the federal government pays the rest. One of the benefits Louisiana enjoys through its centralized IT presence is a system paid by federal funds that can be used for services other than Medicaid. State officials reported they don’t know of many other states taking advantage of this arrangement.
Despite this seemingly low barrier to adopting a new and much-hyped technology, Allison still found himself fighting against state government’s legendary institutional lethargy.
“In private industry, where I came from, where you can change your support model on a whim, the state was very much anchored into the traditional silo of support,” Allison said. “In government you have your storage team, your server team, your virtualization team. These silos were built over time — they’re kind of fixed and even to the point of brick and mortar segregation.”
Getting support for SDN was “extremely challenging,” he said, but the technology leaders stood behind their guiding philosophy.
“[Our] answer was, ‘If we never grow, if we never expand our vision, then the answer is it’s unnecessary,'” Allison said. “But if we really want to start looking and taking advantage of what the emerging technologies contain … we need to have these tools in place to allow us to create technology as a service rather than technology as just a platform.”
Once they got executive support, things became much easier, he said, but that was just one battle.
“The subject matter experts, the engineers, now that took a little bit more adjusting. We had to really get them to buy in to the solution and the future value-add of what we’re trying to do,” he said.
Giving the technical people a sense of ownership over their project crucial, Allison said.
“Bringing them in early and bringing them in often and allowing them to have a voice was key in transitioning that so that they could realize we’re expanding their skill set and also really giving themselves a set of tools that allow them to be more effective,” he said.
Once the technical staff were sold, they had to sell the new arrangement to their anchor agencies.
“Because it’s such a different architectural change, it provided a lot of challenges to talk to the agencies about what the benefits are for a technology that’s unproven in the state environment,” Allison said.
Some of the capabilities the state asked VMware to deliver are years from being available, but this is exactly why government should be involved with these types of cutting-edge technologies, Allison said.
“We can help shape what that product looks like, not just from a commercial and private industry, but from what potentially state and federal government and local governments may need,” he said. Some may see early adoption as risky, Allison said, but the alternative was to stay forever stuck government technology catch-up game.
80 percent failure rate
Though these technologies are uncommon, government has good reasons to pursue them, said Dave Cappuccio, an enterprise data center researcher at Gartner.
“[There’s] cost optimization, trying to get rid of redundancies, trying to get rid of secondary, tertiary sites that are doing the same function,” Cappuccio said. “By consolidating, you essentially optimize the number of resources you have to use to provide the same amount of compute.”
Virtualization allows an organization to double down, he said — to “consolidate and consolidate again.”
The benefits of taking on projects like this are often clear. It’s corporate politics that keep these technologies from thriving in government.
“If you’re talking about consolidating 10 or 15 different sites, each of those sites has their own little IT staff, some staying the same way for the last 20 years, and you’re asking them to give up their responsibility,” Cappuccio said. “All those things tend to slow things down. The projects themselves aren’t that difficult.”
Cappuccio estimated that 80 percent of consolidation projects like this don’t hit their targets — they either go over budget or over deadline, or both. “Scope creep,” the tendency of projects to take on too many tasks, is the primary culprit, he said, and what state technology organizations need is a leader who can keep the project focused by saying ‘no’ each time someone suggests adding “just one more feature.”
Healthy technology
Though the infrastructure portion of the Louisiana project is complete, it will remain to be seen this year how effectively the state can extend its cloud and virtualization modernization to its agencies. Officials said they expect that deployment of that part of the project to be completed this year.
Agency leaders managed to stay focused throughout data center modernization efforts so far because the promised long-term benefits if they complete the work are “intense,” said Matthew Vince, OTS chief design officer and director of project management. Vince noted that the consolidation left the state with an assortment of technologies that wasn’t exactly ideal.
“We joking say that the state owns one of everything ever made,” Vince said.
This new work is allowing the state to be more cost effective, Vince said, but that’s really a side effect of being more stable and effective at delivering services.
“The scalable cloud is huge for us because we get some efficiencies there, but ultimately none of that matters if I’m not getting any service benefits for my in-customers, who are my agencies and constituents,” he said.
Listening to state officials talk about hypervisors, UDP packets, and OpenFlow, it’s easy to lose sight of why they’re bothering with it all. But when it’s done right, technology always comes back to government’s mission of serving the people. Sometimes it means lowering taxes through gained efficiencies and sometimes it means a healthier, more prosperous populace.
As a Medicaid expansion state, Louisiana has about 1.5 million residents who depend on the government to pay for their health care. The state’s new technology provides a “predictable” vehicle for ensuring those payments are delivered, Vince said.
“We’re building a solution that is scalable and expandable and modular enough that as I add on new programs — not only with Medicaid — but across our enterprise, [like] child welfare or early childhood or K-12 education or even tax or public safety, this whole platform can support any of those systems, giving them a highly-available, robust, fault-tolerant environment.”
This story was featured in StateScoop Special Report: Legacy Modernization (2017)
