San Francisco's new CISO comes from Kaiser Permanente, Deloitte
January 18, 2018
The recent hire fills a gap in the city's IT security leadership that been left without a full-time replacement since May.
There's a battle being waged behind the scenes of public safety's first nationwide wireless network.
Colin Wood is the managing editor of StateScoop. Before that, he was a staff writer for Government Technology magazine. Before that, he taught Engl...
State leaders knew this decision was coming for a long time, but they probably didn't think it would be this complicated. What began as a federal IT project to support the nation's first responders has devolved into a turf war over who will control the radio waves.
A contract awarded to AT&T in March left billions of dollars on the table from a competitor, and what recently appeared to be a straightforward process for most states to go along with the federal government now appears less certain. Several industry sources told StateScoop they believe a majority of states are now preparing to opt out of the plan. Among the reasons cited include dissatisfaction with the federal agency's bureaucratic approach and suspicions that the winning vendor's proposal won't align as closely with state interests as they initially had hoped. America won't have to wait long to see if these predictions are realized.
In June, the federal First Responder Network Authority (FirstNet) and AT&T will present to each state a draft plan of the first nationwide wireless data network for public safety. For the first time, states will have a precise description of the next-generation technology the federal plan can offer their firefighters, police officers, paramedics and other first responders — from features to pricing. If they don't like what AT&T offers, they can select another vendor to build a comparable system for their state and hook it up to the national network.
A superficial look at the landscape today would indicate most states are not prepared to opt out. Just a handful — including New Hampshire, Arizona, Colorado, Michigan, Alabama, Wisconsin and California — have issued request for proposals (RFPs) or requests for information (RFIs) to develop alternative plans. And, officially, even those states remain uncommitted — several state contacts assured StateScoop that checking out an alternative plan was mere "due diligence," a piece of common-sense research designed to keep their options open.
With several key details of AT&T's plan still not available, no state was willing to dismiss the federal offering outright, but adoption may be spottier than people initially believed. If enough states say no to the multibillion-dollar AT&T deal, some say it could "imperil" the network's interoperability and introduce complexity, cost and undue cybersecurity risk. The technology that states commit to buying in the coming months could determine the the ultimate fate of the network, and who lives and dies in a big emergency.
Discussions of a network for public safety like the one FirstNet is building have been ongoing since publication of the 9/11 commission report, which highlighted the need for first responders to have dedicated access to communications towers during emergencies and easy interoperability between agencies.
Sometime in June, each state or territory will receive a draft plan along with a proposed funding level. After 45 days of review by the states and then 45 days of review by FirstNet, the federal agency will return to issue final plans. Then the clock starts ticking. Governors will have 90 days to make to make their final opt-in or opt-out decision that will lock each state into a vendor for the following 25 years. States choosing to opt out will be allowed 180 days to develop a plan that meets comparable objectives to the federal AT&T plan.
AT&T has the right experience and will likely do "a great job" on the deployment, said Royce Kincaid, a former executive with aerospace and defense firm Northrop Grumman. When FirstNet began holding board meetings in 2012, Kincaid was there. The company was seeking to participate in some form, but eventually dropped out of the picture after declining to partner with AT&T.
Despite an endorsement of the agency's chosen vendor, Kincaid sympathized with the frustration coming from states — he's been hearing about it for years, he said.
"It took them almost four and a half years to award the contract," Kincaid said. "Sure, FirstNet had to abide by the legislation and they had to get a lot of input. If it took that long in the commercial world, those people would have been fired."
Early tumult and "shenanigans" between board members gave the agency a poor reputation from the start, Kincaid said, but things have since stabilized, especially since Susan Swenson was appointed board chairwoman in 2014.
"But if I were a state," Kincaid said, "having seen what happened in the last four and a half years, I can't believe they couldn't do it themselves more efficiently than a national model."
For a few billion more
FirstNet's objectives-based RFP ultimately fell into the hands of lawyers and bureaucrats. A Department of Interior evaluation and federal court proceeding that deemed vendor Rivada Mercury out of competitive range in March resulted in AT&T becoming the winning bidder.
The prize for AT&T: 20 megahertz of 700 MHz band radio spectrum to operate a public safety network. FirstNet will have priority use of that spectrum when there are emergencies, but AT&T can also use it for its commercial businesses.
The contract stipulates that within the first five years, FirstNet will support the project with $6.5 billion in funding, with the remainder of investment coming from the vendor. AT&T committed to investing $22 billion in its FirstNet business during the course of the 25-year agreement. The company also committed to paying FirstNet $18 billion in cash, $16 billion of which is slated for rural connectivity improvements.
The resulting deal appears to be a safe bet for public safety — a trusted name in the telecommunications industry would lend its expertise and stability to ensure a reliable network is there for fires, for terrorist attacks, for rescuing kittens from trees. But Rivada Networks founder and CEO Declan Ganley argues that any deal would seem like a great deal with nothing to compare it against. (Rivada Mercury is the name given to the partnership that bid for FirstNet; Rivada Networks is the lead company.)
Until now, key details of Rivada Mercury's bid remained hidden under court document redactions. These details reveal a disqualified competitor that was proposing a network with more coverage, lower cost to users, and a vastly larger cash offer to the federal government. Rivada Mercury told StateScoop its offer was to give FirstNet $29.5 billion in cash, plus $22.2 billion in in-kind services, for a total of $51.7 billion — $33.7 billion more than AT&T's $18 billion cash offering.
Rivada Mercury — a coalition of Rivada Networks, Harris Corporation, Ericsson, Nokia, Intel Security, Fujitsu Network Communications, and Black & Veatch — was noted by the court for its depth of experience in wireless communications. The federal agency's source selection authority called the group's proposed members "industry leaders with substantial experience in their respective areas, including wireless infrastructure and public safety."
However, the Department of Interior determined that Rivada Mercury "lacked relevant experience managing subcontractors" and that its plan "lacked financial stability," and that such a level of risk should not be tolerated for an infrastructure project of this size and importance. This determination was upheld by the U.S. Court of Federal Claims' public opinion.
Ganley reckoned that someone must have made a mistake.
"We had the best in the industry — Gartner, outside analysts, and the credit committees of eight of the top banks on Wall Street sign off on this," Ganley said. "Plus, it was bonded by one of the top insurance companies in the world. … The thing was financed layer by layer up to gazoo and then with a bond to bond guarantee of the execution of the contract."
The competitor was also penalized for its pricing model, which was also deemed unstable. Rivada Mercury's proposed monthly per user fee: one cent.
"We got dinged really hard for doing that," Ganley said. "But we felt it's our mission, our job to get the value out of the spectrum, not out of public safety. That was the whole idea of giving them the spectrum in the first place. The value of the capacity on that network is immense. We may find more gold, we may find more oil. We are not going to find any more radio spectrum. It is extraordinarily valuable."
AT&T's pricing is not yet available — the federal agency has so far only reported its pricing will be "aggressive" and "competitive."
Strong AT&T supporters like APCO International maintain that not only was Rivada Mercury's financial model unstable, but that any state now choosing to opt out — using any vendor — would undermine the premise of the network. FirstNet was intended, after all, to work with the consistent reliability and functionality anywhere in the country. Fragmenting the network with more than one vendor is simply not a viable option, Jeff Cohen, the group's chief legal counsel, told StateScoop.
"To opt out, when we look at it from a purely public safety communications perspective, we get very nervous, because the odds are really stacked against [states] in terms of relative resources compared to FirstNet," Cohen said.
The group argues that any state opting out will face a near impossible review process, incur sustainability risk through changes of administration, introduce unnecessary cybersecurity risks, and add cost and complexity to what is already a massive and highly complex project.
"There's no safety valve if a state network fails," Cohen said.
The value of spectrum
In 2012, Congress enacted the Middle Class Tax Relief and Job Creation Act, which included provisions to create FirstNet, fund it and direct the Federal Communications Commission to allocate radio spectrum that public safety could use to operate the network. The network that would fix the problems firefighters and police experienced during America's worst terror attack would finally come to life.
But somewhere along the way, former Maryland Gov. Martin O'Malley told StateScoop, things went wrong.
"FirstNet — I'm disappointed in it, really," O'Malley said. "It was one of the cool, bipartisan things that we all came together around as mayors and as governors."
O'Malley, who serves as an unsalaried member of the Rivada Mercury board, said his problem with FirstNet's approach is that states are being asked wait in line to pay to use something that already belongs to them.
"Administrations change, but the story's always the same," O'Malley said. "The federal government wants to take our public safety spectrum from us, [give it] to a giant national monopoly for a one-time price and pat us on the head and tell us they'll get to us with interoperable communications. And I don't think governors are inclined to buy it."
Once a few states opt out and others see that meeting the federal requirements is not an insurmountable challenge, O'Malley said, even more states will follow.
"If the federal government wanted to be helpful, they could have by making an appropriation," O'Malley said. "Instead, almost 15 years later, the solution they come up with is 'let us monetize your public safety broadband and eventually we'll get around to building a national network.' I wouldn't want to take that chance if I were a governor."
Without substantial user fees to support network costs under Rivada Mercury's proposal, an alternate means of income was proposed by the vendor in the form of yearly, monthly and real-time open market spectrum auctions similar to those found in electricity markets.
States opting out of the federal plan will be required to pay a yet undetermined user access fee to access core network elements, and the law notes that revenue gained by the state can only "be used for constructing, maintaining, operating or improving the radio access network of the state."
But a spectrum auctioning system conceived through research supported by Rivada Networks would theoretically provide adequate income to support network development and beyond, particularly as a growing Internet of Things spurs demand for wireless connectivity. Additionally, the researchers claim a demand-based pricing schema that opens bidding to anyone would spur welcome competition in an innovation-barren oligopoly now found in the wireless market.
Put simply by the researchers: "Network capacity goes to those willing to pay the most for it, and the price paid for capacity is set to balance supply and demand. When capacity is scarce, a high price is paid to efficiently allocate the available supply; when capacity is abundant (uncongested) there is no need to ration demand and the price is set to a nominal floor."
An analysis by consultancy NERA Economic Consulting published in March characterizes the researchers' claim of being able to squeeze more value out of the existing spectrum market as "extremely overstated." The report, which was cited by APCO International in a blog post, posits that if the spectrum exchange doesn't perform as anticipated, an opt-out state would not be able to afford paying network costs for the life of its contract. The author concluded that the costs for New Hampshire, Alabama and Arizona — three states now surveying the market for alternative plans — could reach as high as $48 million, $269 million, and $524 million, respectively.
A question of preemption
In interviews with StateScoop, FirstNet state contacts tended to repeat the same kinds of requirements when asked what they wanted from a national public safety network. Cost, coverage, reliability, resiliency, performance, device options and cybersecurity were common considerations named. Priority and preemption — first-in-line service for first responders to ensure they always have network access — always came up as an essential feature.
Arizona is among the states keeping its options open. It issued an RFP for an alternative plan last year and has received responses from three vendors, which will soon be evaluated, an official said. Rivada Mercury told StateScoop it was one of the companies seeking Arizona's business.
Tim Chung, assistant director of the Technical Services Division at the Arizona Department of Public Safety, said the reason the state is looking at other vendors now is to ensure there's enough time to consider alternative paths when the federal plan is released.
"Law enforcement, fire, and EMS communicating on one band — I think that's going to move law enforcement and public safety into the future," Chung said. "We have to have that."
A dedicated platform for sending photos, video, and location data to doctors, paramedics, police or helicopter pilots before they arrive on the scene will be a tremendous evolution in intelligence gathering and information sharing, he said.
In Arizona, 22 federally recognized tribes, rugged and rural terrain including the Grand Canyon — which receives more than four million visitors each year — make for a unique challenge. But the entire network won't amount to more than a collection decorative steel pylons if first responders aren't granted immediate access to the towers when they need them most.
Duane Schell, director of the Network Services Division at the North Dakota Information Technology Department, said the decision to opt in or opt out will follow a similar decision process to any other technology procurement. North Dakota is keeping an open mind until the details of AT&T's offer are presented, he said.
"Does it have public-safety grade capabilities? The big feature of that on FirstNet is talking about priority and preemption. How well does the solution they are putting forth align with the vision that FirstNet originally created, and how well does that vision align with the requirements we provided to FirstNet a year or so ago?" Schell asked.
How vendors implement preemption is one of the crucial battles that will determine who wins this war. All the administrative policies surrounding contracts won't matter a bit if the users can't get the kind of service they want. During a large event like the Super Bowl or an NCAA Final Four game, first responders attempting to join a cell network today enter the queue alongside thousands of civilians, and sometimes that can mean not getting access when they need it, an unacceptable proposition by all accounts.
Aware of how essential this feature is, both AT&T and Rivada Mercury promise a form of preemption, though the latter holds a patent on a technology it calls "ruthless preemption."
When a cell is full, a first responder enjoying regular preemption will become first in line to take a spot on the network when one becomes available, cutting in front of all others waiting to use the network who do not have priority access.
But ruthless preemption takes things one step further. Non-priority users are moved — or booted, as a last resort — off the network to free up an immediate spot for the priority user. This is the brand of preemption Rivada Mercury advertises. Motorola also holds a patent on a version of ruthless preemption.
In comments provided by email, FirstNet chief customer officer Rich Reed noted that AT&T is providing "true priority access to police, fire and EMS personnel on all of its LTE bands."
Who will control the switch — governors or the federal agency — that gives priority access to first responders over the public during an emergency remains unclear.
Within the past year, several sources told StateScoop that some states have no intention of perusing an alternative plan — that they don't have the resources or desire to meet the stringent requirements and added work that opting out would entail. A South Dakota official, for instance, reported the state doesn't think opting out is a viable option. The Virgin Islands and Puerto Rico, territories with limited populations and resources, also reported plans to opt in.
But like political candidates, vendors seeking control over the national network are vying for population numbers. An alternative vendor won't be able to wrest control of the core network from AT&T, but stealing their customer base would certainly change things. Opt out decisions from just a few large population states would have unpredictable influence on AT&T's plans. The federal agency is compiling a list of features the full breadth of which will become available later this year.
So far, FirstNet reports it will offer "rapid deployment" with 99 percent coverage of the population "expected by the end of 2017." It will offer access to more than 700 AT&T "deployables," including 72 "cell on light truck" (COLT) mobile command centers that will be dedicated to use by first responders. Constant access to customer service, network generation upgrades that parallel the commercial market, and a developer and apps ecosystem will also be included in the federal offering.
One unique feature of Rivada Mercury's offering is its high resolution radio propagation network design. Rivada Mercury says it was the only vendor to comply with the federal agency's request for maps with shapefiles "no greater than 30 [by] 30 meters" of America's entire landmass. Unsure if the request for this high degree of precision was necessary, the vendor followed up with the agency three times asking for more details before embarking on a gigantic technical project that no one had ever completed before.
It cost millions, Ganley said, and resulted in more than 12 terabytes of data — files turned over to the government delineated how 32 billion plots blanketing the U.S. would comprise Rivada Mercury's network coverage. The plan's coverage — deemed "exceptional" by the evaluator — wasn't enough to compensate for several "unacceptable" ratings and prevent an overall disqualification.
By comparison, AT&T's plot size, Ganley claimed, was 1 square mile. He said he knows this because the agency asked his company for plots of this size "for comparison purposes." Citing confidentiality requirements, FirstNet declined to confirm or deny details of any vendor's design.
"It turns out, we're the only bidder that complied with the hardest, most onerous, most technically challenging part of the RFP," Ganley said. "And someone showed up with only having done only a tiny fraction of that and they get deemed OK. I cannot begin to tell you how galling that is."
New Jersey's JerseyNet was as one of FirstNet's precursors — a dedicated, interoperable communications network for public safety serving the state's Route 21 corridor, Camden County and Atlantic City. The network showed how first responders could stay in contact across technology platforms, regardless of what was happening on commercial cell networks.
The vendor isn't important, said Eric Tysarczyk, director of the Policy and Planning Division at the New Jersey Office of Homeland Security and Preparedness — the only thing his state cares about is that it gets the best solution available. Like most states, New Jersey has yet to telegraph any intention of opting in or out of the federal plan.
Cost is a major concern, Tysarczyk said, but it comes second to the functionality the network will offer and the ability of vendors to provide definitive answers in the coming months.
"After all these years of meeting and talking about this network, if all the information isn't solidified, that's going to be a challenge and might be a credibility hurdle," he said. "The way the legislation is set up, they don't make it easy to do it on your own, and there are still some questions of how grant funds would be allocated for that, but we're prepared to impose a high level of scrutiny to get to the best product for the state and our residents."
Kevin O'Connor, assistant to the general president at the International Association of Fire Fighters (IAFF), said that while he anticipated FirstNet will be good for public safety, the "woefully inadequate" funding from the federal government combined with the fact that it's purely a data network means it can't and won't live up to the marketing hype.
"To me, FirstNet's a lot like kissing your sister," O'Connor said. "We recognize data is a very important component, but the simple truth is we've killed people because we haven't had good mission-critical voice communication."
While his group doesn't endorse or condemn specific vendors, O'Connor said he harbored "tremendous" reservations with the government's selection of AT&T.
"They're a large international conglomerate. Their model is based on retail. I don't think they will put enough emphasis into the public safety piece of it, and that causes me a lot of concern. I've articulated that even before the RFP was opened and we knew who all the bidders were," he said. "Basically where we are with it is: caveat emptor."
Editor's Note: This story was updated shortly after publication to clarify and correct several details surrounding the court proceedings, which led to the disqualification of Rivada Mercury and a contract award to AT&T in March. Minor adjustments were also made to the discussion of preemption, including an acknowledgement of a patent held by Motorola. Attribution surrounding technical details of AT&T's network were also changed. A reference to California was also added to the story upon discovery of an RFI published by the state in 2016.