NYC mayor’s report blames IT department for 11-day wireless outage

A report released Friday by New York Mayor Bill de Blasio’s office places blame for April’s 11-day outage of a wireless network used by the city’s public safety and sanitation agencies directly at the feet of the Department of Information Technology and Telecommunications.

The New York City Wireless Network, or NYCWiN, went down from April 6 to April 17 after DoITT failed to install a vital update connected to a change in the Global Position System for which the department had advance warning for more than a year. GPS operations worldwide calculate the precise date using week counters, which need to be reset to zero every 1,024 weeks, or approximately 20 years. Without the reset, a GPS-supported system would be set to the incorrect date, leading to other failures.

But despite knowledge of the needed reset at DoITT and the other city agencies that rely on NYCWiN, including the New York Police Department, which uses it to connect its license-plate readers, the network did not receive the week-counter reset. The report, which was compiled by the consulting firm Gartner, calls out DoITT for outsourcing its IT maintenance tasks without consistently monitoring the contractors it gives those tasks.

“The heavy reliance on the long-standing outsourced maintenance and support model was not counterbalanced by a consistent, well-executed vendor management discipline within DoITT,” the report reads.

In the case of NYCWiN, Northrop Grumman, which won a city contract in 2006 to build and run the network, was slow to notify DoITT after the outage began. While the network glitched out right after GPS rollover occurred at 8 p.m. on April 6, Northrop waited almost three hours to inform DoITT, according to the report.

The department had also given little attention to NYCWiN in its emergency and continuity of operations planning, which contributed to a lack of reliable data analytics about the wireless network, the report states. The outage was two days old before DoITT established a reporting process in conjunction with New York City Emergency Management, though once that process was established, agencies were able to provide updates on the number of devices affected and restoration timelines.

The Gartner report is frequently redacted, and the document makes little mention of Samir Saini, who stepped down as DoITT’s commissioner on June 14. But it does tell a tale of a wireless network that — while advanced at the time of its launch — is barreling toward the end of its life.

When NYCWiN came online in early 2007, DoITT’s then-commissioner, Paul Cosgrave, hailed it as “the most aggressive commitment by any municipality in the country to provide a next-generation public safety network.” And at its peak in the early 2010s, 29 city agencies connected to NYCWiN to support nearly 300 applications and 800,000 devices.

Only 10 use it now, though many of the functions it supports are vital to New York. During the 11-day outage in April, for instance, building inspectors could not access their agency’s database remotely, the Department of Environmental Protection’s air- and water-quality monitors could not transmit data, and about 200 traffic cameras operated by the Department of Transportation lost the ability to collect images, limiting the department’s ability to give the public real-time updates about road congestion.

The city has had plans in place for several years to retire NYCWiN and move municipal agencies’ wireless internet needs to commercial providers, which officials have said will be cheaper than the nearly $40 million it costs to maintain the network every year. Shortly before her retirement in 2018, Saini’s predecessor, Anne Roest, told the New York City Council transitioning off NYCWiN could bring the city’s Wi-Fi costs down to $10 million a year.

The Gartner report, which was given to New York officials in late April, is not meant to be the final word on the NYCWiN outage. But it knocks DoITT one more time for having not completed its own internal review of what led to the GPS lapse.

“Although DOITT has stated that they are working on creating a post-incident report, a formal report had not been issued as of this assessment’s publication,” the document reads. “The extent of DoITT’s post-incident review, including which external stakeholders are involved, is also not clear.”