Automation helped NYC cut its database patching time from hours to minutes

Until a few months ago, when one of the more than 800 databases operated by the New York City government needed a software patch, an engineer with the city’s Office of Technology and Innovation had to implement it manually. 

But thanks to an initiative led by Sridhar Doraikannu, OTI’s director of database engineering and operations, the process of patching the city’s databases is now automated. The city claims the effort has cut the duration of the software patching process from approximately four hours down to an average of six minutes, creating a 97% reduction in downtime for city databases.

The effort led Mayor Eric Adams on Tuesday to name Doraikannu one of 19 winners of this year’s Hayes Innovation Prize, a recognition of early-career professionals who help solve difficult problems. (The award program launched last year as a joint effort by OTI and the Hayes Prize Foundation.)

Vendors regularly issue software patches to resolve vulnerabilities and add new features. In the age of data breaches and cyberattacks, regularly patching software is a best practice commonly cited by organizations like the Cybersecurity and Infrastructure Security Agency, but can render services inaccessible during the updates.

Before automating the city’s software patching, Doraikannu told StateScoop, New York City’s database software vendors issued patches quarterly and that testing and applying them took several months.

“It is not just the application of the patches,” he said. “It also involves coordination with the agencies and we are doing that to ensure the agencies know that the data will not be available when we are patching and that there is a downtime.”

Now, instead of manually coordinating with city agencies on when their databases may be down, the automated processes pick up patched when they’re issued and notify agencies when to expect downtime.

“It takes a lot of time to patch a database,” Doraikannu said. “And if we can automate and provide that same service, we can reduce the downtime window. Thereby, we can help the New Yorkers with very minimal impact so they can have better service and availability. So that was the main thing which drove us to get the automation done.”

Doraikannu said that the first round of automated patches were completed last week and that receiving the Hayes Innovation Prize for the work was additional motivation for undertaking larger projects.

“We were a little bit scared that this is too big of accomplishment — going from the manual, working with agency automating sending the email to the customer, that we are going to take the database everything to be automated — but we accomplished this. This was really an uplift — motivation — not only to me, but from a whole team.”