New York City Council passes landmark AI oversight package

New York City Council on Tuesday unanimously passed the GUARD Act, a sweeping package of bills designed to bring transparency and accountability to the city’s use of artificial intelligence tools.

The legislation creates an independent oversight office, sets mandatory standards for fairness and transparency and creates a public list of every AI tool the city uses. Council member Jennifer Gutierrez, who chairs the council’s technology committee and led the legislation, said the package creates the guardrails the city urgently needs.

“We just got through a campaign cycle here, and [AI] was a big topic about how opponents were utilizing AI tools for the purpose of campaigning, so it’s still very fresh in everyone’s minds,” Gutierrez said in an interview. “I think everyone wants to get behind more transparency and better functioning government that’s going to protect people.”

The centerpiece of the legislation is its establishment of an Office of Algorithmic Data Accountability, an independent watchdog tasked with reviewing, auditing and monitoring AI tools before and after agencies deploy them. The office is also required to investigate public complaints and publish a directory of every AI system it evaluates, giving New Yorkers more transparency into how automated tools are being used, and publish a list of all AI systems it reviews.

“This office should function in this way to keep New Yorkers safe, to ensure that we’re being transparent, that we’re disclosing with the public tools that are being used and that we’re working really hard to check biases when they’re reported,” Gutierrez said.

She said that for years city departments have used both AI and other automation tools to help make decisions about housing access, policing and benefits distribution, but that they’ve operated with almost no oversight, leaving residents vulnerable to biased or error-prone algorithms.

In May, The Markup reported that the city’s Administration for Children’s Services had been using an algorithm to flag families for heightened scrutiny, using AI to predicting which children were most likely to experience harm, without informing parents, attorneys or even caseworkers.

In 2023, the city started enforcing Local Law 144, which regulates the use of automated employment decision tools across city agencies, to prevent bias in hiring decisions and provide annual tool audits.

“This administration had been essentially like piloting some services through AI, but did not really disclose a whole lot of information,” Gutierrez said. “They were not very forthcoming with contracts, with whether or not the like tools were being checked for biases, especially for agencies that were using these tools for public services. It just became evident that the topic of AI tools couldn’t just live under this like new agency. It had to be separate.”

She said that the city’s AI Action Plan, published in 2023 under Mayor Eric Adams, offered guidance, but still left city agencies to navigate the technology on their own, resulting in inconsistent use.

“This administration, I think, very smartly put together a paper which had a really good set of pillars and recommendation,” Gutierrez said. “But they were just recommendations. None of it was going to be enforced.”

The legislation also includes provisions that create mandatory citywide standards requiring agencies to protect residents’ data privacy, and to test AI systems for fairness and undergo independent evaluations before being launched.