New Jersey shares AI translation tool materials with other states

In collaboration with New Jersey and Google.org, the civic tech nonprofit U.S. Digital Response on Thursday announced it will share training materials for other states to use in building their own artificial intelligence-powered translation assistants for unemployment insurance systems.

Through work with the state’s Department of Labor and Office of Innovation, USDR is offering training materials that can be used with New Jersey’s new AI translation assistant. The state deployed the assistant for its staff in April to enhance language access, particularly for Spanish speakers who make up 95% of language access requests in the state’s unemployment cases.

To power the AI assistant, and with help from USDR experts and manpower from Google.org fellows, the state spent more than a year drafting a new glossary of unemployment terms in Spanish, which were vetted by bilingual call center agents, to train the AI assistant.

New Jersey officials said the AI assistant has already tripled translation speed for the state’s call center employees, with quality on par with human translators. Unemployment agencies in other states can use the materials to help development of their own AI translation assistants. According to New Jersey, the resources are compatible with any off-the-shelf large language model, such as ChatGPT, Claude or Gemini.

Gillian Gutierrez, the senior adviser and director of New Jersey’s unemployment insurance modernization project, said the effort began shortly after Gov. Phil Murphy took office in 2018, when officials began discussing how to improve access to benefits programs.

“Whether it’s unemployment, whether it’s paid family medical leave, or whether it’s a workers right, like paid sick days, language has always been a part of that, but one of the challenges that we faced is, how do we make sure that we feel confident [in that translation],” Gutierrez told StateScoop. “The government is always risk averse, right? It’s worried about its legal responsibilities, which it should, right? But how do we manage the risk of offering a translated document that represents the state?”

The COVID-19 pandemic exacerbated the need to improve access to benefits, as did a legal mandate last January. The new law says the state must improve language access through interpretation and in written documents, including for unemployment insurance.

Guiterrez said that in 2022 her department began looking to use the specialized knowledge that the state’s bilingual UI call center agents have in translating the “very jargony” and technical terms related to unemployment used by some Spanish-speaking residents.

“How can we use the knowledge that the agents have and the time that they have spent developing these ways of explaining difficult concepts in order to speed up the process for us to get real, trustworthy translations in front of people and getting them better access, better understanding, better knowledge of what their rights, what their responsibilities are, and what we’re asking them to do. So in the end, they can get the benefits they deserve,” she said.

Guiterrez said the Department of Labor looked at how to expand and improve the state’s glossary of terms found on its unemployment website. She said this was done with the help of call center experts and involved a lot of testing to make sure the descriptions were accurate.

Translations of the glossary terms were also tested against the knowledge of bilingual call center experts, as well as outputs from “other non-informed translations” produced by Google and ChatGPT. This phase of testing and research, she said, took the longest.

U.S. Digital Response helped New Jersey translate the glossary into plain language over a span of eight months. Plain language is a set of federal communication guidelines developed by the Plain Language Action and Information Network that informed the Plain Language Act of 2010, and the style involves grammatically correct and universally understood language that includes complete sentence structure and accurate word usage.

“When we wrapped that project and then published the glossary in the translation guide, that’s when generative AI became commercialized. And just like all of a sudden, ChatGPT blew up,” said Marcie Chin, the language access product delivery manager at USDR. “We started reading all these things around, like research around the translation, how well these large language models actually perform when it comes to language translation. And we thought, ‘Well, we just developed all of this very human centered training data.'”

“These AI models are only as good as the data that they’re being trained on. And so we were really excited that we happened to have, again, all of this really human-centered, really high quality, accurate, vetted data that we can use to start training these large language models,” Chin continued.

Gutierrez said the state is testing additional uses of prompting the assistant, such as integration with other state services, applications, websites and communications beyond just the unemployment system. She said this might include other benefits programs, health care portals or education resources.

Krista Canellakis, USDR’s digital delivery program director, told StateScoop that New Jersey’s solution offers a model for states still exploring how best to use AI, and that the materials can help them “de-risk” their approaches.

“A lot of governments are reasonably hesitant to use these AI technologies for translations because you can’t tell how accurate are these translations, especially when you look at existing technology like the Google Translate widget that is on most government websites. That technology, it translates in a very literal sense versus a contextual sense, so the quality of those translations are not always entirely accurate for one,” Canellakis said. “So in this case, we’re really excited that we’re de-risking translations for governments, because they’re being co-developed with policy experts and bilingual frontline staff, and then also de-risking it for the end user as well for Spanish speaking residents or workers in New Jersey.”

Corrected Dec. 6, 2024: The story was updated to reflect that the announcement was made by U.S. Digital Response. Clarifications were also added regarding how the training materials can be used and what New Jersey is planning next.