Building public-sector gen AI tools? New Jersey has some tips

This month, New Jersey’s Office of Innovation shared a guide for other governments to use when building and working with generative artificial intelligence tools.
The guide — 9 Tips for Building GenAI Tools for Public Sector Professionals — is based on what the state has learned so far building tools like the New Jersey AI Assistant, or the state Department of Labor’s AI translation assistant, which was built last year with help from Google.org and the civic tech nonprofit U.S. Digital Response. The author of the report, Jessica Lax, senior adviser for responsible AI with New Jersey’s Office of Innovation, said these experiences were opportunities to share with other states, cities and even other state agencies in New Jersey what they’ve learned.
“I think it’s just a really great opportunity — as all the other states and localities are starting to leverage AI — to create a space for folks to share those learnings, so we’re not all kind of trudging through mud in the same ways, and we’re building off of each other and having more rapid advancements,” Lax said.
By distilling development of internal generative AI tools, and working through issues such as timeouts and data-processing hurdles, Lax broke the guide down into nine tips. The first two are “Align On Your Responsible AI Approach,” and “Start with Product Fundamentals” — both of which, Lax said, are key strategies to understanding and unifying a strategy.
The third is to “Test Your Use Case,” which Lax said is also strategy-based, and important to determine if generative AI is the right tool to solve the problem at hand. The fastest way to do this, the guide says, is to test the idea using an illustrative example with a secure generative AI chat interface.
“Working with these teams, [it was] just interesting to hear about some of the challenges they were facing and the different issues they were running up against, and the surprises they were seeing,” Lax said. “Working collaboratively with a lot of our engineering teams, we were starting to see that there was a need for this — to share what was happening so folks could more easily address the challenges as they were facing them, as well.”
The fourth and fifth tips — “‘Choose Your LLM and LLM Product” and “Decide Your Tech Stack” — are more technical. The guide lays out how New Jersey went about making these decisions, along with what worked and what didn’t. The sixth and seventh tips relate to improving the quality of data to be used. Improving data quality and size will also improve the functionality of a generative AI tool, the guide says.
The eighth tip, “Manage Timeouts” — time limits for an LLM to complete a requested task — is filled with details about how to keep the technology working and strategies to work around timeouts, such as code optimization or offloading heavier tasks.
The ninth tip is “Evaluate and Scale Your Product.” The guide asks that users rank the experience of using a tool along the measures of accuracy, completeness, bias and tone, among others, and then use that information to make improvements.
Lax said the guide itself will continue to be improved, along with the other documents for public use on the office’s site, such as the GenAI Prompt Toolkit for Designing Government Call Center Menus.
“This is meant to be a living document, we’re going to be updating it,” she said. “We also have other guides on our site, a good resource as we keep doing this work in AI. We intend to continue to share what we learn and hope that it benefits other people.”