‘Tools, not rules’: California publishes generative AI procurement guidelines

The California Government Operations Agency on Thursday released procurement guidelines and a toolkit for state agencies considering software that includes generative artificial intelligence.

Mandated by Gov. Gavin Newsom’s 2023 executive order, the procurement guidelines push state departments to first identify a need for generative AI and communicate with the employees or teams who would use the technology as part of their jobs before making their purchasing request.

The toolkit, developed by California’s Office of Data and Innovation, is designed to be used in conjunction with the guidelines and aims to help government agencies identify understand the technology, which can create text, video or images in response to prompts. The guidelines and toolkit encourage the state government to consider generative AI’s potential uses and risks and provides employees with critical knowledge of its appropriate use.

Jeffery Marino, director of the state data office, said California agencies should see the procurement guidelines as “tools, not rules.”

“We don’t have all of the answers, and that is really core to a human-centered design in general,” Marino said. “But it’s really exciting to think about how we can use that approach as it relates to guidance and policy.”

According to the guidelines, state government departments must also write an assessment of the potential risks and benefits of generative AI tools they intend to use and test them for bias and accuracy. Each state agency must also establish a team to monitor how the technology is being used and report its findings to the California Department of Technology.

The Department of General Services will review procurement applications, assist with risk assessment and help monitor testing, but the decision to purchase any generative AI technology is up to each agency.

“At the end of the day, the departments themselves are responsible for whatever risk or unintended consequences that may result from the use, and implementation of the technology,” Marino said. “And that is why, in these guidelines, we discussed the need for assembling a team at every department and providing all of these tools to them, so they can be as successful as possible.”

Intentional vs. incidental

Jared Johnson, California’s deputy chief information officer, who also worked on the procurement guidelines, told StateScoop that the state — home to 35 of the largest generative artificial intelligence companies — is the frontrunner for using generative AI in state government and that this is just the beginning.

“It’s the next step in developing that overarching playbook in providing state leaders with more information,” Johnson said of the procurement guidelines. “It’s also a starting point for us to begin to evaluate how the technology is working its way into the work environment.”

The procurement guidelines distinguish between incidental purchases — generative AI included as part of another product — and intentional cases of generative AI chosen to meet an operational need.

“There’s AI tools built into products that many of us are using every day, that’s not necessarily their main focus, but they’re built into small functionality, so we consider it just a work productivity tool.” Johnson said. “Intentional [purchase] is when a department goes out and says, ‘We have this use case and we think GenAI is the right technology solution.’ That means they’re really focused on gen AI as a technology, rather than procuring it as a component for something else.”

California ‘on the forefront’

Johnson said he expects the state government to explore using generative AI in a few different areas, including modeling highway designs that offer better traffic flow, improving roadway safety, reducing call center inefficiencies, offering residents more language options in health and human services and expediting health care license inspections.

“A lot of times new technology comes around, but it’s not really a game changer,” Johnson said. “Gen AI is going to be a game changer, so it’s nice to be on the forefront of learning about this technology and preparing the state.”

The procurement guidelines will take effect July 1. Until then, Johnson said, the guidelines help departments assess the risks of potential uses of generative AI and determine their preparedness before proceeding with any purchases.

“There’s also going to be a comprehensive engagement strategy with state leaders to ensure that they have ample opportunity to hear from the experts on the intent of the guidelines and to understand how to operate within them,” Johnson said. “We’re also creating a repository, a digital website that will essentially be a digital version of the toolkit and will allow state leaders to understand the overarching ecosystem of information.” 

Corrected March 25, 2024: This story was corrected to show that the California Department of General Services is involved in reviewing AI procurement applications.