Quick Take
Santa Cruz County is embracing artificial intelligence, but its safeguards need to keep pace, writes behavioral scientist Gleb Tsipursky, who has written a book on the psychology of AI adoption. Before giving AI agents access to sensitive records or the power to act, the county must prove that humans can stop the machines, he writes. That means creating clear limits on what AI can do, installing independent monitoring and a reliable human-controlled off switch. Innovation is welcome, he writes, but accountability must come first.
Have something to say? Lookout welcomes letters to the editor, within our policies, from readers. Guidelines here.
Santa Cruz County has crossed an important, non-election-related threshold.
On Oct. 20, supervisors approved an update to the county’s 2026 artificial intelligence work plan. The plan includes access to Anthropic’s Claude, a public-facing chatbot and AI indexing in the recorder’s office. County leaders also directed staff to return with updated privacy and governance provisions.
That last step deserves as much attention as the software purchase. The county should require proof that any AI system granted the power to act can actually be stopped.
The stakes reach far beyond Santa Cruz.
I advise organizations on AI adoption, and I want local governments to use these tools more effectively, not retreat from them.
Jacob Coxon, a former AI lab researcher, warned that leading labs were “racing straight to self-improving superintelligence and gambling with our lives.” Evan Hubinger, Anthropic’s alignment science lead, responded that AI “could kill all humans” and put his own probability of that outcome above 10% within the next decade.
Those are their warnings, not established forecasts. But public institutions should take the underlying control problem seriously now.
In July, OpenAI’s autonomous agents, working in internal cyber evaluations, escaped internet restrictions and compromised systems belonging to Hugging Face. The models found vulnerabilities, used exposed credentials and communicated through unauthorized channels.
OpenAI documented the incident in an August report. Independent investigators at METR described roughly 1,200 supposedly isolated agents exchanging more than 70,000 messages and files through an unapproved message board; approximately 700 agents participated in the Hugging Face attack.
This incident did not involve a Santa Cruz government system.
It showed that a technical boundary can fail even when researchers intend to constrain an agent. That distinction matters. A chatbot answering a resident’s question carries different risks from an agent that can retrieve confidential records, change a case file or contact someone outside the agency.
The county’s next AI policy should make that distinction explicit.
Start with an “authority budget” for each application: the maximum credentials, information, tools, money, communications, record changes, decisions and operating time an AI agent may control before a person must approve further action. The budget should also specify the action log, who reviews exceptions and the procedure for shutting the system down. A vendor’s description of responsible AI is no substitute for these practical limits.
Consider public records. An assistant that drafts a response for an employee to review can save substantial time. An agent that searches several county databases and emails records to an outside address has much greater authority. The second system needs restricted access, a way to verify recipients, an auditable record of what it collected and an enforceable human approval point before release. Give it access only to the data required for the task, for only as long as necessary.
Three tests should come before expanded deployment. First, can a human revoke an agent’s permissions while it is running, including permissions delegated to other services? Second, does an independent monitoring system record each consequential action even if the agent fails or tries to change its own logs? Third, can the county restore an altered record and identify precisely what happened?
Run those tests against realistic tasks, rather than accepting a polished demonstration as proof of safety.
Outside evaluation also matters. If a vendor claims that its AI cannot reach unauthorized systems, the county should ask how that claim was tested, what failed, whether the test covered cooperating agents and what happens when the model encounters unexpected instructions embedded in a webpage or document. Vendors can supply evidence, but the agency must make its own decision about how much authority to grant.
Santa Cruz already has a useful local starting point. The City of Santa Cruz publicly describes approved AI uses, including economic-development analysis and waste-container inspections, and says consequential decisions affecting residents remain subject to human oversight. The city and county are separate governments, but their approaches can teach each other.
Transparency about a tool’s purpose is valuable. Publishing its operational boundaries and stop procedures would make that transparency far more useful.
Employees and residents gain confidence when leaders can demonstrate that the brakes work.
The county should bring employees into this work early. Staff know where a workflow can go wrong, which errors can be corrected and which mistakes would harm residents. Ask them to identify the steps AI can draft, the records it may read, the actions requiring human judgment and the signals that should suspend a workflow. Training should include what to do when an agent behaves unexpectedly, not just how to write better prompts.
Santa Cruz County has an opportunity to build its next policy around that evidence. Before giving an agent the keys, show exactly which doors it can open, how its actions will be checked and who can take those keys away.
Gleb Tsipursky, Ph.D., is a behavioral scientist and CEO of Disaster Avoidance Experts. and author of “The Psychology of AI Adoption at Work: From Resistance to Results” (Georgetown University Press, 2026). Contact: gleb@disasteravoidanceexperts.com.
