Image credit: SFC Art Department
By
Last summer, New York City’s Administration for Children’s Services (ACS) used an AI tool to categorize families by various “risk” factors. The tool’s algorithm effectively scored the chances a child will be severely injured in a household based on various factors such as neighborhood, a mother’s age, and the number of siblings a child has.
According to reporting by The Markup, the algorithm operated in complete secrecy, using analyses of 279 variables derived from cases in 2013 and 2014. However, as noted by the Electronic Frontier Foundation, “it’s unclear how many cases were analyzed, what kind of auditing and testing was conducted, and whether including data from other years would have altered the scoring.” In addition, ACS staff have admitted that the agency has been more punitive towards Black families—a practice that the algorithm was likely to amplify.
As cities rapidly adopt artificial intelligence to improve and expedite operations and service delivery, a new challenge has emerged: how to ensure these systems are transparent, fair, and accountable.
In 2023, under Mayor Eric Adams, New York City attempted to address these kinds of issues with the Artificial Intelligence Action Plan. At the time, the city’s Office of Technology and Innovation (OTI) laid out a framework of initiatives and commitments to responsibly leverage its tools. But over the last couple of years, the plan’s principles and guidance were voluntary to integrate, and lacked enforcement and compliance. Many agencies were left on their own, deploying AI tools without any centralized review or accountability.
That changed last November with the passing of the GUARD Act (Guaranteeing Unbiased AI Regulation and Disclosure), one of the first comprehensive municipal frameworks intended to bring oversight and transparency to the city’s AI and automated decision systems. According to council member Jennifer Gutierrez, chair of the committee on technology, the legislation is built around letting the public know how the government plans to use its algorithmic tools and to create the necessary guardrails to prevent equity concerns (like ACS) from occurring.
“The city should have a moral compass when it comes to using these tools, because these examples exist,” Gutierrez says. “There’s an impact for a parent who is unaware that she’s being tracked by a city agency based on predictive AI tools.”
The GUARD Act is a crucial step in a new era of governance, and it’s one that other cities are attempting to follow as AI becomes a formidable part of their operations. In Washington, D.C., an AI Values Alignment Group holds public listening sessions to solicit public input on the appropriate government use of AI tools, while city leaders in Lebanon, New Hampshire, maintain an Algorithm Registry that provides transparency on its use of AI. And in Chattanooga, Tennessee, the city government created a “prompt library” for staff to use generative tools effectively and ethically, streamlining tasks across various departments.
Arguably, the most impressive measures being taken belong to San Jose, California. Under chief information officer Khaled Tawfik, the city crafted a list of basic AI guidelines in 2023. Not long after, they turned them into comprehensive policies (thanks to input and feedback from cities across the country), which included an algorithmic impact assessment mandating that vendors that use AI complete an AIFactSheet to explain how they source and manage data. It’s a way for the city to start developing trust with frequently-used third parties, which offer everything from traffic-light systems to automated paperwork and invoice services.
In essence, Tawfik says, the factsheet eliminates vendors from guessing what protocols the city wants them to maintain, creating better communication and clearer expectations. “We add language in our contracts to require vendors to not only tell us what they’re doing today, but also puts the burden on them if they change their AI strategy,” he says. “What we learned through the process is that vendors are really welcoming that engagement.”
That kind of transparency and organization can be just as important for internal usage. In New York, Gutierrez said that, as a former staffer, she noticed various constituencies received different messaging because offices had been using different AI tools for campaign fliers, emails, and other back-office content. “It got to a point where nothing looked uniform, nothing was consistent, messaging was crazy,” she says. It’s a small example of the importance of ensuring vendors have a clear, consistent method of communication and that government agencies are aware of how tools are used.
“We do need to audit more,” Gutierrez says. “Being able to create a system where agencies are using the same model to communicate, to share updates, to be transparent, and to make decisions is really important.”
One way San Jose is thinking about large-scale transparency is through community engagement. Each year, the city receives about 300,000 non-emergency 311 calls, usually made to report potholes, noise complaints, and other daily disruptions. As a way to be more proactive about these issues (and thus ensure more efficiency within government), Tawfik notes that San Jose instituted a pilot program that added cameras to sanitation and parking enforcement vehicles that detect and capture imagery of poor infrastructure and graffiti—with an AI model helping analyze everything the camera picks up.
“The goal is not only to detect graffiti, but to interpret graffiti,” Tawfik says. “If I can interpret the graffiti tagging as a message for violence, then I need to immediately notify police that we need to take care of this. This is really when we completely shift our way of doing business.”
But the addition of new technology and added surveillance lacks efficacy if it’s not explained to impacted residents. According to Chelsea Palacio, the city’s public information manager, any facial data that’s collected is anonymized, and only still photos are sent to the city, not full videos. “It is trained on detecting what a pothole looks like, what trash may look like,” she says. “But it’s not trained to actually identify anything else.” To instill further trust, Palacio frequently attends field events with examples of the cars and cameras, making sure to explain the full scope of how AI is collecting neighborhood data and fielding questions from the community to guide the city’s efforts.
“Once you lose trust, it’s really hard to regain it,” Tawfik says. “We’re cognizant that certain things we’re going to do with AI, the public will never see. And it’s important for us to understand why and intentionally say the public’s not going to see it. But when we do operational changes for the public, we need to engage them.”
The GUARD Act encourages similar levels of transparency and accountability, demanding that agencies disclose how and why they’re using AI tools so that the public has a better understanding of their results and the decisions behind them.
“How is this going to improve your agency? It can’t just be, like, ‘It was cheap, and this is why we’re using it,’” Gutierrez says. “You have the right to ask if you’re interacting with a city agency—and if an unfavorable decision came down, you should know whether a tool was used to make that decision and how you can appeal it.”
The biggest concern Tawfik has about the city’s framework is AI’s rate of change.
As more capabilities are introduced over the coming years, he can see private enterprise and vendors potentially cutting corners as competition increases. But he’s confident that the city’s coalition of other leaders and agencies across the country will help mitigate that risk, citing this winter’s GovAI Coalition Summit, which brings together private-sector, non-profit, and government agencies from seven different countries to discuss and further solidify the trends and guardrails already in place.
“Everybody can negotiate a curb when we’re driving, but when it’s raining, and you’re driving too fast, it really reduces your ability to control the vehicle, and that’s where the majority of people make mistakes,” Tawfik says. “We’re in a better position doing it together.”
Gutierrez has also been interacting with other city officials, sourcing ideas and attaining information about random pricing, housing needs, and other issues AI has a role in affecting. Ultimately, she hopes that over the next few years, legislation like the GUARD Act will equip cities to better unmask the entities behind some of these AI models and keep residents informed about how tools are playing a role in their lives.
“Government will forever need to be a person-to-person interaction. There’s no way around it,” Gutierrez says. “You want it to feel like we’re still engaging with humans.”
