What happened in New York?
In 2023 New York City launched a chatbot for businesses inside its MyCity portal, built on Microsoft's cloud platform. It was meant to help owners find their way through rules and permits. In 2024 tests by The Markup and THE CITY showed answers that, if followed, led to breaking the law.
- The chatbot said an employer could take a cut of workers' tips.
- It suggested landlords could turn away tenants with federal housing vouchers, known as Section 8.
- It said a shop could refuse cash, although a city law forbids it.
The administration at the time kept the chatbot online and promised to fix it. In 2026 the new mayor, Zohran Mamdani, announced it would be shut down, calling it 'functionally unusable'. The service page now says the beta test has ended and points to the city's main portal.
What does Estonia do differently?
Bürokratt is the public virtual assistant run by Estonia's Information System Authority. It understands everyday Estonian, is available around the clock and helps people find information on public services. When it gets stuck, it passes the citizen to a customer-service person. National agencies use it, such as the tax board, and so does one local authority, Rae Parish.
Our reading: the difference is not the AI model, it is the design. In New York a small note said the bot might be wrong, but users had no way of telling when. In Estonia handing over to a person is part of the service from the start.
What does the law ask of a public chatbot today?
- Say it is an AI system. Article 50 of the EU AI Act requires that people talking to an AI assistant know it, unless it is obvious. The information must be clear and given at the latest at the first interaction. It applies from August 2026.
- Label generated text. If the authority publishes AI-written text to inform citizens on matters of public interest, it must say so, unless a person reviews it and holds editorial responsibility.
- Make it understandable. Italy's Law 132 of 2025, Article 14, requires public bodies to make the way their AI works knowable and its use traceable. Responsibility stays with a person at the authority.
If the chatbot handles personal data, for example a tax code for a waste-tax query, GDPR applies too: privacy notice, legal basis and security measures, as for any other online service the authority runs.
How do you keep a chatbot to the authority's own sources?
A worked example, not a real authority. A council's public information office wants an assistant for waste tax, residency and low-traffic-zone permits. Questions about fines, appeals and disputes always go to a person.
- Answers only from the authority's pages and regulations, each with a link to the source.
- A written list of excluded topics, such as fines, appeals and health, that go straight to a person.
- A test before launch on a few hundred real questions received by the office, reviewed by an officer.
- A button to talk to a person, always visible, not hidden after three attempts.
- A log of answers, with spot checks every month.
Where do we stand?
How we would set it up for the office in the example: start with one topic, such as waste tax, with the authority's pages as the only source. The handover to a person would be ready before launch, and we would extend it to other topics only after the first month's spot checks.
So do you go ahead?
- If every answer can come from the authority's pages and regulations, you can start one topic at a time.
- If citizens mostly ask about personal cases, such as fines or appeals, build a fast route to a person first.
- If nobody will review the answers every month, wait: a chatbot without checks is the New York case.
“A public chatbot works when it knows when to stop and hand over to a person.”
DBrothers — on council chatbots
FAQ
Is a council chatbot high-risk under the AI Act?
Usually not, if it gives information about services. The high-risk rules cover, among other things, systems that help decide who is entitled to a service or a benefit.
Is a notice saying the chatbot can be wrong enough?
No. New York had one, and it was not enough. You need answers tied to sources, excluded topics and a person who can be reached.
Who is responsible if the chatbot gets it wrong?
The authority. Under Italian law AI plays a supporting role and responsibility stays with the people in the administration.
If you are considering an assistant for citizens, start from the most frequent questions your office receives and the ones you never want to leave to a machine. That tells you what the chatbot can do.
