20260205 Halluzinierende KI

Hallucinating AI

Progress seems unstoppable — even when it produces utter nonsense. In the US, for example, more and more police reports are being written automatically based on bodycam footage recorded by police officers. And increasingly, this is being done not by the officers involved on the ground, but by office staff and, more recently, by AI-powered software called Draft One.

False Positives

The notorious "wrongly accused" have been with us since the association was founded 17 years ago. Misinterpreted evidence, such as surveillance software, facial recognition, etc., forces them to prove their innocence themselves, as the presumption of innocence suddenly no longer applies to them.

This now also affects the Heber City Police Department in the US state of Utah. There, an official report — the AI-powered software Draft One — claimed that an officer had turned into a frog. This caused problems not only for the officer in question. Salt Lake City-based Fox 13 reported on the case and subsequently inquired with the police department.

In our position, we would not have been satisfied with the official explanation: in the bodycam recording, the Disney film "The Frog Prince" was playing in the background on a television. The necessary demand for such insight is, at a minimum, the scrapping of Draft One — AI must never be allowed to write police reports without human review!

According to t3n.de, critics warn that AI-supported systems could reinforce existing racist or gender-specific prejudices. Studies have shown that generative AI models tend to reproduce discrimination against women and non-white people. The fact that it is impossible to distinguish in the reports which parts are AI-generated and which are written by humans is also a flagrant violation for subsequent legal proceedings based on them.