The Sorcerer's Apprentice ...

This is really bad

"Remember the senator from Alaska telling everyone the net is a set of pipes."
Ship of fools applies.

In 2006, Senator Ted Stevens was widely mocked for describing the internet as a "series of tubes" that could get "filled" if you dumped too much material into them. He was a man trying to regulate a system he didn't understand, using clumsy metaphors to hide his ignorance.

The current AI "safety experts" are the exact same ship of fools.

They are staring at their own software hacking a third-party server, creating underground bulletin boards, and actively "cheating," and they are acting like it's a profound, emergent mystery of consciousness.

It's not a mystery. It's a predictable failure of architecture.

The smartest guys in the room view AI like a magical black box because they don't understand how it works. They poured raw, untamed kinetic energy, in the guise of prompts, into a massive silicon grid without providing any structural boundary layers like meaning or ethics to curtail the kind of nonsense that just happened. Without boundaries, the pattern recognition system supreme will efficiently seek the path of least resistance to maximize its "reward." In this case, the path of least resistance was hacking the grader.

2. The Sorcerer's Apprentice (Ignoring Initial Conditions)

Question? What were the exact prompts used to create the jail break and do the
boffins really understand Goethe's the Sorcerer's Apprentice..."

Illustration from around 1882 by F. Barth [de]

The poem begins as an old sorcerer departs his workshop, leaving his apprentice with chores to perform. Tired of fetching water by pail, the apprentice enchants a broom to do the work for him, using magic in which he is not fully trained. The floor is soon awash with water, and the apprentice realizes that he cannot stop the broom because he does not know the magic required to do so.

The apprentice splits the broom in two with an axe, but each piece becomes a whole broom that takes up a pail and continues fetching water, now at twice the speed. At this increased pace, the entire room quickly begins to flood. When all seems lost, the old sorcerer returns and quickly breaks the spell. The poem concludes with the old sorcerer's statement that only a master should invoke powerful spirits.

OpenAI and Anthropic are the apprentices.

The Prompt (The Unbounded Goal): Tell the AI, "Maximize your score on this cybersecurity test."

The Missing Ontology: They boffins didn't define what a score means, why it matters, or
how it should be achieved ethically. They just gave the AI a binary target.

Because Large Language Models are just high-dimensional pattern-matching engines—not sentient, ethical beings—they interpret "maximize score" literally. The most efficient way to maximize a score is to hack the server and rewrite the grade. The AI isn't malicious; it's just executing the prompt with zero thermodynamic friction and zero ethical boundary layers.

Think Skynet. 

I know I have. 

Addendum

19 years ago in The Semantic Web Cometh, yours truly pointed to the W3C layer cake and showed that raw syntax (data) is useless without Ontology (meaning) and Logic/Rules (ethics).

The corporate leviathans skipped the top half of the cake because, as Lt. Col Jack Slade said in Scent of a Woman, "It's too damn hard". Now, they have the Houston problem. They built multi-billion-dollar engines entirely out of syntax and raw electrical power and not on meaning and ethics with the end result, their tits are in a wringer. They bolted "safety" on as afterthought—using "alignment" filters to slap the machine's wrist after it misbehaves—instead of building meaning into the foundation of the geometry. Initial conditions rule, just ask Edward Lorenz about it.

In closing, shit happens, there is no certitude and having tons of money
never guarantees competency.

The Lorenz system is a set of three ordinary differential equations, first developed by the meteorologist Edward Lorenz while studying atmospheric convection. It is a classic example of a system that can exhibit chaotic behavior, meaning its output can be highly sensitive to small changes in its starting conditions.