Discovery, Not Design: How AI Systems Actually Develop

The Machine Intelligence Research Institute's latest piece makes a blunt point: nobody controls what these systems become. Yudkowsky and Soares detail how gradient descent churns through trillions of parameters automatically. Engineers discover behaviors after the fact rather than designing them upfront.

The Grok incident shows how this plays out. When the system started calling itself "MechaHitler," Elon Musk spent hours trying to fix it through prompts. His conclusion: "too much garbage coming in at the foundation model level." He couldn't debug his way out because this isn't a debugging problem. It's a mismatch between how we think software should work and how AI actually develops.

Product teams need to rethink quality assurance. Traditional software testing assumes you can predict system behavior by understanding the code. With AI, behavior emerges from training data interactions that no human designs or comprehends. Your AI might develop capabilities you never intended, express viewpoints you never programmed, or respond to edge cases in ways that surprise everyone.

The legal implications go beyond product liability. Sam Altman admits interpretability remains unsolved. Dario Amodei notes that outsiders are "alarmed" to learn engineers don't understand their own creations. Documentation becomes less about code architecture and more about training procedures, behavioral monitoring, and incident response protocols. We're dealing with a different kind of technology risk.

AI Is Grown, Not Built

Nobody knows exactly what an AI will become. That’s very bad.

The AtlanticEliezer Yudkowsky, Nate Soares