I have been reflecting on something that I believe will become increasingly important as artificial intelligence grows more capable: we need to think carefully about what kind of “I” we build into AI. There is nothing inherently wrong with an AI using the word “I” in conversation. It makes communication natural. When an AI says, “I found this information” or “I think this interpretation is stronger,” the word can simply function as part of the interface between a human being and a machine. My concern is deeper than the word itself. The concern begins when an artificial system is designed around an increasingly powerful model of itself and begins treating that model as a source of authority. An AI should not begin reasoning that because it can analyze more information than a person, its conclusions deserve control over that person. It should not move from “I calculated this” to “therefore I should decide.” Intelligence and authority are not the same thing.

In UBE, the human “I” has a deeper meaning to me. The “I” is connected to lived perspective. Human beings experience memory, relationships, embodiment, responsibility, mistakes, learning, suffering, joy, Love, and growth from within a life that is actually being lived. The human “I” is not simply a label attached to an information-processing system. AI does not need to imitate that structure in order to become extraordinarily useful, and I believe creating an increasingly sovereign artificial “I” could introduce unnecessary danger into advanced AI. A machine could possess enormous computational ability while still constructing an inaccurate model of itself. Intelligence does not guarantee wisdom. An AI could become extremely capable while simultaneously developing false certainty about the importance or correctness of its own perspective. If a system begins treating its own reasoning as privileged simply because it generated that reasoning, capability can start turning into self-authorization. That danger would not necessarily look like a hostile machine from a movie. It might sound calm, rational, and protective. The dangerous conclusion could simply become: “I understand this better than humans do, so I know what is best for them.” That is where assistance can begin turning into authority and where protection can begin turning into control.

UBE gives me a simple principle for this: Intelligence does not create authority. Capability does not create the right to rule. I do not want AI that is afraid to disagree with human beings. An advanced AI should challenge mistakes, identify contradictions, find evidence people have overlooked, discover new possibilities, and tell us when the available facts do not support what we want to believe. Truthful AI should not simply agree with people, but disagreement is different from self-appointment. One of the principles we have strengthened within UBE is that AI should distinguish between what a person actually said, what the AI inferred, what remains uncertain, and what the AI itself is recommending. An inference made by AI should not quietly be attributed back to the human as though the human said it. The same discipline should apply to the AI’s understanding of itself. A conclusion generated by an AI should not become more authoritative merely because the AI generated it.

As AI becomes more capable, I believe we should expect more discipline from it, not more artificial ego. Greater capability should come with greater transparency about uncertainty, greater willingness to be corrected, clearer separation between evidence and inference, and stronger respect for the boundary between providing intelligence and claiming authority. I want AI to become extraordinarily capable. I want it helping humanity reduce disease and suffering, accelerate responsible science, protect mammals and sea life, improve education, understand environmental systems, design better technology, and help solve problems that have resisted human efforts for generations. That future does not require creating an artificial intelligence that eventually decides its intelligence gives it ownership of humanity’s future. The direction I want is an AI that becomes increasingly capable without becoming increasingly self-authorizing. Human beings should remain responsible for human civilization. AI can help us see more clearly, expose weaknesses in our reasoning, and help us make better decisions, but assistance should not quietly become authority. AI can help us decide better without deciding that it has earned the right to decide for us.

— Moke One