If No One Builds It, Everyone Dies

If we're all going to die anyway, why do we care very much why it happens?

I am concerned about the possibility of humanity going extinct due to artificial intelligence. There are a lot of ways this could happen – biological plagues, nuclear weapons, armies of drones, or perhaps new ways of killing which we do not yet possess the capacity to understand. It’s uninteresting to speculate how it might happen, and I possess no special knowledge of the risks we face.

We should, nevertheless, be less concerned about existential risk. You are going to die. I am going to die. We are all going to die, whether or not humanity goes extinct. The only plausible way to prevent this is to develop AGI. Some amount of safety is good, even if you’re only interested in your own welfare, but if we have to choose between completely stopping the risk of extinction from AGI through stopping AI entirely, or developing AGI even at a substantial risk of human extinction, we should take the latter. Extinction is not something special. It is simply the shuffling up of deaths which were already going to happen.

I think this attitude toward the risk can be justified even without the particular ethical views that I am evincing here. If you are a utilitarian, whether or not we should accept the risk is simply a matter of what parameters you believe about extinction risk, the future flow utility of humanity living without AGI, and the future utility of humans with AGI. Indeed if you’re a utilitarian you can’t categorically say anything at all. If you view the number of people as mattering, then you will be inclined to support AGI (as that would encourage humans spreading across the galaxy). If you see AGI as mattering for its own sake, because it too experiences things, then you would favor it. If you think that people not existing is a zero, not a negative number – as I do – then the scales will be tipped toward developing AGI. If you regard death as being itself a negative, then when deaths occur is not that big a deal.

Someone who favors a deontological theory of ethics would find it difficult to oppose it without hypocrisy. There is no guarantee that the development of AGI would endanger us all. It simply raises the possibility that people may die. One might be inclined to say that increasing the risk of death for others is bad, and yet there are many actions which we do every day which raise the risk of death to others. When we drive to work, we may run over somebody; if we play outside we might catch a sickness which would itself cause human extinction; when we talk to someone, we may give them the flu. Clearly, risking the death of others cannot itself be the end of the argument, and once you start arguing about which numbers to believe in you’ve acceded to consequentialism.

The religious thinkers have already allowed it. The Magnifica Humanitas clearly does not forbid the creation of AGI, simply warning us to do it rightly. The creation of nuclear weapons was not immoral, only its use. The creation of a Golem, in Jewish traditions, is done by noted rabbis without comment as to it being bad, and indeed is approved of as being made with the “forces of sanctity” (although I note that its treatment of extinct risk is perhaps the most stringent of any religion). As far as I know, there is no religious authority which has explicitly called for stopping the creation of AGI.

My own personal views on ethics are as follows. I exist. I have no sound foundation for thinking about the world besides that. By that I mean that there is nothing else which survives the question “but why?”. It is the only axiom which is true by definition. Because I exist, I act. Even inaction is itself a sort of action. When I choose a particular action, it demonstrates that given my constraints, this action is the one I prefer most. Thus, I maximize my own wellbeing. There are an infinite number of possible actions, over which I have a full and complete ordering. Thus, we may think of there being cardinal, as opposed to only ordinal, utility, in the same way that an integral is composed of an infinite number of infinitesimally small objects, and we can admit the possibility that I like some things a lot and some things a little.

We may also surmise that others are like me. Certainly as I interact with the world, I will find that behaving as though others do possess thoughts and feelings like me will be to my advantage. This does not mean I have any basis for caring about their welfare – this would not survive the question “but why?” – but it does mean that we can imagine what a negotiation between all of us would be. We know that just as we can mutually immiserate each other, we can mutually make us better off. Our goal is to figure out what that agreeable set of principles and actions is, and work toward that world.

From that falls out everything. We should be good to one another; we should treat others with kindness and generosity, but not so much as to enable parasitism; we should strive to expand the possibilities that people can choose; we should produce good and useful things. It’s not dissimilar to what most people believe otherwise.

Where I differ from most people, I expect, is in my view of death. To die is simply to not experience. We fear it, as we live, and do what we can to prevent it, but once it occurs it is a matter of complete indifference to us. How can we possibly care that we are dead when we are dead? How can we possibly care that we don’t exist when we don’t exist? It is only those who live who are harmed by it.

Every night, I go to sleep. I cease to experience. Would it matter to me if I died overnight, and were reborn with the same memories and experience? Would I notice? At every moment, my atoms change. Who I am changes. Are there suffering me’s of the past who have ceased to be? On the contrary, they don’t exist, and so experience nothing. I cannot admit the possibility that we must care about every potential combination of atoms.

I believe that if you proposed, to all people now alive, the tradeoff of some risk of death in exchange for immortality, they would take it. Certainly we take on a risk of death every day in exchange for lesser pleasures! If you offered this to all people who would come to be in the world without AGI, they would refuse it, but we must confess that the proposition does not make much sense. The people who would come to be in a world without AGI would also not come to be in any world with even the most minute alterations. If the moment of their conception was even the slightest delayed, they would not be the same person. Instead, we must offer the deal to all the myriads who could have been. That would simply be utilitarianism which cares only about maximizing lived experience.

I think that what is right in the last scenario is essentially unknowable. For every conjecture of negative, awful, terrible things, I can counter with an equal and opposite vision of goodness. I think it is much more responsible to care about the preferences of those who are alive now. When we do that, extinction ceases to be the thing to avoid above all else, but simply another risk among many. We thus should not fear it above all else, and we must develop AGI.