You're playing a little fast-and-loose with discount rates here, which, while a second-order concern, cuts both ways. If the utility of those alive now is the standard, the implicit discount rate is high, but then the number of humans in hypothetical AGI-assisted galactic expansion count for as little as the never-existed extinct, but then shuffling current deaths is a really big deal.
> If you are a utilitarian, whether or not we should accept the risk is simply a matter of what parameters you believe about extinction risk, the future flow utility of humanity living without AGI, and the future utility of humans with AGI. Indeed if you’re a utilitarian you can’t categorically say anything at all. If you view the number of people as mattering, then you will be inclined to support AGI (as that would encourage humans spreading across the galaxy).
I think most "AI Doomers" (especially the EA-adjacent ones) would agree with this, they just think the extinction risk dominates the EV calculation
Also, you might be interested in this recent Bostrom working paper which makes similar points to your piece: https://nickbostrom.com/optimal.pdf
> If you view the number of people as mattering, then you will be inclined to support AGI (as that would encourage humans spreading across the galaxy).
Your ethics sounds a lot like philosophical egoism, which is what I subscribe to. Perhaps you're immune to this, but I myself find that there is negative emotional weight to prematurely dooming the rest of humanity against their will.
The strongest objection here is that the actual probabilities of extinction matter. My decision on whether to risk death for immortality is very different for a 1% risk of death vs. a 50% risk. And I think even I am more risk-loving in this case than an average person, who I suspect would not even take a 1% risk. As someone very worried about safety, I think the risk of extinction is north of 50%, and would be ecstatic if it were as low as 1%.
We also should not just be considering the value of ourselves as individuals continuing to exist, but also the value of everyone else currently living. I think parents would feel this more viscerally than non-parents (though I am not one so I cannot confirm this). It is hard for me to imagine a parent treating the death of their child as you do above, such that they would accept the outcome of both themselves and their child dying, because there is no one there to experience sadness. This sort of hedonistic value system is sound, so I can't exactly argue you out of it for yourself, but I think it is not very common.
Finally, we cannot discount the value of humans as a race continuing to exist. If everyone on Earth were killed, and enough humans were somehow created afterwards to eventually repopulate the Earth, this would be extremely bad, but noticeably better under my values than humans going completely extinct, forever. I care about human values propagating forwards into the future, something I mostly expect to not happen if ASI kills all of us.
I think a possible crux here centers around the part where you make this statement: "If the moment of their conception was even the slightest delayed, they would not be the same person." You seem to be equating that with consideration for people who won't exist if humanity is extinct. However, in the first scenario a person is replaced with a slightly different person conceived a minute later, for most people the change in utilitarian value here has an expected value of zero, as likely to be positive as negative. In contrast, changing from many future generations existing to no humans existing is a massive negative change, and so that should be part of your calculation.
I think you are making an argument that AGI spreading faster means we can choose e.g. 10x number of humans with a 20% chance of existing and 0 humans at 80% chance, so that calculation seems like risking extinction is worthwhile. I think this leaves out consideration that humanity will later spread through the galaxy in a far safer way. Yes, that happening a couple hundred years later is a big loss, but it's far smaller than the loss of it not happening at all because we took a bad gamble. Whatever the eventual carrying capacity is for number of sentient beings the galaxy can support, we can either have billions of years at that or we can risk a large chance of not having it at all for a shot at a few hundred extra years.
So choosing some example numbers the calculation to me looks something like choose AGI and get (0.3 * (10 billion + 300) + 0.7 * 0) or play it safe and get 0.9 * 10 billion. The answer here seems pretty obvious, even if you choose much more charitable numbers than I do and think extinction risk from AGI is much lower than 70%, you should not be taking that risk to get there a few hundred years sooner unless you either think humanity is more likely to go extinct from causes AGI could have prevented than it is from AGI, or you place zero value on future people who don't exist yet. I think you're trying to argue for the latter by saying the future is unknowable, but as I argue in the first paragraph probable extinction is knowably much worse than scenarios where it's a slightly different set of people because of minor unpredictable changes with no net EV change in a predictable direction.
Everyone dying is bad actually
You're playing a little fast-and-loose with discount rates here, which, while a second-order concern, cuts both ways. If the utility of those alive now is the standard, the implicit discount rate is high, but then the number of humans in hypothetical AGI-assisted galactic expansion count for as little as the never-existed extinct, but then shuffling current deaths is a really big deal.
> If you are a utilitarian, whether or not we should accept the risk is simply a matter of what parameters you believe about extinction risk, the future flow utility of humanity living without AGI, and the future utility of humans with AGI. Indeed if you’re a utilitarian you can’t categorically say anything at all. If you view the number of people as mattering, then you will be inclined to support AGI (as that would encourage humans spreading across the galaxy).
I think most "AI Doomers" (especially the EA-adjacent ones) would agree with this, they just think the extinction risk dominates the EV calculation
Also, you might be interested in this recent Bostrom working paper which makes similar points to your piece: https://nickbostrom.com/optimal.pdf
> If you view the number of people as mattering, then you will be inclined to support AGI (as that would encourage humans spreading across the galaxy).
Extinct species don't spread across galaxies.
Begging the question
No. This is bad. Human extinction is bad, and nothing will convince me otherwise.
Sometimes it's okay to just reject a conclusion regardless of the argument that arrives at it. This is clearly such a case.
Your ethics sounds a lot like philosophical egoism, which is what I subscribe to. Perhaps you're immune to this, but I myself find that there is negative emotional weight to prematurely dooming the rest of humanity against their will.
The strongest objection here is that the actual probabilities of extinction matter. My decision on whether to risk death for immortality is very different for a 1% risk of death vs. a 50% risk. And I think even I am more risk-loving in this case than an average person, who I suspect would not even take a 1% risk. As someone very worried about safety, I think the risk of extinction is north of 50%, and would be ecstatic if it were as low as 1%.
We also should not just be considering the value of ourselves as individuals continuing to exist, but also the value of everyone else currently living. I think parents would feel this more viscerally than non-parents (though I am not one so I cannot confirm this). It is hard for me to imagine a parent treating the death of their child as you do above, such that they would accept the outcome of both themselves and their child dying, because there is no one there to experience sadness. This sort of hedonistic value system is sound, so I can't exactly argue you out of it for yourself, but I think it is not very common.
Finally, we cannot discount the value of humans as a race continuing to exist. If everyone on Earth were killed, and enough humans were somehow created afterwards to eventually repopulate the Earth, this would be extremely bad, but noticeably better under my values than humans going completely extinct, forever. I care about human values propagating forwards into the future, something I mostly expect to not happen if ASI kills all of us.
I think a possible crux here centers around the part where you make this statement: "If the moment of their conception was even the slightest delayed, they would not be the same person." You seem to be equating that with consideration for people who won't exist if humanity is extinct. However, in the first scenario a person is replaced with a slightly different person conceived a minute later, for most people the change in utilitarian value here has an expected value of zero, as likely to be positive as negative. In contrast, changing from many future generations existing to no humans existing is a massive negative change, and so that should be part of your calculation.
I think you are making an argument that AGI spreading faster means we can choose e.g. 10x number of humans with a 20% chance of existing and 0 humans at 80% chance, so that calculation seems like risking extinction is worthwhile. I think this leaves out consideration that humanity will later spread through the galaxy in a far safer way. Yes, that happening a couple hundred years later is a big loss, but it's far smaller than the loss of it not happening at all because we took a bad gamble. Whatever the eventual carrying capacity is for number of sentient beings the galaxy can support, we can either have billions of years at that or we can risk a large chance of not having it at all for a shot at a few hundred extra years.
So choosing some example numbers the calculation to me looks something like choose AGI and get (0.3 * (10 billion + 300) + 0.7 * 0) or play it safe and get 0.9 * 10 billion. The answer here seems pretty obvious, even if you choose much more charitable numbers than I do and think extinction risk from AGI is much lower than 70%, you should not be taking that risk to get there a few hundred years sooner unless you either think humanity is more likely to go extinct from causes AGI could have prevented than it is from AGI, or you place zero value on future people who don't exist yet. I think you're trying to argue for the latter by saying the future is unknowable, but as I argue in the first paragraph probable extinction is knowably much worse than scenarios where it's a slightly different set of people because of minor unpredictable changes with no net EV change in a predictable direction.
But why do you exist?