When Jacob Coxon quit Anthropic, his warning rocked the world. His colleague Evan’s “>10% chance” that AI would “kill all humans” might seem extreme, but it’s not unusual:
Most AI researchers think there’s a 10% chance of AI destroying or disempowering humanity, according to our survey, released today. https://x.com/AIImpacts/status/2099581080223576403
I’m an independent researcher who runs the largest and longest-running peer-reviewed survey of AI researchers. These researchers are experts who publish papers at the top machine learning and artificial intelligence venues.
We’ve been conducting these surveys since 2016, so we can see how expert opinion has moved over the years.
Two trends: 1) Researchers' 'timelines' to human-level AI have shrunk, fast. 2) Most researchers have seen major future risks of extinction-level catastrophes for a long time, and their concerns are lately trending upward
This loosely mirrors what I hear from talking to people in the field: 1) AI is advancing much faster than researchers expected 2) Catastrophic risks have been looming for a while, and people are increasingly worried
Also interesting: many people worry that China will race ahead in AI if the US slows down to avoid human extinction. (It’s unclear to me why China would not also want to slow down in this account).
But Asian researchers in our survey were actually more worried! https://x.com/AIImpacts/status/2099581081901211878
e.g. last time I recall playing One Night Ultimate Werewolf I just admitted I was a werewolf because I was lying so ineffectually and flusteredly. AI is already much better at lying than me, but this actually gives me an advantage in being trustworthy.
I’ve really wanted more good short accounts of why AI poses an existential risk. Working on one myself has been one of those incredibly high priorities I keep putting off. Meanwhile Ben Bradford of NPR, has just made a podcast version of my case for AI x-risk that I am thrilled with!
Does anyone have a really satisfying way to store clothes? An account of my present objections to approximately all of the ones I know of, which I'm not recommending you read unless you are curious or want a break from talking about AI: worldspiritsockpuppet.substack…
I hate the way ads add a bunch of fake commercially incentivized voices to the conversation. AI is taking it further, which I dislike for these reasons and more!
Many humans being powerless is a familiar situation, but we think like there's some law against every human being powerless. That changes if you introduce a new species with superior performance at everything.
If I lie down and close my eyes, I often get right to sleep. However, doing so sounds dreadful: I want to live! There are a handful of kinds of living that actually make me sleepy though, and one of them is mathy puzzles you can do in your head. Anyone have good ones?
Katja Grace
in reply to Katja Grace • •Katja Grace
in reply to Katja Grace • •We’ve been conducting these surveys since 2016, so we can see how expert opinion has moved over the years.
Two trends:
1) Researchers' 'timelines' to human-level AI have shrunk, fast.
2) Most researchers have seen major future risks of extinction-level catastrophes for a long time, and their concerns are lately trending upward
Katja Grace
in reply to Katja Grace • •1) AI is advancing much faster than researchers expected
2) Catastrophic risks have been looming for a while, and people are increasingly worried
Katja Grace
in reply to Katja Grace • •Katja Grace
in reply to Katja Grace • •Also interesting: many people worry that China will race ahead in AI if the US slows down to avoid human extinction. (It’s unclear to me why China would not also want to slow down in this account).
But Asian researchers in our survey were actually more worried!
https://x.com/AIImpacts/status/2099581081901211878
Katja Grace
in reply to Katja Grace • •https://x.com/AIImpacts/status/2099581078210236850
Katja Grace
in reply to Katja Grace • •