Interesting post, I generally agree with the flaws you pointed out, especially in the begging (I don't belive we're close to independent agents). However some of your takes are very unconvincing:
1) You claim claim openBrain is just straightforwardly OpenAI. Then when something in the scenario doesn't align with that reading, instead of admitting it weakens your argument you treat it as a flaw in the scenario: eg "but OpenAI didn't invent constitutional AI!". This comes off as cartoonishly bad-faith.
2) "It isn’t really meant to be read, or taken seriously, by anyone who isn’t already a believer of some kind. It is fundamentally an internal dispute that can safely be made public because very few people will actually read it."
What? How does this square with you saying this is intended to be an investment pitch or to sway bureaucrats? Or the fact that the authors did a publicity tour for it and constantly said it's intended to influence public opinion.
re: 1), OpenBrain is so clearly based on OpenAI from the name alone that I think that's straightforward! Erasing competitors from the story for simplicity sort of makes sense, and maybe describing the accomplishments of competitors as the accomplishments of OpenAI makes sense to keep the story going, but this "just happens" to function as attributing something a competitor accomplished to OpenAI.
2) I think you're right and this thought could be better-developed.
In fact, I think I agree enough to take that second passage out, because it's the fastest way to fix it and I don't think the point it is making is very important. Thank you.
As someone who doesn’t work in tech, and doesn’t even know that many people who do, I’ve been fascinated for a while now by the rhetoric about AI making every job obsolete. It’s like they know that there are jobs that aren’t “in tech,” but they only have a vague idea of what they entail, but they’re *also* completely confident that the AI they’re developing will be able to, at some point ~soon, do all of those jobs better than any human. Every single passage after the part where the AI somehow took over the entire white collar job market where they were just acting like they hadn’t said that happened was insane, but I’m not at all surprised they didn’t bother thinking it through further than that at all. They never have any of the other times they’ve claimed they were going to do that!
It's been a global problem for a long time that you'd think people would have noticed. Every time someone tries to make software (including the AI kind) for some specific profession or task outside tech it turns out that profession or task is pretty complicated, and making stuff to do it is really hard.
Nobody's going to top the venture capital guy (ie, he gives startups money for a living) who thinks that venture capital is the only profession that can't be automated, though. His job and his job alone.
Great article! I do think AI 2027 has a lot of "and this tech just magically gets 10x better" but it's still a fun read, you just have to somewhat suspend your disbelief. The part where the president just liquidates all the other AI companies to empower Open"brain" is really funny though. If the AI can negotiate peace between 2 superpowers, surely it can negotiate Google etc. being acquired, no? That part feels really bad faith/investor bait.
AI will probably end up being a pretty nice tool but nothing too revolutionary. Some jobs are gonna get lost but more will get created. IF AI does end up being as revolutionary as what is said, then it inherently becomes impossible to predict. Just like the singularity.
I think it’s a very important note that one of the writer behind AI 2027 released an equally ridiculous sounding post in 2021 predicting AI progress in 2025, and they absolutely nailed it. Also, the former OpenAI employee lost all of his stock in OpenAI by breaking the noncompete.
Interesting and entertaining. One little addendum I kind of want to add is that ideas like "good code" or "the best coders" are just as resistant to quantification as ideas like "research taste". Highly-experienced professional programmers, well-informed by context, will still disagree vociferously about what constitutes good code or a reasonable approach to a coding problem in a given real-world situation—perusing the dev mailing list of a popular programming language, OS, library, etc. will provide many examples of this. I think it's a completely open question, never answered to any reasonable level of satisfaction, what "superhumanly good code" would even look like, on that front.
One thing I think it is fair to say, though, in a qualitative sense, is that LLMs do /not/ write code anywhere near the level of an expert human who knows the problem domain and specific codebase well—nowhere close, at least in my experience. I think in that context they're about as useful as a search engine; sometimes a little more, sometimes a little less. What would it take for them to do better? Some things that would help are a genuine abstract understanding of the theory of the problem domain, a "good feeling" for the design of the surrounding codebase, and a sense of what makes for a nice API. How could they acquire things like this? Who knows—those are pretty subjective criteria to measure! It's pretty hard to get away from qualitative things like this that have no obvious way of being boiled down to something you can put hard numbers on easily.
Okay, so I have to admit that I posted that comment before I had finished reading the entire thing. I was like 2/3rds of the way done. Now that I'm all the way finished, I have a couple more comments to add.
One is that intelligence is honestly just again as squishy and qualitative. IQ is not really a good proxy for it at all—many people with high IQs act in ways or profess beliefs that are crankish and don't show "good sense" in the eyes of most other people, nor do they ever necessarily accomplish anything very impressive, and many people who are often hailed as geniuses in the popular discourse don't necessarily have exceptional IQs. People mean a huge variety of different things when they call someone smart or a genius.
For example, I understand what "everyday people" are getting at when they call contemporary LLMs "dumb as a rock" when they fail comically at tasks like "show me a map of the U.S. with all the states that have the letter 'n' in their names highlighted" or things like that, despite the fact that they sometimes give impressively coherent answers to certain queries. It gives the impression that, in the end, "the LLMs understand nothing," even if they've gotten better at conveying the impression of understanding in certain other scenarios over time. To LLM developers, they may say that their model has gotten "demonstrably smarter" over time, but they're clearly using a very different metric to come to that sort of conclusion, and there's not really an unambigious, true-in-all-cases way of saying whose approach is more legit.
On this basis, I don't think it's clear at all that a "supersmart AI" is even possible. Without further qualification, that's a meaninglessly vague phrase—it lets you wave your hands around and say things like "it knows as much as every human combined" or "it can solve any problem put to it at superhuman speeds," but that glosses over how little it's even been established what it means to "know" something in this context or what would really qualify as a satisfactory "solution" to any given problem. Like, Google Search can spit out a far-vaster amount of information than anyone on Earth can easily recall off the top of their heads, but it clearly doesn't /know/ that information the way a person is capable of knowing it, and I have yet to be convinced that LLMs know it on any particularly deeper level.
As far as evaluating solutions goes, one thing that I thought was funny about the bioweapon design part was like, we already /have/ bioweapons capable of causing apocalyptic pandemics if they were deliberately deployed to do so with enough resources and verve. The thing that keeps that from happening is sociocultural; almost everyone alive doesn't want that outcome, so we all work together to ensure that it doesn't happen. Lo and behold, it doesn't. The obvious danger there is a huge number of people becoming apathetic on this point or vengeful towards humanity or something, not so much a mythical superintelligence designing a better bioweapon. I don't really think we're in massive danger of bioweapon-pandemic-by-apathy (although the current sociopolitical atmosphere in the U.S. does kind of worry me a little about that I guess), but in any case, if it did seem like a serious risk, I don't think it's the kind of risk you could easily R&D your way out of—it's politics, not math. The sorts of people who write these kinds of documents obviously have some ideas about how to manipulate politicians, but I don't know that they have the greatest grasp on "little p" politics.
I agree with every part of this. I basically take for granted that somehow "the best coder" is a meaningful thing -- after all, at least we aren't asked to rely directly on a point scale, right? -- and that if you are, in fact, the best and there are 30,000 of you that is a meaningful thing to have happen. I do think the underlying data/assumption is that LLM benchmarks for coding quantify "good-at-coding-ness", when in reality the job of a SWE is to translate ideas into a thing you can use to accomplish some task, which is a pretty squishy thing to do. But for the sake of interrogating the argument, sure, fine, it's "the best coder".
Basically I agree that people in the field should really be interrogating what they mean by "intelligence" way, way more. I think in narrow ways they actually succeed at that! Inasmuch as "intelligence" is like, trivial pursuit, we do actually have benchmarks for those. But "intelligence is the thing that humans have" is what we mean, and we can't actually quantify most of it easily.
re: politics, I think at least 2-3 of the authors of this piece achieved the exact opposite of their goal. That is: They claim and I take them at their word that they definitely intended to warn primarily about how dangerous AI is. The primary received message by stakeholders appears to be to lean into an arms race as hard as possible because China is very scary. Phenomenal own-goal, an AI policy classic, given that modern AI exists largely because a large group of people evangelized that it should not.
This article had a lot more thought put into it than the entire AI 2027 website. There are so many moments in their writing where they just read everything in the most charitable way possible and then make assumptions that things are going to magically get better from here. Meanwhile, the reality of these things is much more down to earth. The release of gpt5 has been a real big mess for example.
Personally, I just don't buy the main premise that it even is possible to use the current models of machine learning to create something as nuanced as the human brain. There are a number of fundamental issues regarding the hardware, the software, and the architecture. These are probability pattern matching machines, they don't have reasoning or understanding.
this triple posted so i'm taking the liberty of removing dupes
I think it was credible to theorize after the gpt-4 release that you could simply continue to scale, or that something within easy striking distance of current LLMs could hit recursive self-improvement. This is mostly because GPT-4 working as well as it did was so surprising, and we did not know what the space around it would look like.
It's been long enough that I think we can call that this is not the case.
You don't discuss AI 2027's rationale for their "only one AI company" -- which makes sense because the site doesn't really have one.
But their whole scenario seems to fall apart if indeed there are many competing AI vendors of roughly comparable quality. That indeed is more and more the case.
So my question: Do you know of any deep dive into their claim / assumption that there will only be one? And/or any analysis of how the outcomes change if there are many?
I don't think there's any reason for assuming they're the only (important) AI vendor. They are just assuming they are much better than everyone else.
At the end of the story as the AI becomes a major national security thing, they assume that the government will shut down their competitors by fiat and force them to sell their assets, though. That could happen. Otherwise they will continue to have serious competition.
I dont you think you give the author enough credit given that his quiet successfull predictions from 2021 on LW. OAIs revenue growth is also a good putch to investors (and might justify the losses). Its just a specific story as people found AI risks too abstract beforehand, the details are unclear.
Interesting post, I generally agree with the flaws you pointed out, especially in the begging (I don't belive we're close to independent agents). However some of your takes are very unconvincing:
1) You claim claim openBrain is just straightforwardly OpenAI. Then when something in the scenario doesn't align with that reading, instead of admitting it weakens your argument you treat it as a flaw in the scenario: eg "but OpenAI didn't invent constitutional AI!". This comes off as cartoonishly bad-faith.
2) "It isn’t really meant to be read, or taken seriously, by anyone who isn’t already a believer of some kind. It is fundamentally an internal dispute that can safely be made public because very few people will actually read it."
What? How does this square with you saying this is intended to be an investment pitch or to sway bureaucrats? Or the fact that the authors did a publicity tour for it and constantly said it's intended to influence public opinion.
re: 1), OpenBrain is so clearly based on OpenAI from the name alone that I think that's straightforward! Erasing competitors from the story for simplicity sort of makes sense, and maybe describing the accomplishments of competitors as the accomplishments of OpenAI makes sense to keep the story going, but this "just happens" to function as attributing something a competitor accomplished to OpenAI.
2) I think you're right and this thought could be better-developed.
In fact, I think I agree enough to take that second passage out, because it's the fastest way to fix it and I don't think the point it is making is very important. Thank you.
OpenBrain is most likely a portmanteau of OpenAI and Google Brain, and DeepCent a portmanteau of DeepSeek and Tencent.
clever
As someone who doesn’t work in tech, and doesn’t even know that many people who do, I’ve been fascinated for a while now by the rhetoric about AI making every job obsolete. It’s like they know that there are jobs that aren’t “in tech,” but they only have a vague idea of what they entail, but they’re *also* completely confident that the AI they’re developing will be able to, at some point ~soon, do all of those jobs better than any human. Every single passage after the part where the AI somehow took over the entire white collar job market where they were just acting like they hadn’t said that happened was insane, but I’m not at all surprised they didn’t bother thinking it through further than that at all. They never have any of the other times they’ve claimed they were going to do that!
It's been a global problem for a long time that you'd think people would have noticed. Every time someone tries to make software (including the AI kind) for some specific profession or task outside tech it turns out that profession or task is pretty complicated, and making stuff to do it is really hard.
Nobody's going to top the venture capital guy (ie, he gives startups money for a living) who thinks that venture capital is the only profession that can't be automated, though. His job and his job alone.
Great article! I do think AI 2027 has a lot of "and this tech just magically gets 10x better" but it's still a fun read, you just have to somewhat suspend your disbelief. The part where the president just liquidates all the other AI companies to empower Open"brain" is really funny though. If the AI can negotiate peace between 2 superpowers, surely it can negotiate Google etc. being acquired, no? That part feels really bad faith/investor bait.
AI will probably end up being a pretty nice tool but nothing too revolutionary. Some jobs are gonna get lost but more will get created. IF AI does end up being as revolutionary as what is said, then it inherently becomes impossible to predict. Just like the singularity.
I think it’s a very important note that one of the writer behind AI 2027 released an equally ridiculous sounding post in 2021 predicting AI progress in 2025, and they absolutely nailed it. Also, the former OpenAI employee lost all of his stock in OpenAI by breaking the noncompete.
Daniel sued over the non-compete and last I knew still had his stock, the 2021 prediction is also him and it's here: https://www.alignmentforum.org/posts/6Xgy6CAf2jqHhynHL/what-2026-looks-like
I think that prediction is pretty good, and expect AI 2027 to age much less well.
Interesting and entertaining. One little addendum I kind of want to add is that ideas like "good code" or "the best coders" are just as resistant to quantification as ideas like "research taste". Highly-experienced professional programmers, well-informed by context, will still disagree vociferously about what constitutes good code or a reasonable approach to a coding problem in a given real-world situation—perusing the dev mailing list of a popular programming language, OS, library, etc. will provide many examples of this. I think it's a completely open question, never answered to any reasonable level of satisfaction, what "superhumanly good code" would even look like, on that front.
One thing I think it is fair to say, though, in a qualitative sense, is that LLMs do /not/ write code anywhere near the level of an expert human who knows the problem domain and specific codebase well—nowhere close, at least in my experience. I think in that context they're about as useful as a search engine; sometimes a little more, sometimes a little less. What would it take for them to do better? Some things that would help are a genuine abstract understanding of the theory of the problem domain, a "good feeling" for the design of the surrounding codebase, and a sense of what makes for a nice API. How could they acquire things like this? Who knows—those are pretty subjective criteria to measure! It's pretty hard to get away from qualitative things like this that have no obvious way of being boiled down to something you can put hard numbers on easily.
Okay, so I have to admit that I posted that comment before I had finished reading the entire thing. I was like 2/3rds of the way done. Now that I'm all the way finished, I have a couple more comments to add.
One is that intelligence is honestly just again as squishy and qualitative. IQ is not really a good proxy for it at all—many people with high IQs act in ways or profess beliefs that are crankish and don't show "good sense" in the eyes of most other people, nor do they ever necessarily accomplish anything very impressive, and many people who are often hailed as geniuses in the popular discourse don't necessarily have exceptional IQs. People mean a huge variety of different things when they call someone smart or a genius.
For example, I understand what "everyday people" are getting at when they call contemporary LLMs "dumb as a rock" when they fail comically at tasks like "show me a map of the U.S. with all the states that have the letter 'n' in their names highlighted" or things like that, despite the fact that they sometimes give impressively coherent answers to certain queries. It gives the impression that, in the end, "the LLMs understand nothing," even if they've gotten better at conveying the impression of understanding in certain other scenarios over time. To LLM developers, they may say that their model has gotten "demonstrably smarter" over time, but they're clearly using a very different metric to come to that sort of conclusion, and there's not really an unambigious, true-in-all-cases way of saying whose approach is more legit.
On this basis, I don't think it's clear at all that a "supersmart AI" is even possible. Without further qualification, that's a meaninglessly vague phrase—it lets you wave your hands around and say things like "it knows as much as every human combined" or "it can solve any problem put to it at superhuman speeds," but that glosses over how little it's even been established what it means to "know" something in this context or what would really qualify as a satisfactory "solution" to any given problem. Like, Google Search can spit out a far-vaster amount of information than anyone on Earth can easily recall off the top of their heads, but it clearly doesn't /know/ that information the way a person is capable of knowing it, and I have yet to be convinced that LLMs know it on any particularly deeper level.
As far as evaluating solutions goes, one thing that I thought was funny about the bioweapon design part was like, we already /have/ bioweapons capable of causing apocalyptic pandemics if they were deliberately deployed to do so with enough resources and verve. The thing that keeps that from happening is sociocultural; almost everyone alive doesn't want that outcome, so we all work together to ensure that it doesn't happen. Lo and behold, it doesn't. The obvious danger there is a huge number of people becoming apathetic on this point or vengeful towards humanity or something, not so much a mythical superintelligence designing a better bioweapon. I don't really think we're in massive danger of bioweapon-pandemic-by-apathy (although the current sociopolitical atmosphere in the U.S. does kind of worry me a little about that I guess), but in any case, if it did seem like a serious risk, I don't think it's the kind of risk you could easily R&D your way out of—it's politics, not math. The sorts of people who write these kinds of documents obviously have some ideas about how to manipulate politicians, but I don't know that they have the greatest grasp on "little p" politics.
I agree with every part of this. I basically take for granted that somehow "the best coder" is a meaningful thing -- after all, at least we aren't asked to rely directly on a point scale, right? -- and that if you are, in fact, the best and there are 30,000 of you that is a meaningful thing to have happen. I do think the underlying data/assumption is that LLM benchmarks for coding quantify "good-at-coding-ness", when in reality the job of a SWE is to translate ideas into a thing you can use to accomplish some task, which is a pretty squishy thing to do. But for the sake of interrogating the argument, sure, fine, it's "the best coder".
Basically I agree that people in the field should really be interrogating what they mean by "intelligence" way, way more. I think in narrow ways they actually succeed at that! Inasmuch as "intelligence" is like, trivial pursuit, we do actually have benchmarks for those. But "intelligence is the thing that humans have" is what we mean, and we can't actually quantify most of it easily.
re: politics, I think at least 2-3 of the authors of this piece achieved the exact opposite of their goal. That is: They claim and I take them at their word that they definitely intended to warn primarily about how dangerous AI is. The primary received message by stakeholders appears to be to lean into an arms race as hard as possible because China is very scary. Phenomenal own-goal, an AI policy classic, given that modern AI exists largely because a large group of people evangelized that it should not.
This article had a lot more thought put into it than the entire AI 2027 website. There are so many moments in their writing where they just read everything in the most charitable way possible and then make assumptions that things are going to magically get better from here. Meanwhile, the reality of these things is much more down to earth. The release of gpt5 has been a real big mess for example.
Personally, I just don't buy the main premise that it even is possible to use the current models of machine learning to create something as nuanced as the human brain. There are a number of fundamental issues regarding the hardware, the software, and the architecture. These are probability pattern matching machines, they don't have reasoning or understanding.
this triple posted so i'm taking the liberty of removing dupes
I think it was credible to theorize after the gpt-4 release that you could simply continue to scale, or that something within easy striking distance of current LLMs could hit recursive self-improvement. This is mostly because GPT-4 working as well as it did was so surprising, and we did not know what the space around it would look like.
It's been long enough that I think we can call that this is not the case.
You don't discuss AI 2027's rationale for their "only one AI company" -- which makes sense because the site doesn't really have one.
But their whole scenario seems to fall apart if indeed there are many competing AI vendors of roughly comparable quality. That indeed is more and more the case.
So my question: Do you know of any deep dive into their claim / assumption that there will only be one? And/or any analysis of how the outcomes change if there are many?
I don't think there's any reason for assuming they're the only (important) AI vendor. They are just assuming they are much better than everyone else.
At the end of the story as the AI becomes a major national security thing, they assume that the government will shut down their competitors by fiat and force them to sell their assets, though. That could happen. Otherwise they will continue to have serious competition.
Excellent piece. Thank you for close reading of this marketing literature so we don't have to.
I dont you think you give the author enough credit given that his quiet successfull predictions from 2021 on LW. OAIs revenue growth is also a good putch to investors (and might justify the losses). Its just a specific story as people found AI risks too abstract beforehand, the details are unclear.