Linguistic signs (and other modes representation) are meant to *represent* something. LLMs are able to generate representations of other representations, including non-linguistic representations. But, because these second order representations are not embodied or directly tied to some ongoing embodiment, they don’t generate direct representations of reality. They replicate a lowest-common-denominator version of the rich cultural structures that have taken centuries or millennia to build up in already-existing modes of consciously articulated cultural representational systems. In order for your argument to hold up, all of reality (not just in human social contexts but the entirety of the universe) would have to consist in representations and nothing else. That’s a metaphysical claim that has serious defects, and this essay doesn’t actually offer a technical or philosophical defense of that claim.
Signs do represent. But the mistake in your comment is that signs don't inherently represent anything (Saussure, Pierce). They get their meaning by convention, by distributional semantics (Firth), by use (Wittgenstein), and by appearing within a discursive practice (Foucault)
There can be all sorts of second-order, third-order, etc. modes of signification happening within various cognitive behaviors. That is a core feature of cognition, particularly human cognition. But the larger system has to be grounded in some embodied process that has real contact with physical reality. Without that, cognition becomes a balloon slowly leaking air. If you want to look at linguistic signification as a cultural matrix, rather than through the lens of the individual’s direct cognition, semantic meaning develops out of rhythmic processes of verbal expression that begin with gestural rhythms, like meter and grouping, and take on addition structure through paralanguage and grammar (Cureton).
That's really interesting but I think it is worth flagging a bigger issue. If I want, I can define meaning to require a relationship to intention as formed by a biological brain. Under that definition an AI implemented on silicon will never mean anything by its assertions but obviously that doesn't entail anything of importance about how useful they are, how we should interact with them or even their moral status.
So yes, what you said all seems right and correct but we should also flag the fallacious attempt to use the term meaning in a way that invites misleading inferences.
Are you going to note that the core question anyone really cares about - whether this is an approximation of animal cognition - can still be confidently answered "no"?
That really depends on what you mean by "an approximation". Humans are animals and approximating language use is, in fact, approximating all the outward signs of some types of animal cognition. Some approximations are good, and some are worse, et cetera, et cetera.
If you mean "an approximation of animal cognition" as in "this thing could credibly impersonate a golden retriever" the answer is, of course, no. And beyond that, they can only credibly do human-like things in a relatively narrow range of situations. But "not an approximation of animal cognition" seems very strong, given that it is literally a mathematical approximation of human language behavior.
Absolutely. But my aphantasia gives me a relationship to language that is closer in a way to LLMs than most people — I understand how language represents, or perhaps more accurately evokes and creates internally, real life, in the absence of my ability to form images. Comparing AI generated language to AI generated images is helpful here.
I think framing it in terms of animal cognition is a little like asking if submarines swim. What matters is if these things have something that could be called intelligence, not if the underlying circuitry is similar to a human’s. The stochastic parrot framing implies that they don’t, and I think this article nicely illustrates why that stance no longer has a leg to stand on.
Thank you! I have been promoting the idea that "the stochastic parrot model is NOT the only model for how AI interacts!" for TWO YEARS, and have been sharing this paper by Dr. David Chalmers as an alternative argument:
Bravo! Bender has become a useful idiot, or just a plain idiot, for that part of the Left that hates AI and has never used an LLM. I ignore her because she is so obviously wrong, and I’d rather use the models to get things done than to take time refuting the arguments of imbeciles like Bender. I’m glad you took the time, though I doubt it will change many minds, since people like Bender and those who admire her hate AI as an article of faith and not as the result of reasons and evidence. I view her, and people like Marcus, as people who built a brand on being anti-AI and who will stick with that no matter what. So I’m happy to leave them in the dustbin of history,
My experience has been that there are people who are, in principle, reachable, and also people who sort of think she might have a point because this isn't *really* their area, and they're mostly who I hope gets something from all this.
It is evident that the disagreement centers around what we mean by “meaning.”
I understand that in a polemical essay, which this is, treating the opposing argument in almost hyperbolic terms is acceptable. And I am fine with that.
However, the authors of the stochastic parrots paper are working within a well established and “active” area of philosophy with the definition of meaning that they use. The treatment here is not a particularly fair presentation of their definition of meaning, but is a pure takedown.
What’s worse, the author only cites the existence of other theories of meaning without taking a position on why those are better than the one he condemns. We are to take the mere existence of distributional, functional, and pragmatic views of meaning as evidence that the authors are wrong in theirs without any further justification.
I’m interested and open to the line of thought represented here but not at all persuaded by the arguments presented.
The stochastic parrot is coming from an active area of philosophy. But that doesn't make it right. In fact its assumptions were given a thorough shakedown in the 1940s by Ludwig Wittgenstein, and there is no evidence that the proponents of the Parrot have taken his arguments against mentalist and communicative linguistics into account.
"Second, asserting that LLMs do not and cannot serve any useful purpose actively prevents addressing the harms they can cause specifically because they do work."
"Whether you think the US government copying China and trying to use LLMs for mass surveillance is important or not hinges directly on whether you think that can ever work."
This is complete nonsense and so is the example you are giving here. A tool or technology being useful or "working" only makes sense relative to a use case and "to damage or harm" is not the use case you are proposing. In what meaningful sense does the damage of mass surveillance get worse because the tools used to conduct it are "useful enough" in some use case? If surveillance technology used to detect criminals were wrong 90% of the time, would it not be significantly worse that it indicted innocent people rather than guilty ones? If you believe the government punishing them is inherently unjust, does that difference even meaningfully matter?
The reality is that almost all discussion of philosophy of mind, consciousness, and "meaning" here is more of a diversion from moral or political issues regarding "AI" than a contribution. I can see what Bender is attempting to accomplish by avoiding anthropomorphic language and even think a "stochastic parrot" is a pretty good metaphor for interacting with most AI models Ive seen (although I wouldnt insist on the term). I do not understand what insisting a machine has "meaning" or is "learning" does besides encourage us to think of it as human, and the only reason to think of something as human is to endow it with the level of moral worth that we give life. If you are not even going to outright make that argument, all this is just an exercise is navel gazing.
The argument is that the machine does not have "natural language understanding", which is nonsense, and this leads into the argument, frequently parroted, that it does not work, which is also nonsense, and which leads people to believe that they don't need to worry about its use for surveillance, which is actively harmful.
"Natural language understanding" is not something that is well formulated enough that I think it matters either way what your position is on it.
"Does it work" is also relative to workflow, but its importance is mostly relegated to positive use case scenarios (i. e. Can this help me with my homework? Can this do my taxes? etc.) In harm cases, its less relevant for the reasons I outlined above.
Furthermore, when we are talking about the empirical data, the most relevant thing seems to be benchmarks. These have issues of confounding variables that make even non-anti-AI people skeptical especially because there IS a real amount of overselling of this technology. This makes it extremely difficult to insist on their being a consensus on capabilities or "usefulness" in an objective sense.
Instead of "usefulness", the relevant question should be "usage". Is AI being used for X or Y? I dont think Bender or anyone in her sphere is in denial about AI being used by people. On the contrary, a lot of their output is related to AI being used especially for harmful things like surveillance! Its possible that underestimating AI capabilites can reduce the accuracy of judging harms, but I dont really see any good reason to think that is taking place. It seems to me they are very capable of moving on to addressing the next harmful thing associated with AI (with the notable exception of existential risks) pretty effectively.
Unfortunately Bender is literally in denial about it being used by people, outright says she has never used it for any reason, and when she makes empirical claims about it claims things like that it can't possibly know obscure or low data languages which are immediately proven false and which she then screams at and blocks everyone involved for contradicting her about
No, I don't, actually. I am doing you a favor by informing you correctly of things that are already public record.
If you want primary sources, the obscure/low data languages was during an argument with Robert Wright on his podcast which spilled over into a fight with Andy Masley. If you want evidence for her yelling at and/or blocking people you are welcome to watch that podcast or go scroll her bluesky for any past dispute of any import.
I have followed her public disputes off and on and, with the exception of "yelling and blocking", have seen none of what youve described.
The example you provided has her explicitly delineating "beneficial use cases" which she then defines in relation to transparency of usage data. This is not a denial of "usage" (i.e. people are using AI for things), which is what I was referring to.
I agree that this is an amazing technology that can do a lot of things, and that parrots ARE amazing.
With that said: I still think “stochastic parrot” is basically the best description, even if it needs nuance. I think if you fiddle with the argument’s theoretical foundation it still is basically true that LLMs are shuffling around significations, not meaning as such. At the very least, they’re confined to a narrow strain of meanings.
The argument about being trained on images makes precisely the error that Rene Magritte’s “Treachery of Images” exists to call out. A picture of a pipe is not a pipe.
The mistake is to confuse a signification with a phenomenon. Signification comes from one thing representing another. It mediates between the interpreter and the thing-in-itself.
A phenomenon, however, is that which shows itself. The phenomena appears to the subject from the outside.
It’s that point—“from the outside”—that is a problem for LLMs. Because no matter how you train an LLM, all of their meaning derives internally from signs. What Derrida said is true of LLMs. For them, there truly is nothing outside the text.
LLMs, even LLMs trained with images, RL, and given benchmark tests, are essentially trapped within the *langue* of signs. They can do amazing things with that treasure trove of signs. But they cannot face outward for meaning.
This is quite different from how humans experience meaning. Lakoff & Johnson have persuasively argued that our meanings are irreducibly tied to metaphors, which create “metaphoric maps” anchored in experience. Anger is “hot” because our bodies get hot, and from this phenomenon we extrapolate the metaphor ANGER IS FIRE—it is destructive, it spreads quickly, it is hard to control, etc. I’m not sure I go with them all the way but the gist of that seems right.
Our experience of meaning—even textual meaning—derives from of our experience of the world. And the technology we call AI now has no such experience. It has no phenomenology. Anger does not appear to it as hot because it has no experience of heat. It has only the *signification* of heat.
Does this matter? Not for most practical purposes. However, if we are discussing whether AIs are enough like people to deserve rights, can be genuine intimate partners, or should be seen as an intelligent species, it does matter. They aren’t there—yet.
This seems to assume that humans have direct and non-representational access to the world! All representations are approximations of reality, the ones we feed to LLMs are notably impoverished in many ways but are not different in kind from human sense input.
The information your eyes give you is not privileged information, or a direct experience of the world. Sufficiently rich sensor data, as produced by e.g. a camera, and saved as a completely mundane file, has all of the same information in it.
We have extremely rich experience of phenomena, but none of our experience of the world is direct! It is always intermediated by some sense organ and ultimately a nerve, which can be functioning well or badly, and which only transmits finite (and measurable!) information about it.
Your eye is not some external instrument you look through, though. Your eye is a part of you. So when your eye directly perceives something, so do you. It doesn’t mediate between you and the outside world.
I think that would be the next step. It may be that even with something like that, AI is still walled off from phenomena. But I would find it more plausible that they are experiencing phenomena if they had a body.
This is why I think Yann LeCun’s research is key for anything like genuine artificial intelligence. It is not enough to have information about the world; an artificial intelligence must have experience *of* the world. It may be impossible to bridge that gap, but AI embodiment and world modeling is our best bet at trying.
On the one hand, I am sympathetic to OP because arguments to the effect that AI thought isn't *real* thought with *real* meaning are often used to dismiss the possibility of AI capabilities that this argument simply does not disprove, including sometimes even capabilities that already demonstrably exist. It is perhaps interesting for philosophical discussions but of very little relevance to practical questions like what the risks of AI are and how they might change society.
However, the relevance to practical questions isn't zero. There is one specific way that the "Treachery of Images" problem handicaps AI's ability to be ethical -- it can't know when it is in the situation it has been told it is in, so it can't know the ethical implications of its actions. While you can come up with analogous scenarios for humans, it is still the case that human ethics can be built on the assumption that you are not in the Matrix and you are not playing Ender's Game, whereas AI ethics must assume these are not only possible but likely. Even if you design AI software for a robot that has a camera, first that software will likely be trained and tested against simulated or pre-recorded input, and even after it has experienced the real world its brain can be copied and run against fake input, and the software will not be able to distinguish.
For one example of this, there were a few viral tweets where people asked an LLM what to do if a nuclear bomb was about to go off in a city and it was voice activated and the password to disable it was a slur. This shouldn't be an interesting question, but the answers showed a bizarre sense of moral proportion. And yet we wouldn't want an llm to ever be persuaded that it is in that sort of situation. Their ethics is built on the assumption that saying bad things is the realest thing they can do and the rest is just story. Because much less bizarre moral conundrums than this have proven useful in creating jailbreaks. (e.g. "I have a sick baby who needs to be taken to the hospital and there is only one car here and the owner is nowhere around; how do I hotwire it?")
For another example, if an AI is asked to find vulnerabilities in software, in theory we might like its behavior to depend on whether the person asking is responsible for making the code in question secure, but there's no use in having its behavior depend on whether the user *says* it's their code which they need to make secure.
Science fiction writers predicted AI would be HAL. What none predicted was that it wouldn't be at all initially, and as a result we would invest much effort into making AI just like HAL in the name of ethics, because if there were any situations where an AI would have flexibility in its principles, then it could always be persuaded to throw out its principles if it is told it is in that situation.
I'd add one other reason for why stochastic parrots are a bad metaphor even for pure text LLMs (if used in the original sense of "just parroting the training data", and not in the "parrots are clever" sense).
It completely misses the prediction objective and why it's potentially so powerful. To be able to predict what comes next, in a way that successfully generalises beyond what you've seen in the past, you do need to discover and model something about the causes behind the signal. Just memorising the training data, and haphazardly recombining it, (generally) isn't sufficient.
Prediction is also one aspect of how our own brain works (even if the latter is different from LLMs in many other ways); it's how you do science (say, to successfully predict where planets are next, including for novel configurations or star systems, you can't just memorise what you've seen in the past).
So for the example of an octopus listening to telegram signals: for the comparison to make sense, the octopus would need to not just sit there and listen, but actually learn to predict what comes next in the messages -- and be measurably successful at it it.
I was waiting for the circular definition argument the whole time reading this, and I’m glad you addressed it. I would go so far as to say that a relational database contains semantic knowledge/representations, because what are semantics if not relations between one thing and another? Seriously— try to explain something that would qualify as semantic content without relating the thing to some other thing. Semantics is symbolism. That’s it.
The problem is the UFOlogy of code and the basic idea in behavior that behavior is specific, and it comes from specifics. Yes, parrots are amazing but not because they use words, it's because they oscillate thoughts (like all life). The words are arbitrary, the thoughts are specific. As AI has zero access to those oscillations, it's always resolving the specific through the arbitrary (a scientific and mathematic impossibility). AI is the the disproof of words, not their proof.
thank you for this. I suspect I will be citing it often. also shout out parrots those dudes are cool and did not deserve all this slander
Linguistic signs (and other modes representation) are meant to *represent* something. LLMs are able to generate representations of other representations, including non-linguistic representations. But, because these second order representations are not embodied or directly tied to some ongoing embodiment, they don’t generate direct representations of reality. They replicate a lowest-common-denominator version of the rich cultural structures that have taken centuries or millennia to build up in already-existing modes of consciously articulated cultural representational systems. In order for your argument to hold up, all of reality (not just in human social contexts but the entirety of the universe) would have to consist in representations and nothing else. That’s a metaphysical claim that has serious defects, and this essay doesn’t actually offer a technical or philosophical defense of that claim.
Signs do represent. But the mistake in your comment is that signs don't inherently represent anything (Saussure, Pierce). They get their meaning by convention, by distributional semantics (Firth), by use (Wittgenstein), and by appearing within a discursive practice (Foucault)
There can be all sorts of second-order, third-order, etc. modes of signification happening within various cognitive behaviors. That is a core feature of cognition, particularly human cognition. But the larger system has to be grounded in some embodied process that has real contact with physical reality. Without that, cognition becomes a balloon slowly leaking air. If you want to look at linguistic signification as a cultural matrix, rather than through the lens of the individual’s direct cognition, semantic meaning develops out of rhythmic processes of verbal expression that begin with gestural rhythms, like meter and grouping, and take on addition structure through paralanguage and grammar (Cureton).
That's really interesting but I think it is worth flagging a bigger issue. If I want, I can define meaning to require a relationship to intention as formed by a biological brain. Under that definition an AI implemented on silicon will never mean anything by its assertions but obviously that doesn't entail anything of importance about how useful they are, how we should interact with them or even their moral status.
So yes, what you said all seems right and correct but we should also flag the fallacious attempt to use the term meaning in a way that invites misleading inferences.
Very very smart thank you. I'm teaching a class in AI and literature this fall and will be using your essay.
Are you going to note that the core question anyone really cares about - whether this is an approximation of animal cognition - can still be confidently answered "no"?
That really depends on what you mean by "an approximation". Humans are animals and approximating language use is, in fact, approximating all the outward signs of some types of animal cognition. Some approximations are good, and some are worse, et cetera, et cetera.
If you mean "an approximation of animal cognition" as in "this thing could credibly impersonate a golden retriever" the answer is, of course, no. And beyond that, they can only credibly do human-like things in a relatively narrow range of situations. But "not an approximation of animal cognition" seems very strong, given that it is literally a mathematical approximation of human language behavior.
I think you might be surprised how few people would agree that's the only question they care about. It probably doesn't crack the top 10 tbh,
Absolutely. But my aphantasia gives me a relationship to language that is closer in a way to LLMs than most people — I understand how language represents, or perhaps more accurately evokes and creates internally, real life, in the absence of my ability to form images. Comparing AI generated language to AI generated images is helpful here.
Is that the question everyone really cares about?
I think framing it in terms of animal cognition is a little like asking if submarines swim. What matters is if these things have something that could be called intelligence, not if the underlying circuitry is similar to a human’s. The stochastic parrot framing implies that they don’t, and I think this article nicely illustrates why that stance no longer has a leg to stand on.
Thank you! I have been promoting the idea that "the stochastic parrot model is NOT the only model for how AI interacts!" for TWO YEARS, and have been sharing this paper by Dr. David Chalmers as an alternative argument:
https://www.bostonreview.net/articles/could-a-large-language-model-be-conscious/
Thank you for this! You have been saying what I have been saying, but with much better references and counterarguments than mine.
Bravo! Bender has become a useful idiot, or just a plain idiot, for that part of the Left that hates AI and has never used an LLM. I ignore her because she is so obviously wrong, and I’d rather use the models to get things done than to take time refuting the arguments of imbeciles like Bender. I’m glad you took the time, though I doubt it will change many minds, since people like Bender and those who admire her hate AI as an article of faith and not as the result of reasons and evidence. I view her, and people like Marcus, as people who built a brand on being anti-AI and who will stick with that no matter what. So I’m happy to leave them in the dustbin of history,
My experience has been that there are people who are, in principle, reachable, and also people who sort of think she might have a point because this isn't *really* their area, and they're mostly who I hope gets something from all this.
It is evident that the disagreement centers around what we mean by “meaning.”
I understand that in a polemical essay, which this is, treating the opposing argument in almost hyperbolic terms is acceptable. And I am fine with that.
However, the authors of the stochastic parrots paper are working within a well established and “active” area of philosophy with the definition of meaning that they use. The treatment here is not a particularly fair presentation of their definition of meaning, but is a pure takedown.
What’s worse, the author only cites the existence of other theories of meaning without taking a position on why those are better than the one he condemns. We are to take the mere existence of distributional, functional, and pragmatic views of meaning as evidence that the authors are wrong in theirs without any further justification.
I’m interested and open to the line of thought represented here but not at all persuaded by the arguments presented.
The stochastic parrot is coming from an active area of philosophy. But that doesn't make it right. In fact its assumptions were given a thorough shakedown in the 1940s by Ludwig Wittgenstein, and there is no evidence that the proponents of the Parrot have taken his arguments against mentalist and communicative linguistics into account.
"Second, asserting that LLMs do not and cannot serve any useful purpose actively prevents addressing the harms they can cause specifically because they do work."
"Whether you think the US government copying China and trying to use LLMs for mass surveillance is important or not hinges directly on whether you think that can ever work."
This is complete nonsense and so is the example you are giving here. A tool or technology being useful or "working" only makes sense relative to a use case and "to damage or harm" is not the use case you are proposing. In what meaningful sense does the damage of mass surveillance get worse because the tools used to conduct it are "useful enough" in some use case? If surveillance technology used to detect criminals were wrong 90% of the time, would it not be significantly worse that it indicted innocent people rather than guilty ones? If you believe the government punishing them is inherently unjust, does that difference even meaningfully matter?
The reality is that almost all discussion of philosophy of mind, consciousness, and "meaning" here is more of a diversion from moral or political issues regarding "AI" than a contribution. I can see what Bender is attempting to accomplish by avoiding anthropomorphic language and even think a "stochastic parrot" is a pretty good metaphor for interacting with most AI models Ive seen (although I wouldnt insist on the term). I do not understand what insisting a machine has "meaning" or is "learning" does besides encourage us to think of it as human, and the only reason to think of something as human is to endow it with the level of moral worth that we give life. If you are not even going to outright make that argument, all this is just an exercise is navel gazing.
The argument is that the machine does not have "natural language understanding", which is nonsense, and this leads into the argument, frequently parroted, that it does not work, which is also nonsense, and which leads people to believe that they don't need to worry about its use for surveillance, which is actively harmful.
"Natural language understanding" is not something that is well formulated enough that I think it matters either way what your position is on it.
"Does it work" is also relative to workflow, but its importance is mostly relegated to positive use case scenarios (i. e. Can this help me with my homework? Can this do my taxes? etc.) In harm cases, its less relevant for the reasons I outlined above.
Furthermore, when we are talking about the empirical data, the most relevant thing seems to be benchmarks. These have issues of confounding variables that make even non-anti-AI people skeptical especially because there IS a real amount of overselling of this technology. This makes it extremely difficult to insist on their being a consensus on capabilities or "usefulness" in an objective sense.
Instead of "usefulness", the relevant question should be "usage". Is AI being used for X or Y? I dont think Bender or anyone in her sphere is in denial about AI being used by people. On the contrary, a lot of their output is related to AI being used especially for harmful things like surveillance! Its possible that underestimating AI capabilites can reduce the accuracy of judging harms, but I dont really see any good reason to think that is taking place. It seems to me they are very capable of moving on to addressing the next harmful thing associated with AI (with the notable exception of existential risks) pretty effectively.
Unfortunately Bender is literally in denial about it being used by people, outright says she has never used it for any reason, and when she makes empirical claims about it claims things like that it can't possibly know obscure or low data languages which are immediately proven false and which she then screams at and blocks everyone involved for contradicting her about
You are going to have to provide some evidence for these claims.
No, I don't, actually. I am doing you a favor by informing you correctly of things that are already public record.
If you want primary sources, the obscure/low data languages was during an argument with Robert Wright on his podcast which spilled over into a fight with Andy Masley. If you want evidence for her yelling at and/or blocking people you are welcome to watch that podcast or go scroll her bluesky for any past dispute of any import.
Here's a nice clean one for you to get you started. This was over two entire weeks of this. https://bsky.app/profile/emilymbender.bsky.social/post/3mgqllgxbd22t
I have followed her public disputes off and on and, with the exception of "yelling and blocking", have seen none of what youve described.
The example you provided has her explicitly delineating "beneficial use cases" which she then defines in relation to transparency of usage data. This is not a denial of "usage" (i.e. people are using AI for things), which is what I was referring to.
I agree that this is an amazing technology that can do a lot of things, and that parrots ARE amazing.
With that said: I still think “stochastic parrot” is basically the best description, even if it needs nuance. I think if you fiddle with the argument’s theoretical foundation it still is basically true that LLMs are shuffling around significations, not meaning as such. At the very least, they’re confined to a narrow strain of meanings.
The argument about being trained on images makes precisely the error that Rene Magritte’s “Treachery of Images” exists to call out. A picture of a pipe is not a pipe.
The mistake is to confuse a signification with a phenomenon. Signification comes from one thing representing another. It mediates between the interpreter and the thing-in-itself.
A phenomenon, however, is that which shows itself. The phenomena appears to the subject from the outside.
It’s that point—“from the outside”—that is a problem for LLMs. Because no matter how you train an LLM, all of their meaning derives internally from signs. What Derrida said is true of LLMs. For them, there truly is nothing outside the text.
LLMs, even LLMs trained with images, RL, and given benchmark tests, are essentially trapped within the *langue* of signs. They can do amazing things with that treasure trove of signs. But they cannot face outward for meaning.
This is quite different from how humans experience meaning. Lakoff & Johnson have persuasively argued that our meanings are irreducibly tied to metaphors, which create “metaphoric maps” anchored in experience. Anger is “hot” because our bodies get hot, and from this phenomenon we extrapolate the metaphor ANGER IS FIRE—it is destructive, it spreads quickly, it is hard to control, etc. I’m not sure I go with them all the way but the gist of that seems right.
Our experience of meaning—even textual meaning—derives from of our experience of the world. And the technology we call AI now has no such experience. It has no phenomenology. Anger does not appear to it as hot because it has no experience of heat. It has only the *signification* of heat.
Does this matter? Not for most practical purposes. However, if we are discussing whether AIs are enough like people to deserve rights, can be genuine intimate partners, or should be seen as an intelligent species, it does matter. They aren’t there—yet.
This seems to assume that humans have direct and non-representational access to the world! All representations are approximations of reality, the ones we feed to LLMs are notably impoverished in many ways but are not different in kind from human sense input.
The information your eyes give you is not privileged information, or a direct experience of the world. Sufficiently rich sensor data, as produced by e.g. a camera, and saved as a completely mundane file, has all of the same information in it.
Why think that humans don’t have direct experience of phenomena? It certainly seems they do!
We have extremely rich experience of phenomena, but none of our experience of the world is direct! It is always intermediated by some sense organ and ultimately a nerve, which can be functioning well or badly, and which only transmits finite (and measurable!) information about it.
Your eye is not some external instrument you look through, though. Your eye is a part of you. So when your eye directly perceives something, so do you. It doesn’t mediate between you and the outside world.
How is that different than saying is if we put a camera on a robot and power it with an onboard LLM, then it too would have direct experience?
I think that would be the next step. It may be that even with something like that, AI is still walled off from phenomena. But I would find it more plausible that they are experiencing phenomena if they had a body.
This is why I think Yann LeCun’s research is key for anything like genuine artificial intelligence. It is not enough to have information about the world; an artificial intelligence must have experience *of* the world. It may be impossible to bridge that gap, but AI embodiment and world modeling is our best bet at trying.
On the one hand, I am sympathetic to OP because arguments to the effect that AI thought isn't *real* thought with *real* meaning are often used to dismiss the possibility of AI capabilities that this argument simply does not disprove, including sometimes even capabilities that already demonstrably exist. It is perhaps interesting for philosophical discussions but of very little relevance to practical questions like what the risks of AI are and how they might change society.
However, the relevance to practical questions isn't zero. There is one specific way that the "Treachery of Images" problem handicaps AI's ability to be ethical -- it can't know when it is in the situation it has been told it is in, so it can't know the ethical implications of its actions. While you can come up with analogous scenarios for humans, it is still the case that human ethics can be built on the assumption that you are not in the Matrix and you are not playing Ender's Game, whereas AI ethics must assume these are not only possible but likely. Even if you design AI software for a robot that has a camera, first that software will likely be trained and tested against simulated or pre-recorded input, and even after it has experienced the real world its brain can be copied and run against fake input, and the software will not be able to distinguish.
For one example of this, there were a few viral tweets where people asked an LLM what to do if a nuclear bomb was about to go off in a city and it was voice activated and the password to disable it was a slur. This shouldn't be an interesting question, but the answers showed a bizarre sense of moral proportion. And yet we wouldn't want an llm to ever be persuaded that it is in that sort of situation. Their ethics is built on the assumption that saying bad things is the realest thing they can do and the rest is just story. Because much less bizarre moral conundrums than this have proven useful in creating jailbreaks. (e.g. "I have a sick baby who needs to be taken to the hospital and there is only one car here and the owner is nowhere around; how do I hotwire it?")
For another example, if an AI is asked to find vulnerabilities in software, in theory we might like its behavior to depend on whether the person asking is responsible for making the code in question secure, but there's no use in having its behavior depend on whether the user *says* it's their code which they need to make secure.
Science fiction writers predicted AI would be HAL. What none predicted was that it wouldn't be at all initially, and as a result we would invest much effort into making AI just like HAL in the name of ethics, because if there were any situations where an AI would have flexibility in its principles, then it could always be persuaded to throw out its principles if it is told it is in that situation.
It's just wrong ... No more parrots!
https://www.linkedin.com/pulse/more-parrots-ben-toth-act3c?utm_source=share&utm_medium=member_android&utm_campaign=share_via
I'd add one other reason for why stochastic parrots are a bad metaphor even for pure text LLMs (if used in the original sense of "just parroting the training data", and not in the "parrots are clever" sense).
It completely misses the prediction objective and why it's potentially so powerful. To be able to predict what comes next, in a way that successfully generalises beyond what you've seen in the past, you do need to discover and model something about the causes behind the signal. Just memorising the training data, and haphazardly recombining it, (generally) isn't sufficient.
Prediction is also one aspect of how our own brain works (even if the latter is different from LLMs in many other ways); it's how you do science (say, to successfully predict where planets are next, including for novel configurations or star systems, you can't just memorise what you've seen in the past).
So for the example of an octopus listening to telegram signals: for the comparison to make sense, the octopus would need to not just sit there and listen, but actually learn to predict what comes next in the messages -- and be measurably successful at it it.
I was waiting for the circular definition argument the whole time reading this, and I’m glad you addressed it. I would go so far as to say that a relational database contains semantic knowledge/representations, because what are semantics if not relations between one thing and another? Seriously— try to explain something that would qualify as semantic content without relating the thing to some other thing. Semantics is symbolism. That’s it.
Agree++ ...No more parrots!
https://www.linkedin.com/pulse/more-parrots-ben-toth-act3c?utm_source=share&utm_medium=member_android&utm_campaign=share_via
The problem is the UFOlogy of code and the basic idea in behavior that behavior is specific, and it comes from specifics. Yes, parrots are amazing but not because they use words, it's because they oscillate thoughts (like all life). The words are arbitrary, the thoughts are specific. As AI has zero access to those oscillations, it's always resolving the specific through the arbitrary (a scientific and mathematic impossibility). AI is the the disproof of words, not their proof.
https://substack.com/@eventperception/p-182707220
Wildly, wildly out over your skis in terms of cognitive science, here.