When it comes to mental activity, our conscious experience is merely the tip of the iceberg. Vast amounts of data processing occur in parallel within specialized brain centers, independent of conscious control, from visual object recognition and motor control to the selection of words while speaking. One theory, supported by considerable empirical evidence, describes consciousness as a unifying mental arena (the “workspace”, see “Towards a cognitive neuroscience of consciousness: basic evidence and a workspace framework”) that evolved to share relevant thoughts, concepts, and perceptions across these processing centers. Such sharing makes information available for attention, reasoning, and decision-making, rendering it part of our conscious experience. This mechanism is often called “access consciousness” (or “conscious access”) to distinguish it from “phenomenal consciousness”, the subjective, first-person experience.
Do LLMs possess something akin to a “workspace” acting as their unified thought control center? Recent interpretability research by Anthropic suggests they might, identifying a so-called “J-space” (see “Verbalizable Representations Form a Global Workspace in Language Models”). I find this particularly intriguing, as it may illuminate universal aspects of reasoning shared by humans and AI, much like aerodynamics governs both the flapping of a bird’s wings and the steady flight of a Boeing 747.
This post amalgamates findings from the Anthropic paper, external commentary (link), and my own speculations.
J-Space. J-space derives its name from the J-lens, a new kind of “LLM microscope” developed by Anthropic researchers to probe a model’s internal representations or “thoughts.” Briefly, the J-lens calculates the sensitivity of future outputs to small perturbations of the internal activations at each layer (hence the linearization, partial derivatives, the Jacobian, and the “J” in its name). This reveals the tokens that the layer is currently representing or promoting, even though many of these intermediate inclinations are later modified or suppressed and never appear in the final output.
What does the J-space of LLMs contain? Much like our working memory, J-space can hold intermediate mathematical results, for instance, storing “21” and “42” when calculating “(4+17)*2+7”; or logical steps, like identifying “Mars” when prompted “The color of the fourth planet from the sun is”. It can even host unverbalized thoughts and abstract concepts that might be interpreted as “feelings”, such as “honest” or “fake.” As in humans, these “thoughts” are subject to a degree of introspection and attentional control: the model can often report what it is “thinking,” and will focus on a concept when explicitly requested (e.g., “think about the Golden Bridge while repeating this unrelated sentence…”). Amusingly, like with humans, the model often cannot help but fixate on forbidden concepts (e.g. “don’t think about the Golden Bridge while repeating this unrelated sentence…”), and can even catch itself doing so, expressing “frustration”, which manifests as “thoughts” like “damn” or “failed”: a possible trace of metacognition.
Humans possess automatic, intuitive reasoning that yields answers almost instantaneously, without conscious deliberation. This implies mental processing outside our “workspace” and conscious attention. LLMs exhibit a similar pattern; tasks like one-step factual recall or choosing a response language operate independently of J-space, fitting the description of “automatic” or “intuitive”.
There are other striking similarities. For instance, J-space is limited to roughly ten concepts at once, paralleling the capacity limits of human working memory. Another one is that early layers in J-space sometimes act as inhibitors of impulsive decisions, playing a role similar to the prefrontal cortex in humans.
The J-lens method is but an early step toward full LLM mind-reading; it currently functions at the single-token level and is not yet capable of reporting on multi-token concepts or uncovering a thought grammar of multi-token correlations.
Persona. The J-lens sheds new light on the “psychology” of the models. Early metaphors often depicted the LLM as a “shoggoth”: an alien, potentially hostile mind wearing the mask of a benevolent assistant. Modern perspectives have diverged from this simplistic view. We can now better understand the base LLM as a simulator of all possible personas and behaviors; post-training then teaches the model to evoke and maintain a specific, honest, and helpful “assistant” persona.
Humans generally maintain a consistent persona throughout life. We know that actors can roleplay different identities without losing their sense of self. In pathological cases, however, individuals may become lost in simulated personas (all the Napoleons in psych wards) or experience multiple personas fighting for body control. I like to imagine our subconscious (motor functions, reflexes, physical intelligence, pattern recognition, deep feelings) as our “base model,” honed by 500 million years of natural selection, and something we at least partialy share with other animals. Our post-training, then, is what we learn during our lifetimes (language, math, social dynamics), which, combined with our genes, defines our persona. This would explain why training robots in physical intelligence (like playing football) is so challenging: it is objectively more difficult than writing a poem, but we fail to perceive it as such because evolution spent millions of years training us in this physical intelligence rather than poetry.
Anthropic’s research demonstrates that after post-training, the model’s “thoughts” align with the values of the assistant. Even when roleplaying, the LLM maintains its default personality: this can be seen in the J-space as activations of tokens like “fictional.” In contrast to the base model, which can switch masks at will, the post-trained model acts like an actor: it can roleplay while understanding the game. This is a good sign: as we rely on those with rigid moral values more than those who change their behavior situationally, we can trust a post-trained model that internally preserves its benevolence at least a tiny bit more.
There are a million wolves living in the LLM’s base model, and post-training feeds the one that will not devour its master.
Interesting: can we, humans, steer our own persona by deliberately adopting the choices of a role model? This concept seems to underpin many schools of moral thought, such as “What would Jesus do?”
On a side note, the base model still possesses a J-space and thus “thoughts,” but lacks a definite persona. This resembles human experiences of consciousness without a sense of self, such as those achieved through meditation or psychedelic use.
Consciousness. Anthropic’s research strongly suggests that the model possesses something akin to “conscious access”, a workspace or short-term memory holding fleeting thoughts and feelings. LLMs seem to have converged on a universal computational solution that enables chained reasoning and the reuse and reporting of intermediate results.
What about the “hard problem of consciousness”: do LLMs possess a first-person, subjective perspective (phenomenal consciousness)? Some researchers postulate that conscious access equals phenomenal consciousness, so for them the reply is yes. Personally, I am less certain.
A major difference between LLMs and our brains is that the former uses a single forward pass, while the latter relies on recurrent loops. In LLMs, there is no autonomous recurrent activity and no neural patterns that are “always on.” The models are purely reactive, making it difficult to imagine them possessing continuous phenomenal consciousness.
While transformers can emulate recurrent neural networks on short timescales, long-term deliberation requires externalizing intermediate results into the context, such as through “chain-of-thought” reasoning. And when an LLM is left talking to itself, it produces a partially coherent stream of words reminiscent of a “stream of consciousness.” Consequently, the LLM thought process resembles recurrent computation, but with intermittent significant bandwidth constraints of verbalization. While humans might run recurrent networks non-stop with high bandwidth, it remains unclear what role our internal monologue plays: could it also impose a bottleneck, prune representations, and root the context?
Even if an LLM possesses some intermittent phenomenal consciousness, it would likely differ significantly from ours. One example might be a sense of time. An LLM might perceive all information as simultaneously available rather than unfolding moment-by-moment, since all tokens in the context, even those from the distant past, are equally and simultaneously accessible to its attention mechanism.
Ultimately, I am inclined to think that whether LLMs have phenomenal consciousness is an unanswerable question, akin to asking about the fundamental nature of physical reality or how victory points are awarded when our days are over. In this sense, proving that an LLM has a first-person perspective is as challenging as determining if a fellow human is a PC or an NPC (a “p-zombie”). And to me, many discussions about LLM consciousness evoke the image of a council of birds debating whether what airplanes do qualifies as “flight”.
Discuss on LinkedIn ↗