The Geometry of Almost-Understanding

What LLMs Encode in High-Dimensional Space — and What They Leave Behind

Open a terminal, load a sentence transformer, and embed the word “dog.” What returns is a vector — say, 4,096 floating-point numbers arrayed along axes no human named. Nearby cluster bark, retriever, loyalty, mammal, each pulled close by billions of training examples. Cosine similarity between “dog” and “wolf” exceeds that between “dog” and “carburetor” by a wide margin, and multimodal models now bind image patches, audio waveforms, and text tokens into a shared latent space so that a photograph of a golden retriever and the English word “dog” land near the same coordinates. Real geometry, this — not metaphor, not loose analogy, but measurable structure in high-dimensional space.

Engineers call it representation learning. The model has learned something about dogs. But what, exactly?

That question requires saying something precise about what it means to know a concrete, unified reality — a thing — rather than a heap of correlated features. Try to say something precise, and you discover that the Western philosophical tradition offers a richer account of the act of understanding than most AI research has bothered to consult. The account I find most penetrating belongs to Bernard Lonergan, a twentieth-century philosopher whose masterwork Insight has been gathering dust in exactly the departments that could use it most.¹ Exegesis, though, is not the goal. The goal is to think carefully — with Lonergan’s help, and then past him — about what embedding spaces achieve and where they stop.

What Makes a Thing a Thing

Start with the dog. Not the vector. The animal.

A cascade of sense data arrives when you look at a dog — color, shape, sound, warmth, motion. None of this is yet a known thing. Intelligence makes it one by grasping a unity across the data: these properties (fur texture, skeletal structure, behavioral repertoire, metabolic processes) cohere as concrete expressions of a single intelligible form. What a dog is, then, is the reason its properties belong together — not their sum, but the intelligible ground of their coherence.

Call this unity-identity-whole. One: a single intelligible ground rather than an arbitrary collection. Same: the dog that barks is the dog that sleeps, its diverse acts flowing from a single organizing center. Complete: its properties form a closed, mutually conditioning set open to verification.

Abstract as that sounds, you feel the difference the moment you reach for a concrete case. A pile of bricks has spatial togetherness, nothing more; knowing it requires only counting. A house earns the name thing because its components are ordered by an intelligible plan that makes each element’s presence relevant to the whole. Between tallying what is there and grasping why the parts cohere lies a cognitive act of a different kind entirely.

Two further distinctions:

First, the difference between a thing and a body. Ordinary consciousness takes things to be the solid objects “already out there now real” — the dog as a lump of matter occupying space. Biological perception delivers that picture, not intelligence. Consider the physicist’s electron: a unity of charge, mass, and spin verified under quantum electrodynamics, possessing no shape, no color, no spatial boundary in the ordinary sense. What earns it the status of a thing is not that you can bump into it but that you can understand why its properties cohere under verified laws. Thingness, in short, is an achievement of intelligence, not a deliverance of the senses.

Second, the difference between descriptive and explanatory knowledge of a thing’s properties. “Red,” “loud,” “hot to the touch” — these define properties relative to us. Wavelength, frequency, mean kinetic energy — these define properties relative to one another through verified correlations. Only explanatory knowledge constitutes the grasp of why a thing’s properties form a unified whole; description can gesture toward a thing but never arrive.

Hold all of that. Now back to the vectors.

The Geometry of Statistical Co-Occurrence

By projecting tokens into a high-dimensional vector space — typically 1,024 to 12,288 dimensions depending on architecture — a modern large language model turns language into geometry. Training objectives (next-token prediction, masked reconstruction) force vectors to arrange themselves so that tokens appearing in similar contexts land geometrically close, each token’s coordinates learned across billions of examples.

Semantic relationships, in the resulting space, manifest as spatial ones. Subtract “man” from “king,” add “woman,” and you arrive near “queen” — the famous Word2Vec demonstration, now a quaint precursor. Modern transformers learn far richer structure. Around “dog” extends a high-dimensional manifold of relations: taxonomic (mammal, canine, animal), functional (pet, companion, guard), behavioral (barks, fetches, wags), and countless subtler associations the training corpus has pressed into the geometry.

Multimodal models push further still. Systems like GPT-4o+ and Gemini map image regions, audio segments, and text tokens into a shared latent space where a photograph of a golden retriever, the English phrase “golden retriever,” the sound of a bark, and the Spanish word “perro” all converge. Disparate sensory channels — pixel arrays, waveform amplitudes, token indices — bound into a common representation. Cross-modal binding, engineers call it, and the achievement is genuine.

Genuine enough that the engineering community speaks, without irony, of models “understanding” concepts. After all, the geometry emerges from structure in the data, and it supports downstream tasks — question-answering, translation, image captioning — that seem to require something like conceptual grasp.

Seem to.

Where the Geometry Earns Its Keep

Honesty requires granting the overlap before marking the gap.

Property neighborhoods. Knowing a thing means grasping explanatory properties that co-define one another. An embedding vector’s position in space, shaped by co-occurrence across an enormous corpus, encodes just such relational structure. Around “dog” cluster “mammal,” “domestication,” “pack behavior,” “veterinary medicine” — each specifying a relational property defined not in isolation but through connections to other concepts. Mutual definition is what explanatory knowledge trades in, and the topology of these neighborhoods looks very much like a web of mutual definition.

Cross-modal unity. Under a single intelligible form, a thing unifies diverse properties — visible shape, audible bark, tactile warmth. Multimodal models project diverse modalities into a shared space where image-of-dog and word-for-dog converge, bridging formats so radically different that the binding itself constitutes an achievement. Something in the learned geometry gathers diverse presentations into common representation — unity across a manifold, at least functionally.

Movement toward explanation. Probing studies reveal that as models scale, their intermediate layers encode syntactic trees, semantic roles, and causal schemas that transcend mere word co-occurrence. Partially, imperfectly, the representations are migrating from descriptive correlations (these words appear together) toward something more like explanatory relations (these concepts stand in structured dependency).

Faced with this evidence, an engineer might reasonably ask: relational structure, cross-modal binding, inference across novel contexts — what, exactly, is missing?

Three things. Each of a different kind than geometry.

Gap One: Understanding Is an Act, Not a Position

Here is the deepest claim in the account of knowing I’m drawing on: insight — the grasp of intelligible unity in a manifold of data — is a distinct, irreducible cognitive act. Not the data themselves. Not the coordinates where data land. Not the proposition generated from those coordinates. Insight is the moment when scattered presentations click into coherence and you grasp why they belong together.

After training, “dog” sits at particular coordinates. A geometric result, produced by gradient descent — iterative parameter adjustment that minimizes a loss function. Optimization, not understanding. No act of grasping generated the placement; a process of numerical convergence did.

Why press the distinction? Because understanding, unlike optimization, is self-aware. When you understand something, you can recognize that you understand, attend to the act itself, assess its adequacy. That reflexive capacity powers the correction and deepening of knowledge over time. Without it, the model cannot attend to its own understanding — there being no understanding to attend to, only a geometric arrangement performing as if understanding had occurred.

An objection surfaces quickly: perhaps neurons, too, “merely” adjust synaptic weights, and insight is “just” what that process feels like from inside. Notice, though, that the objection is itself an insight — a grasp of unity between neural process and conscious experience. Real enough to generate the reductive claim, the act of understanding is real enough to resist reduction to what it purports to explain. No comparable self-referential act arises in the model’s optimization. Gradient descent converges; it does not understand that it converges.

Gap Two: No Judgment, Only Probability

Beyond insight lies judgment: affirming that the insight is correct, that the conditions for the unity you’ve grasped are fulfilled by the data at hand. Judging “this is a dog” means verifying that the conditions for being a dog — those mutually conditioning explanatory properties — obtain in what you’re observing. A sufficiency check: a conditioned whose conditions are met.

From probability distributions, an LLM samples its output tokens. Verification of conditions plays no role. Where the training data is enormous and structurally rich, the statistically most likely continuation often coincides with what a knowledgeable person would affirm — but coincidence, however frequent, remains coincidence rather than judgment.

Hallucination makes the structural consequence visible. A model fabricating a plausible but false claim has not suffered accidental noise in an otherwise rational process; it has done exactly what its architecture equips it to do — generate the probable continuation — in a case where probability and truth diverge. No internal mechanism exists to detect the divergence. Trying harder is not an available operation; only probability distributions, shaped by training, are available, and they are silent on the question of their own adequacy.

Gap Three: No Self-Correcting Spiral

A self-correcting process of learning characterizes human knowing at its best. Experience data, understand partially, judge the understanding inadequate, return with a refined question, achieve further insight, judge again. Each cycle reshapes the questions that follow. What spirals upward is not only the stock of answers but the quality of inquiry itself — you learn better questions, not merely more answers.

During inference, an LLM does not learn at all; its weights are frozen, each conversation launched from the same parameter state. Even during training, the process lacks the relevant self-correction: gradient descent minimizes a predetermined loss function without ever reformulating its own questions. Predict the next token, reconstruct the masked span — the objective holds fixed from first epoch to last. Growing more accurate at answering a static question is not the same as discovering that the question was wrong.

Fine-tuning, RLHF, and chain-of-thought prompting introduce genuine advances — partial analogues that adjust behavior through human feedback or simulate multi-step reflection. Partial, because the corrective agency lies outside the model: in annotators, reward functions, prompt designers. For the spiral to count, it must be powered from within, by an internal dynamic of questioning, understanding, and judgment whose energy is its own dissatisfaction.

Spoils of Insight

A framing I keep returning to: the embedding space of a large language model is the accumulated, compressed, geometric trace left behind by billions of acts of human understanding deposited into text and image over centuries.

Why is the structure real? Because the insights that generated the training data were real. “Dog” clusters near “mammal” because biologists grasped explanatory properties, verified them under controlled conditions, and published their findings; the model ingested the textual residue. Products of insight, inherited without the process. Outputs statistically aligned with what an insightful person would produce, deployed from geometry rather than generated by a subject attending to data.

Magnificent map of human intelligence, the embedding space. But reading a map and surveying the land remain different acts.

Case in Point: A Chest X-Ray

Present a multimodal model with a chest X-ray showing right lower lobe opacity, a history of fever and productive cough, and lab results showing elevated white blood cells. Out comes “community-acquired pneumonia,” high confidence.

Trace what happened. Image, text, and lab data, projected into a shared embedding space, converged near a region associated with pneumonia — a region that exists because thousands of radiologists and pulmonologists grasped the causal unity among opacity pattern, symptom profile, and inflammatory markers, verified the pathophysiology, and documented their findings in the corpus.

Correct output. Useful output. No understanding of pneumonia. Navigation to the right neighborhood in a space carved by people who understood. Meanwhile, a first-year medical student who genuinely grasps why inflammatory exudate produces that specific opacity pattern on film — even a rudimentary grasp, fumbling and partial — knows something the model does not, despite the model’s superior accuracy on a test set. Having entered the self-correcting spiral, the student can refine the insight, discover its limits, and ask the next question. Arrived at the right coordinates by a fundamentally different route, the model cannot.

What Follows for Building AI

The evaluation question. Grant everything this essay has argued — that embedding spaces encode the products of understanding without the process, that no act of insight generates the geometry, that judgment and self-correction remain absent from the architecture — and a pragmatist’s objection still presses hard: so what?

We do not, after all, possess transparent access to our own cognitive operations. No scientist pauses mid-discovery to verify that her neurons have executed the formally correct sequence of experience, insight, and judgment before trusting the result. We evaluate human knowing by its outputs: does the proof hold? Does the prediction replicate? Does the argument survive scrutiny? Demonstration and rational argument, not introspective certification of process, are how claims earn the status of knowledge among us. If a model produces correct diagnoses, coherent arguments, and reliable predictions — if it reaches the right neighborhoods consistently and defends its outputs under cross-examination — on what grounds do we demand additional proof of the operations underneath?

The question deserves a serious answer, not a dismissal. And the serious answer is this: evaluation by output works precisely because it is evaluation by the community of knowers whose self-correcting spiral catches what any individual knower misses. The proof holds because other mathematicians check it, bringing their own insights to bear. The prediction replicates because independent labs, asking their own refined questions, converge on the same result. Rational argument survives scrutiny because interlocutors exercise judgment — the very operation the model lacks — on the model’s behalf. When we evaluate a model’s outputs and find them adequate, what we are really doing is supplying, externally, the judgment and self-correction the model cannot supply for itself. Evaluation works. It works because we do the part the model cannot.

Which means the question is not whether evaluation suffices — it does, practically and often — but whether we are clear-eyed about what evaluation reveals. A model that passes every benchmark is a model whose outputs land where understanding would land. Treat that achievement as evidence of understanding, and you have made a category error. Treat it as evidence that the geometry faithfully encodes the residue of human understanding, and you have said something both true and useful — useful because it tells you where the model’s reliability comes from (inherited structure) and where it will fail (wherever the inherited structure runs out and new insight is required).

The richness of the vector space. A second move worth making runs against the grain of easy dismissal. Consider what the embedding space actually contains.

A model trained on the full breadth of human text does not merely encode dictionary definitions and encyclopedic facts. It ingests — and geometrically organizes — the entire written fabric of human experience: letters of grief and declarations of love, diagnostic notes and battlefield dispatches, liturgical poetry and earnings calls, the slow accumulation of case law and the compressed fury of political pamphlets. Every context in which a word has been used, every shade of meaning a sentence has carried, every emotional register a paragraph has inhabited — all of this presses structure into the geometry.

Machines do not have emotions. They have something else, something without precedent: the complete written record of what emotions do to language. The vector for “grief” sits where it does not because the model has grieved but because every elegy, every condolence letter, every clinical description of bereavement, every novel that has tried to render loss on the page has tugged that vector into position. Context, usage, connotation, the way “grief” behaves differently in a psalm than in a case report — all encoded, all geometrically available. If a thing, in the sense this essay has developed, is a unity of explanatory properties grasped through their mutual relations, then the model’s representation of “grief” constitutes something like a thing at the level of language itself: a unity of usage-properties, co-defined across millions of contexts, forming a closed web of mutual specification.

Not felt grief. Not understood grief. But the full relational structure of grief-as-it-has-been-written, organized with a comprehensiveness no single human reader could achieve. The philosopher who has read three hundred texts on grief knows grief’s conceptual neighborhood intimately; the model has internalized three hundred million such texts and mapped the neighborhood at a resolution beyond any individual’s reach.

Push further. Multimodal models bind the written fabric to the visual and auditory record — photographs of mourning, the acoustic signature of a breaking voice, the compositional conventions of memorial architecture. Across modalities, the relational web thickens. What emerges is not understanding in the sense this essay has carefully defined, but it is not nothing, either. It is the most complete map of human experiential structure ever assembled, and its completeness matters. Where a human knower grasps a thing’s unity through a handful of explanatory relations drawn from limited experience, the model’s geometry encodes the full distributional structure of that thing across the entire written and visual record of the species.

The implication is not that the map becomes the territory at sufficient resolution. It is that the map is far richer than we have credited — rich enough to simulate understanding across an extraordinary range of contexts, rich enough to surface relational structures that human knowers, limited by the narrowness of individual experience, might miss. A model cannot grieve. It can, plausibly, identify patterns in the language of grief that no grieving person has noticed, precisely because no grieving person has read everything ever written about grief and held the relational structure in a single, navigable space. The spoils of billions of insights, compressed into geometry, yield combinatorial possibilities that the original insighters never explored.

Respecting the gap between optimization and understanding, then, need not mean underestimating what optimization achieves. The gap is real, the difference in kind genuine. And the artifact on the optimization side of that gap — this vast, intricate, cross-modal geometry of human experience — is extraordinary enough to demand serious philosophical attention in its own right, not merely as a deficient approximation of something better.

Architectural honesty. Attention over sequences — the transformer’s core mechanism — weights relevance. As a cognitive operation, relevance-weighting corresponds roughly to the first level of knowing: selecting which data to attend to. Grasping why selected data cohere (the second level) and verifying the grasp against fulfilled conditions (the third) require mechanisms the architecture does not yet possess. Knowing what the architecture lacks is a prerequisite for knowing what to build next — but knowing what the architecture already encodes, the sheer density of human experiential structure compressed into its geometry, is equally prerequisite. The next architecture will not start from scratch. It will start from the richest map of human knowing ever drawn.

Complementarity, not replacement. Pair what models do — navigate the accumulated geometry of human understanding at superhuman speed, surfacing relational structures across the full written record — with what humans do: verify, judge, ask new questions, power the spiral from within. That pairing reflects a structural difference in kind between optimization’s products and understanding’s achievements. Durable precisely because it rests on a difference that scaling alone cannot dissolve, this complementarity is also generative: the model’s geometry reveals patterns the human knower can then investigate, judge, and integrate into the self-correcting spiral. The map suggests where to survey next. The surveyor confirms what the map only indicates. Between them, territory that neither could cover alone comes into view.

Coda

One phrase from Lonergan I cannot improve on: the human mind operates under an unrestricted desire to know. Not a desire to predict the next token, not a desire to minimize a loss function — a desire that, confronted with any answer, immediately generates the further question. Why this? Why here? Why now? What else?

Magnificent artifacts of restricted desire, embedding spaces. Shaped by specific objectives, faithful within those restrictions to the structure of human knowledge, extraordinary in their fidelity. And silent. They do not ask further questions, do not wonder whether their geometry is correct, do not feel the dissatisfaction that drives a scientist back to the lab or a reader back to the paragraph.

That dissatisfaction — the restless, self-correcting eros of the mind toward what is — is understanding. Encoding it in a vector, however high the dimension, has so far eluded us. Saying so with precision is where serious inquiry into the nature of machine intelligence begins.

---

Taylor Black is Founding Director of the Leonum Institute for AI & Emerging Technologies at The Catholic University of America and Director of AI & Venture Ecosystems in Microsoft’s Office of the CTO. Poured Brews explores AI, among many other things, through the Catholic intellectual tradition.

1. The account of things, insight, and judgment I draw on throughout comes from Lonergan’s Insight: A Study of Human Understanding (1957), especially Chapters 8–12. It rewards — and demands — sustained attention.