Who Decides When the Agent Acts?
On Engineering Sophistication and the Question of Agency
This is the second of three essays responding to Satya Nadella's year-end reflection, "Looking Ahead to 2026." The first addressed his call for a new theory of mind that accounts for humans equipped with cognitive tools. This essay takes up his observation that we are shifting from models to systems—and the question of agency that shift provokes. The third will examine what "societal permission" actually requires.
Satya Nadella’s year-end reflection identified a shift underway in how we deploy artificial intelligence. Models, he observed, will give way to systems—rich scaffolds that orchestrate multiple models and agents, account for memory and entitlements, and enable sophisticated tool use. 2026 will see this come to fruition at scale. Models in isolation remain impressive but inert; they generate outputs without affecting anything beyond the screen. Systems connected to databases, APIs, and real-world processes can actually change things. The engineering challenge Nadella identifies is genuine: building infrastructure robust enough to enable AI capabilities to operate in the world while managing their jagged edges.
But beneath the architecture diagrams and orchestration layers, a question persists that engineering alone cannot resolve. When the system acts—when it queries a database, sends a message, modifies a file, triggers a downstream process—who decides what should happen? Not who designed the system or deployed it or typed the initial prompt, but who decides in the operative sense: who deliberates among possibilities, weighs them against values, commits to one course, and accepts responsibility for what follows? The engineering sophistication Nadella describes extends the capability for action dramatically while leaving this question not so much answered as unasked.
Understanding why the question matters requires clarity about what action actually involves. There is a tendency, reinforced by computational metaphors, to think of action as output plus execution—a system generates a recommendation, triggers a process, and something happens in the world. But action in the full sense encompasses operations that this picture omits.
When a person acts, she deliberates: holding possibilities in view, imagining consequences, considering what each would mean. She evaluates: measuring options against values she holds, goods she pursues, commitments she has made to others, and to herself. She decides: selecting one path, which means setting aside others, accepting that this choice forecloses alternatives. And she commits: taking the action as her own, owning what follows, standing answerable for the outcome. These operations constitute the fourth level of human cognition—what follows when experience has been gathered, understanding achieved, and judgment rendered about what is true. Having grasped the situation, the person faces the question of what to do about it. The question is not merely cognitive but existential; in deciding, she constitutes herself as someone who chose this rather than that.
A system that processes inputs, generates an action representation, and triggers execution has not performed these operations. It has not deliberated, because deliberation requires a subject holding possibilities in view, and the system is not a subject. It has not evaluated, because evaluation requires values against which to measure, and the system has no values—only objective functions imposed from outside. It has not decided, because decision requires commitment, and commitment requires someone who commits. The processing may be extraordinarily sophisticated. The execution may be consequential. But the operations constitutive of action—the operations that make action someone’s action—have not occurred. They cannot occur, because they require what no architecture provides: a subject who holds values and bears responsibility.
Nadella describes systems that orchestrate multiple models and agents, and the architectural logic is compelling. Different models excel at different tasks; orchestration layers can route queries appropriately, aggregate outputs intelligently, manage complex workflows that no single model could handle. Capabilities multiply through combination. But what orchestration cannot accomplish is the combination of lower-level cognitive operations into higher-level ones.
Judgment—the third level of cognition, where understanding submits itself to the question of truth—requires grasping that evidence suffices for affirmation. This grasp is not a computation but a reflective act: the mind turning back on its own operations, recognizing that the conditions for warranted assertion have been met. No amount of aggregated pattern-matching produces this recognition. You can ensemble a thousand models, weight their outputs by confidence scores, synthesize their predictions through learned meta-models. What you have achieved is very sophisticated processing at levels one and two. What you have not achieved is judgment, because judgment answers a different question. Processing asks what patterns appear; judgment asks whether the patterns are true. These questions operate in different registers, and no volume of the first yields an answer to the second.
Decision—the fourth level—requires a subject weighing values and committing to action. Orchestrating execution chains, however elaborate, does not produce such a subject. The chain executes according to its design; nothing in the chain decides according to its values. Add persistent memory, and the chain can maintain context across sessions. Add entitlements and the chain operates within permitted bounds. Add tool access, and the chain can affect external systems. Each addition extends capability genuinely. None creates an agent in the sense that matters: someone who acts and is accountable for acting, someone from whom the action flows as theirs.
The “jagged edges” that Nadella acknowledges—the unpredictable failures where models confidently produce nonsense—are often discussed as capability gaps awaiting patches. Some are. But others mark categorical boundaries that no patch can smooth. The edge where pattern-matching ends and judgment begins is not jagged; it is a cliff face. No path leads across because the terrain on the other side requires operations that processing cannot perform. Systems can create the appearance of smooth passage through redundancy and verification and constraint. What they cannot do is actually cross, because crossing would require being the kind of thing that judges and decides.
Consider more closely the capabilities Nadella specifies: memory, entitlements, and tool use. Each extends something real and valuable. Understanding what each extends—and what it does not—illuminates where the boundaries actually lie.
Memory extends the first level of cognition: data persistence, context retrieval, pattern continuity across sessions. A system with memory can maintain coherent engagement over time, reference prior interactions, and build on accumulated context rather than starting fresh with each query. This is genuine amplification, and its practical value is immense. But memory in this sense is not experience in the full sense. Experience involves a subject attending to data, and attending is an act of consciousness. Memory systems store and retrieve representations; they do not attend to them.1 The continuity they provide is informational rather than existential—a record of what happened, not the lived presence of someone to whom it happened. The system with memory knows what occurred in past sessions, the way a database knows what it contains: the data is there, retrievable, but no one is home.
Entitlements constrain what the system may access, modify, or trigger. This is a crucial governance function—without it, capable systems become dangerous systems—and it participates in what might be called the good of order: the institutional arrangements that channel capabilities toward acceptable ends. But entitlements are external constraints substituting for internal evaluation. The well-entitled system acts within permitted scope because the scope has been defined and enforced from outside, not because the system has weighed the merits of restraint. The entitlement framework does the work that judgment would do if judgment were available; its necessity is a symptom of judgment’s absence.
Tool use extends execution capability, enabling models to reach beyond text generation into the world of databases, APIs, and file systems. The extension is dramatic: a model with tools can query, write, send, modify, and deploy. But tool use is still execution. The model processes inputs, generates a representation of an action, and triggers a tool. What happens next may be significant—messages sent, records updated, processes initiated—but the operation remains at the level of sophisticated automation. The tools provide hands; nothing in the architecture provides a will that could direct them. The tool-using model is like a factory robot with an expanded range of motion: it can do more things, but it still does only what its programming specifies. The programming has gotten more abstract, more responsive to context, more capable of handling novelty. It has not become someone who decides.
There is a useful way of thinking about what technology serves that distinguishes three levels of human good. At the base, particular goods satisfy individual needs: this meal, that shelter, the answer to my question now. Above them, the good of order comprises institutions and arrangements ensuring regular access to particular goods: agricultural systems that make meals available reliably, housing markets that provide shelter at scale, search engines that answer questions on demand. At the apex, terminal values are what are genuinely worthwhile—the standards against which social arrangements should be assessed, the ends that give the whole structure its point.
AI systems produce particular goods constantly: answers, summaries, drafts, translations, analyses. They also increasingly constitute elements of the good of order—infrastructure that makes information reliably accessible, communication routinely possible, and services consistently available. A well-designed AI system becomes part of the institutional fabric, as much as roads or power grids or financial networks. The integration is not dystopian; it is the ordinary way that useful technologies find their place.
But the good of order is not self-justifying. Efficient infrastructure still requires the question: efficient toward what? Reliable systems still require: reliable for whom? The answers must come from terminal values—from judgment about what human flourishing actually requires and decision about which goods are worth pursuing. These operations cannot be automated because they are the operations that determine what automation should serve. Building elaborate infrastructure while neglecting the evaluative questions is like constructing highways without asking where they lead. The engineering may be excellent, the traffic flow optimized, the maintenance impeccable. But highways are for going somewhere. Infrastructure that perfects movement while obscuring destination can take us anywhere at all, including places we should never go.
The industry increasingly describes AI systems as “agents,” and the language deserves scrutiny. An agent, properly understood, is one who acts—not merely one who causes effects, but one who decides and bears responsibility for deciding. The etymology traces to agere: to do, to drive, to conduct. Agency implies that responsibility has somewhere to land. The agent is the origin of the action in a sense that permits praise or blame, justification, or regret. The agent is the one who can be asked, “Why did you do that?” and is expected to own the answer.
Current AI “agents” are models with tool access and execution loops. They process queries, generate plans, take steps, observe outcomes, and adjust accordingly. The architecture resembles agency in its outward form: goals and actions and adaptation, the appearance of pursuing ends through chosen means. But resemblance is not identity. The model processes without deciding. The loop executes without committing. No subject inhabits the architecture who could be asked why it acted and could genuinely answer. The system has no “why” of its own; it has only the objectives it was given and the patterns it learned.
This matters beyond semantics. When an AI “agent” takes action with real consequences—when it sends a message that damages a relationship, makes a trade that loses money, triggers a process that cannot be undone—who bears responsibility? Not the model, which only processed. Not the execution loop, which only ran. The designer? Too distant from the particular action, which emerged from a context the designer never saw. The deployer? Too removed from the inner workings, which operated according to dynamics the deployer may not understand. The user? Perhaps, but the framing of “use” obscures how much autonomy the system exercised, how little the user controlled or even observed the intermediate steps.
Responsibility diffuses until it evaporates. Everyone can point elsewhere; no one clearly bears the weight. This is not a governance problem awaiting a policy solution. It is a structural feature of systems that execute without deciding. Where there is no deciding subject, there is no responsibility—only causation, only effects, only things that happened without anyone having done them.
If systems cannot decide, and if consequential action requires decision, then human judgment must remain structurally central—not an optional add-on, not a governance formality, but a constitutive element without which the system does not genuinely act. The practical question becomes: where in the architecture does human judgment actually enter?
Current designs often position it peripherally. The human approves outputs at the end of execution chains, provides oversight when metrics flag anomalies, and intervenes when things go wrong. This architecture treats human judgment as exception handling rather than as decision-making. The system operates; the human monitors. The system proposes; the human ratifies. The workflow is efficient precisely because the human mostly stays out of the way.
A different architecture would position human judgment centrally—not as checkpoint but as origin. The system gathers data, surfaces considerations, presents the material that judgment requires. The human deliberates, evaluates, decides. The system then extends that decision into execution, applying capability to carry out what the human determined should be done. The difference is not cosmetic. In the peripheral model, the system acts, and humans constrain. In the central model, humans decide, and systems extend. The first borrows the appearance of human agency while hollowing out its substance. The second preserves agency’s structure while amplifying its reach.
The practical challenge is that central architectures are slower, more effortful, and less amenable to automation’s efficiencies. They require humans to actually engage rather than passively approve. They interrupt workflows that would otherwise proceed without friction. Every design decision trades off capability against agency, efficiency against responsibility, what the system could do against what the human should decide. There is no architecture that maximizes all values simultaneously. The question is which values we optimize for when they conflict—and that question, too, requires human judgment to answer.
What makes this urgent is that the erosion of agency happens incrementally, through convenience rather than coercion. Each individual delegation seems reasonable: why deliberate when the system handles it well enough? Why evaluate when the outputs are good enough? Why decide when the default works? The delegations accumulate until the human’s role has become nominal—a signature on forms the system prepared, approval of recommendations the system generated, ratification of decisions the system already made. No single step felt like surrender. The destination is surrender nonetheless.
Nadella is right that we need engineering sophistication to extract real value from AI in the real world. The technical challenges he names are genuine: orchestration across models, persistent memory, entitlement frameworks, robust tool integration. Solving these challenges matters. But engineering sophistication fully conceived extends beyond the technical.
It includes cognitional sophistication: understanding which operations AI systems can perform and which remain constitutively human. Building without this understanding means building blind—not knowing where the boundaries lie, mistaking execution for decision, and conflating pattern matching with judgment. The jagged edges become invisible until someone falls off a cliff that no one realized was there.
It includes design wisdom: shaping architectures that support human agency rather than simulating or supplanting it. This is more difficult than adding capability, because capability compounds, whereas agency must be cultivated. The path of least resistance automates decision away. The wiser path preserves decision’s structure while making it better informed, more consequential, and more clearly the human’s own.
It includes evaluative honesty: measuring not only whether systems perform but also whether humans using them flourish. Does this system make its users more responsible agents in the world, more attentive to what matters, more discerning about what is true, more deliberate about what to do? If the system optimizes its metrics while its users atrophy as deciders, we have succeeded at the wrong task.
These measures do not yet exist in any systematic form. Developing them is itself an engineering challenge, one that demands the cognitional and design sophistication it aims to measure. The circularity is not vicious but developmental: we build what we understand, building reveals what we missed, and understanding deepens through the building. But the development requires intention. Left to its own momentum, optimization finds what is measurable, and what matters most—agency, responsibility, the human capacity to decide—resists easy measurement. The metrics we lack are the ones we most need.
Systems are coming that will act in the world with unprecedented capability and autonomy. Orchestration layers, persistent memory, entitlement frameworks, expansive tool access—the infrastructure is being built rapidly and at scale. The trajectory Nadella traces is not speculation but observation. Whether we find it exhilarating or alarming, it is happening.
What remains undetermined is whether these systems will be designed to preserve human agency at their center or to simulate it at their periphery. Both architectures are technically feasible. Both will be built, likely in competition with one another. The question is which we will recognize as genuinely sophisticated—which we will fund, adopt, integrate, and normalize.
Execution without decision is not action but automation. Automation has its place; no one wants to deliberate over every database query or agonize over every API call. But automation that absorbs decisions rather than executing them produces systems that act without anyone acting, that affect the world without anyone being responsible for the effects. The technical capability to do this exists and is expanding. The wisdom not to do it, or to do it only where appropriate, requires the operations that AI systems lack: judgment about what is fitting, decision about what we will build, commitment to the values that make us more than our tools.
The engineering challenge is real. So is the human one. Sophistication worthy of the name must encompass both.
Taylor Black writes about AI, human flourishing, and the Catholic intellectual tradition. He serves as head of AI & venture ecosystems in Microsoft’s Office of the CTO and is Founding Director of the Institute for AI & Emerging Technologies at Catholic University of America.
“Attention” in AI refers to a mechanism that allows models to dynamically weight the relevance of different parts of an input when producing each part of an output—essentially letting the model “focus” on what matters most for the task at hand.
The core intuition: When you read the sentence “The cat sat on the mat because it was tired,” understanding what “it” refers to requires attending back to “cat” rather than “mat.” Attention mechanisms give neural networks this same capacity to selectively emphasize relevant context.
Mechanically, attention works through three learned projections of the input: queries (what am I looking for?), keys (what do I contain?), and values (what information do I carry?). The model computes similarity scores between queries and keys, then uses those scores to create weighted combinations of values. High similarity means “pay more attention here.”
Why it transformed AI: Before attention, sequence models like RNNs processed inputs step-by-step, creating a bottleneck where distant context had to squeeze through intermediate states. Attention allows direct connections between any positions—a word at the end of a document can directly attend to a word at the beginning. The 2017 “Attention Is All You Need” paper showed you could build entire architectures (transformers) on this principle alone, abandoning recurrence entirely.
Self-attention specifically means each position in a sequence attends to all other positions in that same sequence, allowing the model to build rich contextual representations. This is the foundation of models like GPT, BERT, and the architecture underlying this conversation.
The metaphor of “attention” is apt but imperfect—it’s less like human focal attention and more like a learned, parallelized relevance-weighting system operating across all positions simultaneously.