“Agent OS” is now attached to products that share almost nothing: a terminal wrapper, a workflow builder, a hosted agent platform, a kanban board with an AI panel. The phrase is doing marketing work, which is a shame, because underneath it there is a real and specific idea with a published design.
The idea is that running many LLM agents concurrently is a resource-management problem, and resource management is what operating systems do. This article covers where the term comes from, the architecture the research proposes, which parts of it exist in tools you can use today, and how to tell a genuine claim from a badge.
TL;DR
The term traces to AIOS, a Rutgers paper published at COLM 2025 that isolates LLM-specific services into a kernel providing scheduling, context management, memory management, storage management, tool management, and access control, and reports up to 2.1x faster execution for agents built on various frameworks. Most products called an agent OS implement a slice of that and skip the kernel. The useful question is not whether something is an OS; it is which layer it occupies and which resource it actually manages.
Where the term comes from
The reference point is AIOS: LLM Agent Operating System by Mei and colleagues, first posted in March 2024 and published as a full paper at COLM 2025. The problem statement is concrete. Unrestricted access to LLM or tool resources leads to inefficient and potentially harmful allocation, and the absence of scheduling and resource management in current agent designs hinders concurrent processing and limits system efficiency.
The proposed answer follows the classic OS move: isolate shared services out of applications and into a kernel. AIOS separates resources and LLM-specific services from agent applications into an AIOS kernel, which provides fundamental services for runtime agents, with an SDK on top. The reported result is up to 2.1x faster execution when serving agents built by various agent frameworks. The implementation is public at agiresearch/AIOS.
The analogy holds because the pressures rhyme. Many processes, one scarce resource, no coordination, and the result is contention. Substitute agents for processes and a model endpoint for a CPU and the shape is familiar.
The architecture
AIOS is described as three tiers. The application layer holds agents built against the SDK. The kernel layer holds the LLM kernel beside the traditional OS kernel. The hardware layer is the CPU, GPU, and storage underneath.
The kernel is where the design lives, and its modules map to recognizable OS concepts:
| Module | Responsibility | OS analogue |
|---|---|---|
| Agent scheduler | Schedules and prioritizes agent requests to optimize use of the model | Process scheduler |
| Context manager | Snapshots and restores intermediate generation state, manages the context window | Context switching |
| Memory manager | Provides short-term memory for each agent’s interaction log | RAM management |
| Storage manager | Persists state beyond the session | Filesystem |
| Tool manager | Manages access to external tools and APIs | Device drivers |
| Access manager | Enforces what an agent may reach | Permissions |
Read that table next to a real multi-agent setup and the gaps are obvious. Most teams have no scheduler: agents start when a person starts them. Most have no access manager: the agent inherits whatever the shell had. Context and memory are handled per runtime, differently, with no shared view.
What exists today, and at which layer
Three distinct layers get called an agent OS, and separating them makes evaluation possible.
The runtime layer is Claude Code, Codex, Gemini CLI, and their peers. Each manages its own context window, its own tool calls, and its own session memory. Within one runtime this works well. Across runtimes nothing is shared, which is why context does not travel when you switch a task between agents.
The kernel layer is what AIOS proposes and what almost nobody ships in production. Serving stacks solve pieces of it for throughput reasons, and scheduling across agents from different vendors, with one context manager and one access manager, remains largely a research artifact.
The management layer is where tasks, assignment, isolation, review, and the record live. This is the layer that exists commercially, and the layer most products labelled an agent OS actually occupy. Sharkly sits here by design: it does not replace a runtime and does not schedule tokens. It holds the Task, dispatches a run to an eligible Runtime on a connected Computer, gives the run an isolated worktree, and returns results, blockers, and evidence to the Task for human review.
Some kernel concerns do show up at the management layer, which is the interesting part. Access control appears as scoped credentials, short-lived install tokens, and agents that never authenticate as you. Storage appears as the execution log and persisted agent comments. Scheduling appears as concurrency limits per Computer and a waiting state when a required local directory is busy. These are the OS ideas arriving through the door that was open.
Use cases where the framing earns its keep
Running agents from several vendors at once. The moment two runtimes are in play, somebody has to decide which gets the task and what happens when both finish, and routing becomes a real decision.
Contention on a shared resource. A repository, a database, a staging environment, or a rate-limited API. Isolation per run solves the first case; the rest need a queue and a waiting state you can see.
Unattended operation. Anything running while nobody watches needs the equivalent of process states and a way to surface a stuck one, which is the practical half of unattended runs.
Governance. “What could this agent reach and what did it do” is an access-manager question and an audit question at the same time, and coding agent permissions covers the three layers it spans.
How to read an agent OS claim
Four questions cut through most of the marketing.
Which resource does it actually manage? A genuine claim names one: model capacity, context, tool access, or the working directory. A vague claim names none.
Does it schedule, or does it launch? Starting ten agents is not scheduling. Scheduling means deciding what runs next when capacity is scarce, and having a queue with visible states.
What happens when two agents want the same thing? If the answer is “last write wins,” there is no kernel, whatever the homepage says.
Is it neutral about runtimes? An OS that runs one vendor’s binaries is that vendor’s product. Neutrality is the whole value of the abstraction, and it is the provider-neutral argument in different words.
Where this goes
The honest position is that the OS framing is directionally right and premature as a product category. The research has the architecture and the working implementation. Production tooling has the management layer and a handful of kernel-shaped features borrowed into it. The gap between them is real work, and the interesting question for the next year is whether kernel services arrive as infrastructure below the management layer or get absorbed into it one feature at a time.
For a team deciding what to use this quarter, the framing is still useful as a checklist rather than a purchase. Ask which layer a tool occupies, make sure you have exactly one management layer, and let the runtimes be replaceable. That is the part of the OS analogy that pays now: the thing that lasts is the interface, not the implementation behind it.
FAQ
What is an agent OS?
A layer that manages shared resources for multiple LLM agents: scheduling model access, handling context and memory, mediating tool use, and enforcing permissions. The reference design is the AIOS kernel described in arXiv 2403.16971.
Is an agent OS the same as an agent framework?
No. A framework like LangGraph or the Agents SDK helps you build one agent’s logic. An OS-style layer manages many agents competing for the same resources, and AIOS explicitly serves agents built by various frameworks.
Is Sharkly an agent OS?
No, and the distinction matters. Sharkly is a management layer above the runtimes you already run: tasks, assignment, isolated execution, review, and the record. It does not schedule model capacity or manage context windows, which is what the kernel in the research handles.
Do I need an agent OS today?
Probably not as a product. You need the specific functions: isolation per run, a visible queue, scoped credentials, and an attention signal. Get those from your management layer and treat the OS label as a description of ambition.
What should I read first?
The AIOS paper for the architecture, then look at your own setup and mark which of the six kernel modules you have improvised. Most teams find they have storage and nothing else. Getting started with Sharkly covers the management-layer half in an afternoon.
