AgentX — Agentic AI Workspace

What it is

AgentX is an agentic AI workspace: a single ReAct agent — deliberately not a supervisor routing between specialised subgraphs — with its own Python sandbox, a shell, a skills library it loads on demand, document retrieval, optional MCP tool connections and a persistent memory of the user. Given a task, it does the work and streams every step back as it happens.

The backend is roughly 31,000 lines of Python across 219 files, with 167 tests and 185 commits. An earlier version did use a supervisor routing between seven subgraphs via an intent classifier; it was replaced because a single agent with a good tool list did the same job with far less machinery.

Agent architecture

The compiled graph has two nodes: an agent node and a tool node, with a conditional edge looping between them until the agent stops calling tools.

• The agent node builds one flat tool list per turn — always-on sandbox tools, skill-discovery tools, the user’s enabled native tools, and whatever MCP tools are currently connected — then binds them all to the LLM.
• The tool node executes every tool call from the last message in parallel via asyncio.gather, wrapping each in pre/post hooks, timing and a result cache.
• The LLM client is a cached singleton rather than being rebuilt each turn, and checkpointing is Redis-backed when available, falling back to in-memory.

Generation is not tied to the HTTP request

The most consequential design decision in the project. Originally, the code that drove the agent and the code that served the HTTP response were the same coroutine — so when a browser disconnected, the framework cancelled that coroutine and the cancellation propagated straight into the running agent. Reloading the page killed your generation.

The fix was to separate running the agent from watching the agent run. A small in-process registry launches agent execution as a detached background task and fans its events out to any number of subscriber queues, replaying everything already published to a late subscriber. The HTTP handler that starts a turn is now just the first viewer — if it disconnects, the task keeps running. A reload, or even a second tab, attaches a fresh viewer to the same running turn and catches up seamlessly.

The Stop button had to be rebuilt too, since it used to work by aborting the fetch and relying on exactly the cancellation behaviour that was removed. It is now an explicit cancel against the task.

Sandbox and security

The agent runs user-authored Python and shell code, so isolation is the load-bearing property of the whole system. It was hardened iteratively, driven by adversarial testing of the running system rather than by reading the code.

• Every user gets a persistent workspace with a lazily-created virtualenv shared across their conversations, while uploads, outputs and working files are scoped one level deeper, per conversation — isolation that turned out to be load-bearing rather than cosmetic.
• Sandboxed code runs under a dedicated non-root UID with all Linux capabilities dropped and only SETUID, SETGID, CHOWN and DAC_OVERRIDE added back — four independent capabilities that each had to be identified separately.
• An optional remote execution mode streams code to a separate, network-isolated host and runs it in a gVisor container per execution: kernel-level isolation on a different machine, not a subprocess boundary. It fails closed — an unreachable sandbox host returns an error rather than silently running the code locally.
• A single centralized pre-tool hook performs path and command checks, instead of each tool re-implementing its own guards.

Cost control

Per-turn spend is computed from actual token counts rather than estimates, and checked against a per-user cap. A grace buffer lets an in-flight turn finish rather than being cut off mid-answer — it is the next turn that gets blocked.

The interesting bug here was a time-of-check-to-time-of-use race: the check and the charge were separated by the entire duration of a turn, ten to ninety seconds, with nothing limiting a user to one turn at a time. Several concurrent turns would each read the same pre-deduction balance and each pass the check. Firing five genuinely simultaneous requests confirmed all five ran. A Redis SET NX lock, acquired atomically at the same point as the credit check and released in the turn’s finally block, brought that to exactly one proceeding and four rejected.

The lock fails open if Redis is unreachable — losing double-submit protection during an outage is acceptable; taking all of chat down with it is not.

Retrieval, skills and voice

• Agentic RAG: documents are parsed for tables and images, chunked and embedded into Qdrant. Uploads are asynchronous — the route returns a job id immediately and reports progress through Redis. Vector search and web search run in parallel so one source failing does not kill retrieval.
• Skills are markdown manuals the agent discovers and loads on demand rather than being force-fed into every system prompt: a cheap listing call returns names and descriptions, and only the chosen skill’s full body is fetched.
• MCP tools from externally connected servers are converted to LangChain tools dynamically, with connections cached to avoid a network round-trip on every turn.
• A speech-to-speech voice mode runs on the same agent graph as text chat, with the same tools and retrieval, and is provider-swappable on both the speech-to-text and text-to-speech legs.

Infrastructure

Two purpose-built EC2 boxes and a managed data layer, with no container orchestration platform. The backend runs in Docker behind nginx. A second box holds the LLM gateway, a PDF parser service and the sandbox execution service, behind a shared API-key gate and never exposed to the public internet.

Continuous deployment runs the full test suite on every push, and the deploy workflow only triggers after tests pass. It rsyncs to a fresh versioned directory, carries the environment file forward, builds a new image, stops and renames the old container rather than deleting it, starts the new one and health-checks it — rolling back automatically on failure. Every previously deployed container is kept, stopped, for instant rollback.