AI Engineeringllm

From "Just a Chatbot" to a Full AI System: The Layers Explained

P&C

Prompt & Circuit Editorial

May 15, 2026 · 12 Min Read

From "Just a Chatbot" to a Full AI System: The Layers Explained

More than a text box

Ask most people to picture "AI" and they will describe a text box that answers questions. Production AI systems look very different. The model is still at the centre, but around it sit layers for retrieving data, taking actions, connecting to other systems and keeping long-running work reliable.

None of these layers is a smarter model. Each exists because the layer beneath it has a specific limitation. Understanding those limitations is the fastest way to decide which layers your product actually needs.

Layer 1: The large language model

What it is: A neural network trained on very large amounts of text to predict the next token. That simple objective produces strong abilities in language, summarisation, coding and reasoning.

What it cannot do on its own:

  • Know anything after its training data, or anything private to your organisation.
  • Guarantee facts. Output is sampled to be plausible, which is why models sometimes produce confident, wrong statements ("hallucinations").
  • Act. A bare model produces text. It cannot query a database, call an API or send an email.
  • Remember. Each request is independent. Anything the model should "remember" has to be sent again in the context window, which is large but finite.

Key point: LLMs are excellent at language and pattern-matching, and unreliable as a source of ground truth or state.

Layer 2: Retrieval-augmented generation (RAG)

What it is: Before the model answers, the system searches a knowledge source (a vector database, a search index, or both) and places the most relevant passages in the prompt. The technique was named in a 2020 paper by Lewis et al., which paired a generator with a retriever over an indexed document store.

What it adds:

  • Answers grounded in current, private data without retraining the model.
  • The ability to cite where an answer came from, so users can check it.
  • Effective access to a knowledge base far larger than any context window, because only the relevant slices are sent each time.

What it costs: Retrieval quality becomes the bottleneck. If the wrong passages come back, the model will confidently answer from them. Chunking, ranking and evaluation become ongoing work. And RAG still only reads; it cannot change anything.

Layer 3: Tools and agents

What it is: The model is given tools (functions it can request, such as "search orders" or "create ticket") and runs in a loop: decide on an action, execute it, observe the result, decide again.

Anthropic's widely cited guide to building agents draws a useful distinction:

  • Workflows are systems where "LLMs and tools are orchestrated through predefined code paths."
  • Agents are systems where "LLMs dynamically direct their own processes and tool usage."

Both build on what the guide calls the augmented LLM: "an LLM enhanced with augmentations such as retrieval, tools, and memory."

Example: Asked to plan a weekend trip, an agent might check a weather API, search for hotels, compare prices and draft an itinerary, choosing each step based on what the previous one returned.

What it costs: Every tool is a new failure mode and a new security surface. Agents can loop, call the wrong tool, or be manipulated by malicious text in tool output (prompt injection). The same guide advises "finding the simplest solution possible, and only increasing complexity when needed." Many problems that look like they need an agent are better served by a fixed workflow.

Layer 4: The Model Context Protocol (MCP)

Once agents need many tools, a practical problem appears: every AI application has to write a custom integration for every system it connects to.

What it is: The Model Context Protocol is "an open-source standard for connecting AI applications to external systems." Its documentation compares it to a USB-C port: one standard connector instead of a different cable for every device. It is supported by a wide range of clients, including Claude, ChatGPT, Visual Studio Code and Cursor.

How it works:

  • An MCP host (an AI application) creates one MCP client per connection to an MCP server.
  • Servers expose three core primitives: tools (actions to execute), resources (data for context) and prompts (reusable templates).
  • Messages use JSON-RPC 2.0, over standard input/output for local servers or Streamable HTTP for remote ones.

What it is not: MCP is a connectivity protocol, not a runtime or an operating system for agents. Its documentation is explicit that it "focuses solely on the protocol for context exchange" and "does not dictate how AI applications use LLMs or manage the provided context." It does not provide long-term memory, a code sandbox or an observability platform.

What it costs: Each MCP server you connect is code with access to your systems. Vet servers, scope their credentials, and treat their output as untrusted input to the model.

Layer 5: The runtime around it all

The capabilities sometimes wrongly attributed to MCP still matter. They live in the runtime and orchestration layer:

  • Durable state. Long-running tasks need their state saved so they can survive restarts and resume later. LangGraph, for example, does this with checkpointers that persist graph state after each step, backed by a database such as Postgres in production.
  • Human review. Pausing for approval before costly or irreversible actions.
  • Isolation. Running generated code or risky tools in sandboxes with limited permissions.
  • Observability. Traces of every model call, tool call and decision, plus evaluations that catch regressions.
  • Budgets. Limits on steps, tokens, time and spend.

This is where a prototype becomes a system you can operate.

The stack at a glance

+-----------------------------------------------+
|  Runtime: state, review, isolation, tracing   |
+-----------------------------------------------+
|  MCP: standard connections to tools and data  |
+-----------------------------------------------+
|  Agents and workflows: tools in a loop        |
+-----------------------------------------------+
|  RAG: retrieval of relevant knowledge         |
+-----------------------------------------------+
|  LLM: language and reasoning                  |
+-----------------------------------------------+

The layers are not strictly sequential. A system can use MCP without RAG, or RAG without an agent. Think of them as capabilities you add when the previous ones run out.

LayerWhat you gainWhat you must manage
LLMLanguage fluency, reasoningHallucinations, no state, no actions
RAGCurrent and private knowledge, citationsRetrieval quality, freshness, latency
Agents and workflowsActions and multi-step automationLoops, errors, tool security
MCPReusable, standard integrationsServer trust, credentials, permissions
RuntimeReliability, auditability, controlPlatform effort and cost

Who should care

  • Product managers: The stack gives you the right questions. Does this feature need private data (RAG)? Does it need to act (tools)? What happens when it is wrong (review and runtime)?
  • Architects and engineers: Add layers only when a real limitation demands them. Each one adds operational and security work.
  • Anyone explaining AI to customers: A system that retrieves, acts and is supervised is a different product from a chatbot, and should be described and governed as one.

A minimal starting path

  1. Start with a model API and a well-designed prompt. Measure where it falls short.
  2. Add retrieval if answers need private or current data.
  3. Add one or two tools if the system needs to act, starting with a fixed workflow before a free-form agent.
  4. Use MCP to connect to tools and data sources through a standard interface rather than one-off integrations.
  5. Add durable state, human review and tracing before anything reaches production users.

Final thoughts

The evolution from chatbot to AI system is a staircase, and each step was built because the one below kept tripping people up. LLM, RAG, agents, MCP and the runtime each solve a specific problem, and each brings new responsibilities. Build only the steps you need, and build each one properly.

Sources

Tags:llmragai agentsmcparchitecture
Share Article

Responses (0)

Sign in to join the conversation and share your thoughts.

P&C

Prompt & Circuit Editorial

Contributing Editor

Articles on AI infrastructure for cloud architects, written from official vendor sources and checked claim by claim before publishing.