The Agentic AI Stack

Scope

This article is about agentic AI stacks, primarily for business workflows.

It is not about all other types of AI, and it is not about coding agents.

AI is a vast field; I'm focusing purely on AI agents that can be used for business workflows.

Definitions

Now, let's get started:

The Agentic AI Stack Is a Strategic Decision

I'm a fan of getting the full potential out of AI. When I thought about building an agentic AI stack for my consulting business, it was clear to me that I want to own it, not rent it. It's like a house — I want to own an asset, not rent one. That's how I came to open source. But I'm not a dogmatist. And I'm not a large company either.

My motivation for writing this article was to understand what types of agentic AI stacks exist.

Not Everything Fits Everyone

Before deciding on an agentic AI stack, you need to research which types are suitable for your industry, your organization's size, your type of organization, your strategy, your employees and partners, etc.

A custom agentic AI stack isn't worthwhile for all business areas either. Some areas are so cookie-cutter that you can just buy off-the-shelf solutions and integrate them.

In some industries, however, there is no choice due to regulation, highly sensitive data, or trade secrets.

So, you have to do your own intensive research to determine which AI stack is right for you. Depending on the industry or company size, you'll choose a different AI stack. Of course, cost and how quickly you want to see results also factor into the decision.

In very large corporations, it's even possible for different departments to have different agentic AI stacks because their requirements vary.

Freedom vs. Discipline

While researching, I realised that in some cases, it's a matter of the tension between freedom vs. discipline. The more freedom I want, the more disciplined I have to be in development and governance.

So, it's a trade-off.

The Agentic AI Stack Determines the Operating Model

After reading this article, it will become clear relatively quickly that the AI stack decisively determines the operating model, organizational structure, and way of working. I'll go into what that means in detail in later articles. Today is about giving you an overview, and you'll very quickly see where the differences lie.

No Vendor Recommendations

I have deliberately omitted the names of existing vendors and solutions. You have to research that yourself. I don't work for any vendor and have no preferences.

I've also intentionally not written any recommendations. It really depends on so many factors. That doesn't all fit into one article; it's something you can only figure out together in several personal conversations.

Mistakes Happen

Errors may have crept in, and I'd appreciate it if you point them out to me. Please help me improve this article. Thank you!

The 6 Architectural Layers of an Agentic AI Stack

Each layer builds on the previous one. Without an understanding of the lower layers, decisions in the upper layers are not well-founded.

1

Infrastructure

Definition and Purpose: The infrastructure layer is the physical or virtual foundation on which all other layers operate. It answers the question: "Where do the GPUs that compute the models run?"

Infrastructure provides computing power, storage, and network connectivity. It's the foundation.

Options:

• Cloud—Hyperscalers (US): Full-stack cloud with GPU instances, globally available
• Cloud—Hyperscalers (EU): EU regions available
• Sovereign Cloud: Data in the EU, GDPR-compliant
• GPU Specialists: GPU-optimized infrastructure, often cheaper
• Serverless GPU: Pay-per-use, no infrastructure management
• Local Hardware: Own servers in a data center
• Own local hardware without servers

Important:

Infrastructure determines data location (on-prem vs. cloud), scalability (cloud bursting), and compliance (GDPR, BaFin, sovereignty). A wrong infrastructure decision is expensive to correct later.

2

Model Serving—How Models Are Hosted

Definition and Purpose: Model serving is the software layer that loads, manages, and serves inference requests for AI models. It's the "brain" of the AI—without model serving, there's no intelligence.

Model serving provides models as a service—with APIs that can be called by agents. It manages model versions, batch sizes, and performance optimization.

Options:

Proprietary

• API: Frontier Models—best performance, no own infrastructure
• Enterprise APIs: Enterprise features, compliance certifications
• Self-Hosted Enterprise Serving: High performance, enterprise support

Open-Source / Open-Weight

• API (hosted): Open-weight models—models are open, hosting by the provider
• Model Marketplaces: Many models, pay-per-use
• Self-Hosted Inference: Full control, own infrastructure

Important:

Model serving determines what intelligence is available, how quickly responses arrive, and where data is processed. Task-Model Matching means that not every task needs the largest and most expensive model. You assign the right model to each task. This reduces costs and improves performance. The prerequisite for this is understanding which task requires which type of model.

3

Agent Runtime

Definition and Purpose: The agent runtime is the execution environment for AI agents. It can consist of the following components, although not all existing agent runtimes cover all features:

• Process Management: Starting, stopping, and persistently running agents. Lifecycle from initialisation to termination.
• State Management: Persisting the state of a running agent across sessions. Checkpointing on interruptions.
• Durable Execution: Persistently saving every execution step. On cancellation, error, or interruption, resume exactly where the agent left off.
• Context Management: Controls what goes into the model's context window. Conversation history, relevant context, summaries.
• Sandboxing: Isolated execution environment per agent. Code execution, file access, and commands without risk to production systems.
• Tool Integration: Registering external systems as callable functions. APIs, databases, file systems, browsers, MCP servers. Credential management and scoped permissions per agent.
• Skill System: Storing reusable solution patterns from past tasks, making them searchable, and applying them to new tasks. The agent learns from its own work.
• Multi-Agent Coordination: Delegation, parallelisation, communication between agents, shared task lists. Subagents for sequential subtasks, agent teams for parallel workstreams.
• Self-Evaluation: Agent defines success criteria, evaluates its own results, and iterates until the criteria are met.
• Message Routing: Communication with users and external systems via various channels.
• Scheduling: Time-based, event-based, or agent-triggered execution.
• Model Routing: Controls which model is used for which agent or task.
• Observability: Tracing, debugging, audit trail of all agent actions.

Options:

Proprietary

• Managed Platforms: Fully managed, enterprise features
• Cloud-Native: Integration into cloud ecosystem
• Enterprise RPA+AI: Existing RPA infrastructure

Open-Source / Hybrid

• Frameworks: Full control, self-hosted
• Low-Code: Visual workflow creation
• Personal Agents: Local installation, IDE integration

Important:

The agent runtime determines how workflows are built, orchestrated, and connected to tools. It is the layer with the highest lock-in risk because the entire execution logic resides here. Changing the runtime usually means rebuilding workflows. Portability is also limited between open-source frameworks.

4

Data & Knowledge

Definition and Purpose: The Data & Knowledge layer provides persistent storage for data and knowledge. It is the "memory" of the system. Without it, agents forget everything after each session.

This layer can include:

• Memory (multi-level): Short-term memory holds the context within a running task. Long-term memory stores knowledge about users, projects, and past interactions across sessions. Procedural memory stores reusable solution patterns.
• Retrieval (RAG): Agents search for relevant knowledge before responding. The trend is moving away from simple vector lookup towards hybrid retrieval, which combines semantic search with structured queries.
• Knowledge Graphs: Model relationships between entities (people, processes, systems, documents). Enable multi-hop reasoning. Standard RAG alone cannot do this.
• Data Ingestion: Parsing, chunking, extracting metadata, embedding, and indexing documents. Without an ingestion pipeline, the knowledge layer is empty. The quality of ingestion determines the quality of all answers.
• Data Freshness: Knowledge becomes outdated. Ingestion pipelines must run continuously, not just once. Outdated or contradictory knowledge in the system is worse than no knowledge at all.

Options:

The following technologies belong to the agent system's actual memory. In addition, AI agents can access any data source via tool integrations (but that would be layer 3). Layer 4 includes:

• Vector Databases (Embeddings for semantic search and RAG)
• Knowledge Graphs (Relationships between entities, multi-hop reasoning)
• Document Stores (Unstructured documents, full-text search)
• In-Memory Stores (Session state, short-term memory, caches)
• Personal Knowledge (Wikis, note-taking systems, Markdown collections)

Important:

Data & Knowledge determines what AI agents know. The quality of this layer limits the quality of all agent responses—regardless of how good the model or runtime is. Bad data in a good runtime with a good model produces bad results. Without this layer, AI agents start from scratch with every request.

The difference between this layer and database queries via tool calls from layer 3 deserves its own article.

5

Governance & Control

Definition and Purpose: The Governance & Control layer enforces rules, policies, and compliance requirements. It ensures that AI agents operate within defined boundaries.

It includes:

• Policy Enforcement: Machine-readable rules (e.g., no PII data without masking, no actions above certain thresholds without approval)
• Access Control: Who can use which AI agents, see which data, and perform which actions (RBAC, ABAC)
• Data Classification: Which data may enter agent contexts and which may not. Sensitivity labels
• Input/Output Guardrails: Prompt injection protection, hallucination detection, content filters
• Audit Logging: Complete logging of all agent actions (who, when, what, with what result)
• Compliance Reporting: Reports for authorities and internal stakeholders (BaFin, GDPR, EU AI Act, industry-specific regulations)

Options:

• Policy Engines (Declarative, machine-readable rules as code)
• Identity & Access Management (SSO, RBAC, OAuth2, OIDC, user lifecycle)
• Logging & Monitoring (Logs, metrics, alerting, dashboards)
• AI Observability (Agent-specific tracing, token tracking, latency, cost per agent call)
• Compliance Platforms (GDPR, SOC2, ISO, EU AI Act, industry-specific)
• Cloud-Native Governance (Policy enforcement integrated into cloud infrastructure)
• AI Safety & Guardrails
• Data Classification & DLP
• Model Governance (Model versioning, registry, approval workflows)
• Audit Trail & Forensics
• Cost Governance (Token budgets per agent, team, use case; alerting at thresholds)

Important:

Governance is the difference between demo and production. Without governance, there is no compliance, no audit, no trust. The question is whether you build it yourself or get it from the vendor.

6

Interface

Definition and Purpose: The Interface layer is the access point for users and external systems. It is the "face" of the system, the interface through which people and other systems interact with agents.

An interface enables:

• User Interaction: Chat, portal, mobile app
• System Integration: APIs for other applications
• Multi-Channel: Slack, Teams, WhatsApp, web, mobile
• Personalisation: User-specific experiences

Options:

• Messaging Platforms (Enterprise/community chat integration)
• Business Messaging (Mobile-first, high reach)
• Web Frameworks (Custom web apps)
• Low-Code Portals (Rapid portal development)
• API Gateways (API management, rate limiting)
• Chatbot Platforms (Visual bot creation)
• Voice Interfaces (Voice control)
• IDE Integration (Developer focus)

Important:

The interface determines user adoption. The best AI agent system is useless if users can't or won't use it.

Interim Summary

The different layers are an abstract way of looking at things that helps to break down "the big unknown topic" into understandable units.

Even if you're not a techie, you can get a decent idea of what each layer is about.

This abstraction helps enormously in understanding the different types of agentic AI stacks.

Here are the ones I have identified. There might be other types. I have limited myself to the most common ones.

Different Types of Agentic AI Stacks

Not every organization has to build all 6 layers itself. Platforms cover different layers. Each type is presented with a complete 6-layer diagram.

The following types exist:

1. Ecosystem-Integrated
2. Managed Control Plane
3. Enterprise Application Suites
4. Infrastructure-Only
5. Managed Execution & Self-hosted Control
6. Self-Built / Open-Source Frameworks

There are also hybrid forms, but the article is already very long, so I will only consider these.

For each type, we'll look at the architectural layers.

Legend:

Type 1: Ecosystem-Integrated Platforms

AI as a feature in the existing ecosystem. Deeply integrated, but only usable within the ecosystem.

This type describes platforms that provide AI for all employees. Type 3 would be more for specific business areas. Some vendors are both.

Governance in Detail

Vendor takes over: Complete DLP, compliance center with templates, audit logs in ecosystem format (for eDiscovery), automatic data classification, integration into ecosystem apps, prompt injection protection, basic output validation.

Organization must do itself: Configure ecosystem-specific policies, manage users in the ecosystem, reporting only possible within the ecosystem framework, assign sensitivity labels to documents. External agents not covered.

Workflow Adaptability & Use Case Freedom

• Workflow Design: Ecosystem paradigm (e.g., Copilot Studio: Power Fx, Google: AppScript)
• Use Case Selection: Only use cases within the ecosystem (Teams, Outlook, Gmail, Docs)
• Integrations: Only ecosystem apps + limited external connectors
• Business Logic: Only if the ecosystem supports it
• Scaling: Ecosystem scales automatically, but licence costs increase linearly
• Business Transformation: Only optimization of existing ecosystem processes, no new business models

Type 2: Managed Control Plane Platforms

Fully managed platforms that provide layers 1, 3, 4, 5, and 6. The organization only decides on layer 2 (model selection) and defines its own workflows.

Governance in Detail

Vendor takes over: RBAC/SSO integration, audit logging infrastructure, basic policies (rate limits, quotas), compliance certification of the platform (SOC2, ISO27001), monitoring dashboard, prompt injection protection, basic output validation.

Organization must do itself: Define own business policies, configure user roles, create compliance reports for authorities (BaFin, GDPR), design human-in-the-loop workflows, define data classification policies.

Workflow Adaptability & Use Case Freedom

• Workflow Design: Workflows must follow the platform paradigm
• Use Case Selection: Any use cases possible, as long as platform models support them
• Integrations: Platform connectors available, custom integrations require platform-specific code
• Business Logic: Complex custom logic only if the platform supports it
• Scaling: Platform scales automatically, you are responsible for cost control
• Business Transformation: Quickly test new use cases, but risk of lock-in if successful

Type 3: Enterprise Application Suites

Complete packages from enterprise vendors. Combine everything from infrastructure to interface in one suite.

Governance in Detail

Vendor takes over: Complete suite-internal governance, integration into suite apps, compliance for suite use cases (certified), audit logs for suite actions, data governance within the suite, prompt injection protection, basic output validation.

Organization must do itself: Configure suite policies, manage users in the suite, governance only within the suite (external agents not covered), reporting only for suite use cases, build hybrid scenarios yourself.

Workflow Adaptability & Use Case Freedom

• Workflow Design: Suite paradigm
• Use Case Selection: Only use cases the suite supports
• Integrations: Deep suite integration, external systems only via suite connectors
• Business Logic: Only if the suite supports it
• Scaling: Suite scales automatically, but licence costs increase sharply
• Business Transformation: Only transformation within the suite framework, no AI-native products outside of it

Type 4: Infrastructure-Only Providers

Only infrastructure and model serving (layers 1-2). The organization builds layers 3-6 itself.

Governance in Detail

Vendor takes over: Only infrastructure security (physical security, network isolation). No governance features, no audit logging, no compliance tools, no prompt protection, no output validation, no human-in-the-loop.

Organization must do itself: Build the entire governance stack, implement model-specific guardrails (e.g., no PII in model requests), handle audit logging yourself (all model calls with user ID, timestamp, input/output hash), create compliance reports yourself, handle prompt injection protection yourself, handle output validation yourself, handle escalation workflows yourself.

Workflow Adaptability & Use Case Freedom

• Workflow Design: Any paradigm, custom code
• Use Case Selection: Any use case is possible, model APIs are generic
• Integrations: Every integration self-built, no vendor connector limits
• Business Logic: No limits, own code between user and model
• Scaling: Provider scales infrastructure, you control the costs
• Business Transformation: Completely new business models, model APIs are a commodity

Type 5: Managed Execution & Self-hosted Control

The model provider offers inference and tool-use-capable execution via an API. The organization builds the entire agent logic on top of this itself: memory, skills, orchestration, scheduling, multi-agent coordination.

Infrastructure (Layer 1) and Model Serving (Layer 2) are with the provider. Layer 3 is shared: execution is with the provider, everything above that is with the company. The organization builds layers 4, 5, and 6 itself.

Governance in Detail

Vendor takes over: Model's own safety guardrails (content filters, built into the model), API-level rate limiting and usage tracking, token consumption data via API dashboard.

Organization must do itself: OPA/Kyverno policies, Keycloak/Auth0, output validation (hallucination detection, content filters, JSON schema checks), ELK/Grafana, audit logging, compliance reports (BaFin-MaRisk, GDPR, GxP), escalation workflows, prompt injection protection.

Workflow Adaptability & Use Case Freedom

• Workflow Design: Any paradigm for the self-built orchestration, but execution patterns are dictated by the vendor
• Use Case Selection: Any use case is possible, as long as it can be mapped via the provider's API
• Integrations: Every integration is self-built
• Business Logic: No platform limits in your own code, but execution capabilities depend on the API provider
• Scaling: Scale your own layers yourself, the provider scales inference, you are responsible for token costs
• Business Transformation: New business models are possible, with a dependency on the execution layer. Differentiation lies in the self-built orchestration and domain knowledge.

Type 6: Self-Built / Open-Source Frameworks

A self-built agentic AI stack, possibly with open-source frameworks that are self-hosted and operated. Full control over all layers, high own effort.

Governance in Detail

Since there is no vendor in Type 6, you only get what open-source frameworks might offer. Therefore: no managed services, no monitoring, no audit logs, no compliance tools, no human-in-the-loop, no guardrails.

Organization must do itself:

• Write OPA/Kyverno policies (e.g., agent may not read PII data without masking)
• Operate Keycloak/Auth0 (SSO, RBAC, user lifecycle)
• Output Validation: Hallucination detection, content filters, JSON schema checks
• Set up an ELK/Grafana stack (logs, metrics, alerting)
• Implement audit logging
• Generate compliance reports yourself (BaFin-MaRisk, GDPR, GxP)
• Code escalation workflows yourself
• Integrate prompt injection protection yourself (Lakera Guard or custom rules)

Workflow Adaptability & Use Case Freedom

• Workflow Design: Any paradigm (state machines, event-driven, multi-agent, custom)
• Use Case Selection: Any use case is possible, no platform restrictions
• Integrations: Every integration is self-built, no vendor limits
• Business Logic: No platform limits, own code, own rules
• Scaling: Must scale yourself (Kubernetes, load balancing), but no vendor cost cap
• Business Transformation: Completely new business models are possible, the platform does not limit you

At a Glance

Conclusion

While working on this article, it became clear to me that AI, at its best, expands business strategy because I can build new business models. But at the same time, business strategy limits AI strategy.

Depending on what my business strategy looks like, I decide against certain types of agentic AI stack.

The decision depends on many factors.

There Is No "One" Operating Model

You've probably also realised while reading that, depending on the stack, you need completely different roles to build or implement it and ultimately operate it. The development is different. The governance part is different. And if you want to build new business models, you might need roles that you haven't even considered yet.

This means: Each type of agentic AI stack has different consequences for the organization. This results in:

• different operating models
• different team compositions
• different ways of working
• different organizational designs

And you will very likely need different AI consultants for each type.