AI Models — An Overview

After researching the various models, I thought: this topic is so trivial, it's not actually worth an article. There are different models for different purposes, you don't have to change them, this is how you call them. Done.

I'm publishing this article anyway, because it proves a point:

Where the real difficulties lie

The real difficulties with agentic AI systems lie in various other areas — and in none of these areas does a US AI giant have more experience than most people who have been working for years.

Just think about it logically: the challenges lie in the following areas:

• Understanding the business
• Simplifying processes
• Improving UI/UX and user centricity
• Changing ways of working
• Breaking down and redesigning organizational structures
• Having data in the right form at the right time in the right place
• Codifying knowledge
• Setting up governance properly
• Building stable yet modular architectures
• Really good classic software engineering
• etc.

Large US AI companies don't have a long tradition in these disciplines either. That's why they're buddying up with other firms, like management consultancies.

A political comment on Germany

And here's my perhaps somewhat political opinion regarding Germany:

We all suffered from the recession of recent years. We need hope and a sense of new beginnings.

When I hear that large, well-known German companies are partnering with large, well-known US AI giants, it leaves me baffled. Why? We have so many top people! We have all the knowledge needed to build really good agentic AI. So let's just do it!

Now that I've discovered how trivial it is to call a model, I seriously wonder what decades of tradition a US AI giant brings to the really important disciplines — which is nothing, because these companies haven't been around that long.

So let's just build it ourselves. Then we'll also have the freedom to build with local AI when it's sensible and important — and that might be the case more often than you think.

Now to the actual topic

This is not an exhaustive list or a technical manual. This article is just intended to provide a rough overview.

For non-techies: You should at least have heard and understood the terms. AI will play a major role in professional life in the future, and you need a certain basic knowledge — just as you once learned what terms like Deployment, CI/CD, or DevOps mean.

For engineers: You won't learn anything new, but the next section might help you get stakeholders back on track. It happens so often in professional life that discussions head in the wrong direction, and then people debate something for weeks that isn't even the point. Often too technical and too far removed from the business. It's very helpful when there are engineers who can rein in the stakeholders and steer them back towards: why are we building this, what's the workflow, and how can we simplify it.

What types of models are there?

LLM

Large Language Models

Large language models, trained on vast text datasets to understand and generate human language.

• Hundreds of billions to trillions of parameters
• General-purpose: Summarization, analysis, conversation, reasoning, code

Use: When the task is broad, open, or complex. Complex reasoning, creative text generation, multi-faceted analysis, orchestration in agentic systems.

Examples: GPT-4o, Claude Sonnet/Opus, Gemini, Llama 4, DeepSeek R1, Mistral Large

SLM

Small Language Models

Small language models with typically 1 to 13 billion parameters.

• Trained on focused, often domain-specific data
• Can be run locally, which simplifies governance and data privacy
• Significantly lower operating costs than LLMs

Use: When the task is narrowly defined and repeatable. Classification, extraction, routing, simple summaries, edge scenarios, regulated industries where compliance and data sovereignty are a priority.

Examples: Mistral 7B, Phi-3, Gemma 2B/7B, Qwen 3 4B

CODE

Code Models

Language models specifically trained or fine-tuned on source code and programming languages.

• Understand syntax, logic, and dependencies across programming languages

Use: Code completion, code generation from natural language, debugging, refactoring, code review, test generation.

Examples: Codestral (Mistral), StarCoder, Code Llama, DeepSeek Coder, GPT-4o (with code focus)

EMB

Embedding Models

Models that convert text, images, or other data into dense numerical vectors (typically 768, 1024, or 1536 dimensions).

• Semantically similar content is located close together in the vector space
• Foundation for Retrieval Augmented Generation (RAG), semantic search, and recommendation systems
• Small, fast, efficient, can be run locally
• Are used together with vector databases

Use: Semantic search, similarity comparisons, RAG pipelines, clustering, anomaly detection, document comparison, recommendation systems.

Examples: OpenAI text-embedding-3, NV-EmbedQA, Cohere Embed, Sentence Transformers (Open Source), Amazon Titan Text Embeddings

IMG

Image Generation Models

Models that generate new images from text descriptions (Text-to-Image) or existing images.

• Technically based on diffusion models or Transformer architectures

Use: Marketing visuals, prototyping, product photography styling, illustration, design concepts.

Examples: Stable Diffusion 3.5, DALL-E 3, Midjourney, Kling 1.6 Pro, Recraft v3, Flux

VLM

Vision Language Models

AI systems that combine image understanding and language processing.

• Architecture: a visual encoder (e.g., ViT or CLIP) extracts image features, a language model (LLM) converts them into text
• Can interpret images, describe them, and answer questions about their content
• Distinction: All VLMs are multimodal, but not all multimodal models are VLMs. VLMs are specifically focused on image plus language

Use: Document analysis (invoices, forms, scans), quality control in manufacturing, medical image analysis, visual search, accessibility (image descriptions).

Examples: GPT-4o (Vision), Gemini, Claude (Vision), LLaVA, Qwen-VL, Llama 4 Scout

LMM

Multimodal Models (Large Multimodal Models)

Models that process and/or generate more than two modalities simultaneously: text, image, audio, video.

• Development is moving from text-to-text towards any-to-any models
• Difference to VLMs: LMMs are the broader umbrella term and also include audio, video, and other sensor data

Use: Complex workflows that require different data types simultaneously. Video analysis with text summarization, voice input with visual output, multimodal agents.

Examples: GPT-4o (Audio + Vision + Text), Gemini 3, Meta 4M

STT/TTS

Speech to Text and Text to Speech

STT / ASR (Automatic Speech Recognition): Converts spoken language into written text.

• Supports real-time streaming and batch transcription
• Features: Speaker diarization, automatic punctuation, profanity filtering, custom vocabulary

TTS (Text to Speech): Converts written text into spoken language.

• Neural TTS models produce natural-sounding voices
• Voice cloning enables brand-specific voices

Use STT: Transcription of meetings, interviews, call centers, podcasts. Voice control. Accessibility.

Use TTS: Voicebots, voice assistants, audiobook generation, conversational agents, accessibility.

Examples: OpenAI Whisper (STT, Open Source), Google Speech-to-Text, Azure Speech, Amazon Transcribe, ElevenLabs (TTS), OpenAI TTS, NVIDIA Riva, Azure Custom Neural Voice

VID

Video Generation Models

Models that generate video content from text descriptions, images, or short clips.

• Technically based on diffusion models that are extended for temporal coherence and motion
• The results are now almost indistinguishable from filmed material

Use: Commercials, special effects, concept visualisation, storytelling, product videos.

Examples: Sora (OpenAI), Veo 3 (Google DeepMind), Gen-4 (Runway), Kling Video, NVIDIA Cosmos

RWD

Reward Models

Models trained to represent human preferences.

• Evaluate the quality of a language model's responses on a scale
• Are used in the RLHF (Reinforcement Learning from Human Feedback) process to steer the actual language model
• Act as a bridge between human feedback and model behavior

Use: Alignment training of LLMs. Quality assessment of model responses. Filtering and ranking of outputs. Not directly for end-users, but part of the model development pipeline.

Examples: Reward Models from OpenAI, Anthropic, Nemotron-Reward (NVIDIA)

TS

Time Series Models

Foundation models pretrained on large, cross-domain time series data.

• Can provide predictions, anomaly detection, and classification on new data without task-specific training (zero-shot)
• Application areas: Finance, energy, healthcare, manufacturing, IoT
• Limitation: Time series data is domain-specific (seasonality, trends, irregular sampling), which is why specialised models are often more accurate in practice than general foundation models

Use: Sales planning, energy demand forecasting, predictive maintenance, financial forecasting, anomaly detection in sensor data. Especially valuable when historical data is missing or insufficient.

Examples: TimesFM 2.5 (Google), Chronos 2 (Amazon), MOMENT, Lag-Llama

DOM

Domain-Specific Foundation Models

Pretrained models that are specifically trained on data from a particular industry or domain.

• Difference from general LLMs: deeper understanding of subject-specific terminology, contexts, and regulations
• Can be fine-tuned for industry-specific downstream tasks

Use: Medicine (radiology, pathology, clinical texts), law (contract analysis, regulatory matters), life sciences (protein structure, genomics), finance (risk assessment, compliance), manufacturing (quality control, process optimization).

Examples: Med-PaLM (Google, Medicine), ESMFold (Meta, Protein Structure), BloombergGPT (Finance), BioMistral (Biomedicine), SecLM (Cybersecurity)

Where do you host the model?

There are several options:

• Cloud API (Managed Service): You use the model via an AI model provider's API. The provider handles hosting, scaling, and maintenance. You pay per token or per request. No need for your own GPUs.
• Hyperscaler Public Cloud: You run models on GPU instances with hyperscalers. Shared infrastructure, virtually isolated. You use the hyperscaler's ecosystem (monitoring, logging, IAM), but you are responsible for operation and scaling yourself.
• Private Cloud: Dedicated, physically isolated infrastructure at a provider. Single-tenant. Only you use the hardware. Relevant when regulations require demonstrable separation from other tenants.
• Self-Hosted Cloud (own infrastructure in the cloud): You run models on rented GPU servers. Full control over the model and data, but you are responsible for operation, scaling, and updates.
• On-Premise (local, own hardware): You run models on your own hardware in your own data center.

Decision factors: Data privacy requirements, cost (per-token vs. fixed costs), latency, scaling needs, regulations.

How do you call the model?

• API Endpoint (REST API): The standard way — an HTTP request to an endpoint. You send a request (prompt, configuration) and receive the response. Most providers use an OpenAI-compatible API format, which is considered the de facto standard. Self-hosted solutions (Ollama, vLLM, LocalAI) also offer OpenAI-compatible endpoints, so you can replace cloud services with local models without changing your code.
• SDK (Software Development Kit): Libraries in Python, TypeScript, etc., that abstract the API call. Examples: OpenAI Python SDK, Anthropic SDK, LangChain, LlamaIndex.
• Inference Server: For self-hosted models, a server process that loads the model and accepts requests. Frameworks: vLLM, TGI (Text Generation Inference), NVIDIA Triton, Ollama.

This is what it looks like in practice

Here is a very simplified example that ignores all other problems for now — this is just about the principle of calling a model.

A business user creates a report. Relevant data is loaded from internal systems beforehand. This section is only about the step in which the model is called.

Option 1: Local model with Ollama

answer = requests.post(
    "http://localhost:11434/v1/chat/completions",
    json={
        "model": "mistral",
        "messages": [
            {
                "role": "user",
                "content": f"Create a report from this data: {xyz_data}"
            }
        ]
    }
)

The model runs on your machine. No data leaves the hardware.

Option 2: Cloud API

answer = requests.post(
    "https://api.openai.com/v1/chat/completions",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "model": "gpt-4o",
        "messages": [
            {
                "role": "user",
                "content": f"Create a report from this data: {xyz_data}"
            }
        ]
    }
)

What this shows is: the code is almost identical in both cases. The only difference is the URL — localhost:11434 instead of api.openai.com. That's why it's called OpenAI-compatible: you can switch providers without rewriting your code.

What you can do with models

For very specific requirements, you can modify a model.

You can customize it: retrain the weights with your own data so that it behaves better in a specific domain.

And you can optimize it: compress the weights so the model becomes smaller and runs faster.

For the vast majority of business use cases, you need neither.

Customizing and optimizing require machine learning expertise, GPU infrastructure, and complex evaluation — that's expensive and slow. At the same time, the base models of 2026 are so powerful that they can solve most tasks without any modification.

What's much more important is the architecture around it.

Still, just so you've heard it:

Customizing a model

• Fine-tuning retrains all weights with a custom dataset.
• LoRA freezes most weights and only trains a small additional layer.
• QLoRA does the same on a compressed basis, allowing it to run on consumer hardware.
• RLHF and DPO align the model with human preferences.
• Model merging combines the weights of multiple models without training.

Optimizing a model

• Quantization reduces the numerical precision of the weights, making the model up to 75% smaller.
• Pruning removes weights that contribute little.
• Knowledge distillation has a large model train a smaller one that performs similarly well.

The only open question

The only thing that hasn't really been settled yet — and I have to say, it annoys me a little. There are so many professors, researchers, and AI influencers, and none of them will just decide on it. Or if someone has decided, they haven't told the world yet. I couldn't find anything about it.

So what is the official symbol for drawing a model?!

Honestly, I personally find drawing little robots, stars, or brains very childish. I want to work professionally.

The symbol must be quick to draw by hand, in case you're in a meeting and drawing on a whiteboard. So all those symbols with lots of nodes and lines are out.

Take the cylinder for databases as a blueprint.

I hereby establish the following symbol. Depending on the type of model, you insert different letters. Feel free to draw it a bit nicer:

LLM symbol for architecture diagrams, made by Bianca J. Schulz

Feel free to forward this to people who could make the decision. If no one wants to decide, then I've hereby decided it myself 😎