# What is Multimodal Model?

Also called vision language model, multimodal AI.

A multimodal model accepts or produces more than one type of data, such as text together with images, audio, or video. Inputs from each modality are converted into a shared internal representation so the model can relate them, for example answering a question about a photograph. Most widely used assistants now accept images alongside text and reply in text.

The usual construction pairs a language model with an encoder for the other modality. An image is divided into patches, each patch is turned into a vector, and those vectors are projected into the same representational space the language model uses for text tokens. From the model's perspective the image simply becomes another sequence it can attend to alongside the words.

Capability is asymmetric and often misunderstood. A model that reads images does not necessarily generate them, and one that generates images is frequently a separate system invoked as a tool by the assistant. Audio support splits similarly between transcription, native audio understanding, and speech generation, which are distinct capabilities with different limits and different pricing.

Images consume tokens, sometimes a great many of them. A high-resolution photograph can cost as much context as several pages of text, and providers usually downscale or tile large images before processing, which is why small print and dense diagrams are read unreliably. The exact resizing and cropping behavior varies by provider and is worth checking.

Practical accuracy varies sharply by task. Describing a scene, reading a clear printed document, or interpreting a simple chart usually works well. Counting many similar objects, reading handwriting, judging fine spatial relationships, and interpreting dense tables remain error-prone, so any output that drives a consequential decision should be verified against the source.

## Key points

- Handles more than one data type, commonly text plus images.
- Non-text inputs are encoded into the same space as text tokens.
- Reading a modality and generating it are separate capabilities.
- Images consume significant context and are often downscaled.
- Counting, handwriting, and dense tables remain unreliable.

## In practice

You photograph a restaurant receipt and ask for the total, the tip, and whether any item was charged twice. The model encodes the image into patches, aligns them with your question, and reads the printed lines. It returns the totals correctly and flags a duplicated side dish. Asked how many olives are visible on a plate, the same model will often miscount.

## Related terms

- [Large Language Model](/en/glossary/large-language-model)
- [Foundation Model](/en/glossary/foundation-model)
- [Embedding Model](/en/glossary/embedding-model)
- [Token](/en/glossary/token)
- [Context Window](/en/glossary/context-window)

[Back to the AI Glossary](/en/glossary)