What is multimodal AI?

Multimodal AI is AI that can handle more than one kind of data, or "modality": text, images, audio, video and more. Most early chatbots only read and wrote text. A multimodal model can take in a photo and describe it, read a chart, listen to your voice, or combine several of these at once.

How it works

Each type of data is first turned into a form the model can work with, and the results are then combined, or "fused," so the model can connect what it sees with what it reads or hears. Claude, for example, splits an image into small squares it treats as visual tokens.

Some models bolt separate parts together after training. Google says its Gemini models were instead trained on text, code, audio, images and video from the start, which it calls natively multimodal.

Examples you can use today

• Image understanding: you can upload photos, screenshots or documents to Claude, ChatGPT or Gemini and ask questions about them.
• Voice conversations: Google's Gemini Live lets people talk to the model in real time and interrupt it, in 70 languages.
• Image creation: OpenAI's image models can generate new images or edit existing ones from text and image prompts.
• Beyond chat: IBM points to medical imaging, voice assistants and self-driving cars that combine camera and lidar data.

The limits

Seeing isn't the same as understanding perfectly. Anthropic says Claude can make mistakes with blurry, rotated or very small images and gives only approximate counts, and it shouldn't replace a doctor reading a scan. OpenAI lists similar weak spots, including non-Latin text, precise layouts and specialist medical images. Like text-only models, multimodal models can also hallucinate.