Definition
A multimodal model learns or applies relationships across more than one modality, such as text, images, audio, video, sensor signals, or structured data. It may accept several modalities, generate several modalities, or map between them.
Architectures vary. Some use a separate encoder for each modality and align their representations in a shared space. Others convert different inputs into a common token sequence or connect modality-specific encoders to one generative decoder. The fusion point determines which relationships the model can learn directly.
CLIP, published by Alec Radford and coauthors in 2021, is an influential example. It trained image and text encoders together so matched image-text pairs occupied related positions in a shared representation space. That design supports cross-modal retrieval and zero-shot image classification, but it is only one form of multimodal modeling.
Distinguish it from nearby terms
- A multimodal system can connect separate single-modality models through application code. The system is multimodal even when no individual model is.
- A multi-task model performs several tasks, which may all use the same modality.
- An ensemble combines several models. It is not multimodal unless the combined system works across distinct data forms.
- A vision-language model is a multimodal model focused on visual and linguistic data.
Operational significance
Evaluate each modality and the relationships between them. A model may transcribe clear speech well but fail when the audio conflicts with on-screen text, or classify images well while missing spatial details needed by a question. Input limits, synchronization, preprocessing, and missing-modality behavior belong in the test plan.
Check your understanding
A text model calls a separate optical-character-recognition service and reads the returned text. The application handles images and text, so it is a multimodal system. The text model itself remains single-modality.