Generative AI beyond text

This page is an overview of generative AI for media other than text: images, audio, and video. It explains the main techniques in plain terms, where the tools are used, and the problems they raise. Product names appear only as examples, because this part of the field changes even faster than language models.

From text to every medium

Generative AI covers any model that produces new content. For several years, each medium needed its own kind of model. That is changing: multimodal models increasingly read and produce text, images, and audio within one system. A single assistant can now describe a photo, edit it on request, and answer aloud.

The common pattern is the same everywhere. A model learns the structure of a huge amount of media during training, and at generation time it produces new media that fits a description, the prompt, or a reference such as an image or a voice sample.

Images

Image generation reached a wide audience in 2022, when DALL·E 2, Midjourney, and the open model Stable Diffusion appeared within a few months of each other.

Most image models use diffusion. During training, the model sees images that have been covered with increasing amounts of random noise and learns to remove it, step by step. To generate a new image, it starts from pure noise and removes noise repeatedly, guided by the text prompt, until an image appears that matches the description.

Newer models generate images inside a multimodal language model. Because the same model understands language well, these models follow detailed instructions more precisely, render readable text inside images, and edit existing images through conversation: “make the background darker”, “put the logo on the left”.

Common uses are illustrations, marketing material, product mockups, concept art, photo editing, and design drafts.

Audio and speech

Audio covers several separate tasks:

• Speech recognition turns speech into text. OpenAI’s open Whisper model, released in 2022, made accurate transcription in many languages widely available. It powers meeting notes, subtitles, and voice input.
• Speech synthesis, or text-to-speech, reads text aloud in a natural voice, with emotion and emphasis. Companies such as ElevenLabs specialize in it.
• Voice cloning reproduces a specific person’s voice from a short recording.
• Voice conversation lets a model listen and answer in speech directly, with short delays and the ability to be interrupted, much like a phone call.
• Music generation creates songs with vocals and instruments from a text description. Suno and Udio are known examples.

Audio is also where misuse is most direct: a cloned voice on a phone call is a powerful tool for fraud.

Video

Video adds time to images: every frame must be consistent with the frames before and after it, and objects must move in a physically believable way. OpenAI previewed its video model Sora in early 2024, and Google, Runway, Kuaishou (Kling), and others followed with their own models. Current models generate clips of several seconds to about a minute, and the newest ones generate matching sound and speech along with the picture.

Uses include advertising, social media content, storyboards, visual effects, and training material. Long, coherent videos with consistent characters are still difficult.

Related research builds world models: systems that generate interactive environments a user or an agent can move through in real time. They are interesting for games, simulation, and training robots.

Other media

Generative models also produce 3D objects for games and product design, molecules and proteins for drug research, code for user interfaces, and slides and diagrams. The same idea applies throughout: learn the structure of a domain from examples, then generate new examples on request.

Copyright and consent

Generative media raises legal and ethical questions that are not fully settled:

• Training data. Models learn from millions of images, songs, and videos, often collected from the web without asking the creators. Artists, photo agencies, music labels, and authors have sued AI companies. Courts in different countries are reaching different conclusions, and some AI companies now license content instead.
• Ownership of the output. In many countries, including the United States, copyright protects only works with enough human authorship. A purely generated image may not be protected at all.
• Style and likeness. Imitating a living artist’s style or a real person’s face or voice raises questions of consent, personality rights, and fair compensation.

Telling generated from real

As generated media becomes realistic, people and platforms need ways to tell it apart:

• Labels. Platforms and laws increasingly require that AI-generated content is marked. The EU AI Act includes transparency rules for deepfakes and generated content.
• Watermarks. Some providers embed invisible signals in generated images, audio, or video. Google’s SynthID is an example. Watermarks help, but they can be weakened by editing.
• Content credentials. The C2PA standard, backed by Adobe, Microsoft, Google, camera makers, and others, attaches signed information to a file about where it came from and how it was edited.

None of these is complete on its own. A healthy skepticism toward surprising images, audio, and video, especially when they ask for money or urgent action, remains the most reliable protection.