A small text file next to every video, audio file and image, saying what it is and what is in it.
Like an .srt for subtitles, for everything: what the file is, when and where it was made, who is in it, what happens, what is said, what is written, what plays. Software and language models read it without decoding the media.
One file per media file
IMG_1731.MOV gets IMG_1731.MOV.smartmedia next to it. Every value is either a fact of the file's bytes or a reading of the picture or the sound, and the format says which. It describes; it never recommends.
{
"format": "smartmedia", "version": "1.1",
"type": "video/footage",
"source": {
"name": "IMG_1731.MOV", "bytes": 61230411,
"sha256": "5c5e…", "duration": 48.21,
"camera": { "make": "Apple", "model": "iPhone 17 Pro" },
"recorded": { "local": "2026-09-06T17:12:40", "offset": "+02:00" },
"location": { "city": "Barcelona",
"venue": { "name": "Plaça de la Virreina" } }
},
"summary": "A handheld vertical take of a child on a swing …",
"camera": { "support": "handheld",
"moves": [ { "s": 12.30, "e": 15.10, "kind": "pan right" } ] },
"people": [ { "label": "A", "description": "A child of about four …" } ],
"speech": { "language": "es", "paragraphs": [ … ],
"silences": [ { "s": 8.20, "e": 11.90 } ] },
"events": [ { "t": 18.40, "kind": "moment",
"text": "A looks straight at the phone and grins." } ]
}Readable by anything. grep, a shell loop, a Mac app, an LLM with the file pasted in. A folder of sidecars is a folder of documents an assistant can read in one pass.
Bound to the original. The SHA-256 of the file's bytes and a quick hash of its size, head and tail. A renamed or moved original still matches; a different file with the same name does not.
Measured or observed, per field. Measured values are reproducible facts of the samples. Observed values are a model's reading of what is there, never what to do with it. No field anywhere holds an opinion.
Six types, one envelope. Footage and edits, speech and music, scenes and graphics. The writer declares the type; a reader never second-guesses it.
Six types
Each is a separate format on its own page, sharing the envelope and the rules.
One continuous take, as recorded: what was in front of the lens and what the camera did.
Cut together: scenes and shots, subtitles and titles, a soundtrack.
People talking: the transcript, the voices, the silences, the levels.
A track or a mix: which recording it is, which part plays, tempo, key, beats, energy.
A view of something: form, composition, people, objects, light. No time.
Made of marks: a screen, a document, a chart. Layout and text, not a viewpoint.
Three ways to get one
The protocol is open. The API writes it for you. The Mac app runs the API over a folder.
The protocol
The format itself: the file, the rules every reader can rely on, the envelope, and each type's sections and vocabularies. Free to read and to write.
Read the specification →The API
A file in, a sidecar out. Measured with ffmpeg, OpenCV and librosa; observed by vision and speech models; priced per minute or per image, prepaid, nothing charged for a failed job.
See the API →The Mac app
Drop videos, audio and images on a window, say what they are, and each file gets its sidecar next to it. Large videos go up as a small proxy; the sidecar still describes the original.
See the app →