What it is for
What a camera wrote between record and stop, with nothing added afterwards: a phone clip, a take from a shoot, a screen recording. One shot, so the camera, the picture and who is in it sit at the top level; what changes over time is carried by the sections that have time in them.
not for
Anything cut together, which is video/edit.
never has
scenes, shots, subtitles, titles
Every video file starts with the same envelope: format, version, type, generated, source. What follows is this type's own.
MmeasuredOobservedDderivedMOmixed
The one shot
After the envelope and summary, the take itself: what the camera did, what the pixels look like, how the frame is built, who and what is in it. All at the top level, because footage has exactly one shot.
| field | shape | meaning | |
|---|---|---|---|
| camera | M | { support, jitter, zoom, centreZoom, subjectScale, pan, tilt, moves [ { s, e, kind, amount, unit } ], obstructions [ { s, e, kind, coverage } ] } | What the camera did. pan and tilt are the net motion in percent of the frame per second; zoom the background's net scale; jitter the frame centre's detrended shift as a fraction of width per frame, from which support follows. moves is the list that matters in a long take. |
| picture | M | { palette [hex], warmth, saturation, tonalRange, luma, contrast, sharpness, clippedHighlights, crushedBlacks, frameChange } | Colour and tone of the pixels over the take. frameChange is the mean frame-to-frame difference. |
| form | O | { size, angle, lens, depthOfField, lightKey, lightQuality, lightSource, timeOfDay, interior, setting } | The shot as a cinematographer names it, the values that hold for the take. |
| people[] | O | [ { label, description, size, position, facesCamera, eyeline, speaking } ] | The subjects: who the camera is on, who performs, speaks or is addressed. Letters in order of first appearance. Since there is one shot, who they are and how they are framed sit on the same item. |
| clothing, composition, background, description | O | string | What the subjects wear; how the frame is built, in a sentence; what is behind them; what the take shows, in a paragraph. |
| objects[], details[] | O | [ string ] | Things worth naming; small things a viewer would miss. |
| quality[] | O | [ string ] | Conditions visibly there: blurry, shaky, dark. Never a grade. |
| text[] | O | [ { s, e, text, where } ] | Text in the world: a sign, a screen, a label, with the time it is legible. Nothing in footage was added afterwards. |
What happens
| section | shape | meaning | |
|---|---|---|---|
| bystanders[] | O | [ { s, e, text } ] | Unlabelled people passing through, with the times they are in frame. Distinct from the background string, which is the setting behind the subjects. |
| activities[] | O | [ { s, e, name, who [labels], text } ] | The continuous things happening, covering the take. |
| moments[] | O | [ { t, activity, kind, text } ] | The instants an activity's state changes: begins, completes, fails, turns to camera… |
| actions[] | O | [ { s, e, who [labels], text, camera, details [] } ] | A running description, one every few seconds, no gaps. camera says what the camera did during it, in words. |
Speech
Present when anyone speaks. Spoken words only; singing in the room is music.
| field | shape | meaning | |
|---|---|---|---|
| language | M | BCP 47 | en, es-419. One per file. |
| speakers[] | MO | [ { id, person } ] | id is the voice's index in words and paragraphs, measured. person is the letter in people[]: whoever is seen (or heard) speaking, or "off camera" for a voice never seen. |
| paragraphs[] | M | [ { s, e, speaker, sentences [ { s, e, text } ] } ] | The reading view: one speaker per paragraph, punctuated sentences with their own spans. |
| words[] | M | [ { w, s, e, confidence, speaker } ] | Every word with its span, punctuation attached, confidence 0 to 1, speaker index. The alignment view. |
| fillers[] | M | [ { w, t } ] | The ums and uhs, also in words; listed again so an editor can cut them. |
| silences[] | M | [ { s, e } ] | Spans below the noise floor, 0.5 s and up. |
| loudness | M | { integrated, truePeak, range } | Of the track: LUFS, dBTP, LU, as broadcast meters report them. |
Music
Music that was in the room: a radio, a band, a phone. Absent means nobody listened for it; an empty heard means there was none.
| section | shape | meaning | |
|---|---|---|---|
| heard[] | O | [ { s, e, text } ] | Where music plays and what it sounds like there. |
| songs[] | M | [ { label, pieces [ { s, e, at } ], seconds, tempo, key, keyConfidence, recognition } ] | One per distinct recording, lettered. pieces are the spans of the file where it plays, each with at: the position in the recognised recording that plays at s, so a reader subtracts to know which part of the song any moment is. seconds is the total played. tempo in BPM, key as "F# minor" with keyConfidence 0 to 1. |
| songs[].recognition | M | { match, title, artists [], album, releaseDate, label, duration, language, genres [], isrc, upc, score, confidence, matchedAt, spotify, youtube, deezer, musicbrainz } | null | The recording as a catalogue names it. match is recording when the file holds that very recording, cover when another performance of the same song. duration is the recognised recording's full length. confidence is identified (score ≥ 70) or likely (40 to 69); null means not in the catalogue, common for library music. It names the recording; it says nothing about rights. |
| beats[] | M | [ t, … ] | Every beat, whole file. |
Events
| field | shape | meaning | |
|---|---|---|---|
| events[] | D | [ { t, e, kind, text } ] | Everything with a time, flat and time-ordered. A writer regenerates it, never edits it. |
Footage answers "what did the camera see and do, for how long". An editor's assistant reads camera.moves, moments and speech.silences to find the usable stretch; a library reads summary, form and source.location to file it.
Vocabularies
Closed lists. Each includes other; a reader that meets an unknown value treats it as other and keeps going.
camera
support: locked off, handheld, moving, unknown. moves[].kind: pan left, pan right, tilt up, tilt down, zoom in, zoom out, push in, pull out, other; unit: % of width per s, % of height per s, x. obstructions[].kind: occlusion, foreground pass, other.
moments[].kind
begins, ends, attempt, reaches position, completes, fails, enters, leaves, picks up, puts down, turns to camera, turns away, transition, other.
form
size: extreme close-up, close-up, medium close-up, medium, medium wide, wide, extreme wide, no subject. angle: eye level, high, low, overhead, dutch. lens: wide, normal, long, unknown. depthOfField: shallow, deep, unknown. lightKey: front, side, back, top, flat, unknown. lightQuality: hard, soft, mixed, unknown. lightSource: daylight, artificial, mixed, screen, unknown. timeOfDay: day, golden hour, night, indoor unknown, unknown. interior: interior, exterior, unknown.
people[]
size: close-up, medium, wide. position: left, center, right. eyeline: camera, left, right, down, up, away, unknown.
events[].kind
move, activity, moment, music, speech, pause, silence, action, text, person, quality, other.
quality[]
blurry, shaky, dark, overexposed, subject cut off, obstructed, other.
Example
Illustrative and abbreviated.
{
"format": "smartmedia", "version": "1.1", "type": "video/footage",
"generated": { "at": "2026-09-14T10:02:11Z", "by": "smartmedia-api 0.1",
"models": { "vision": "gemini-3.7-flash", "speech": "deepgram nova-3" } },
"source": { "name": "IMG_1731.MOV", "bytes": 61230411, "sha256": "5c5e…", "quickHash": "b742…",
"container": "mov,mp4,m4a,3gp,3g2,mj2", "duration": 48.21,
"video": { "codec": "hevc", "width": 1080, "height": 1920, "fps": 30, "rotation": 0, "hdr": true },
"audio": { "codec": "aac", "channels": 2, "sampleRate": 48000 },
"camera": { "make": "Apple", "model": "iPhone 17 Pro" },
"recorded": { "utc": "2026-09-06T15:12:40Z", "local": "2026-09-06T17:12:40", "offset": "+02:00",
"timeOfDay": { "bucket": "afternoon", "sunUp": true, "minutesToSunset": 189 } },
"location": { "lat": 41.4036, "lon": 2.1744, "address": { "city": "Barcelona", "countryCode": "ES" },
"venue": { "name": "Plaça de la Virreina", "reason": "A small playground in a paved square." } } },
"summary": "A handheld vertical take of a child on a swing in a paved square, pushed by a woman who talks to her …",
"camera": { "support": "handheld", "jitter": 0.006, "zoom": 1.0, "pan": 0.4, "tilt": -0.1,
"moves": [ { "s": 12.30, "e": 15.10, "kind": "pan right", "amount": 22, "unit": "% of width per s" } ],
"obstructions": [] },
"picture": { "palette": [ "#8C9A6E", "#D8D2C4", "#3A4A2E" ], "warmth": 0.12, "luma": 0.55, "contrast": 0.29, "sharpness": 412 },
"form": { "size": "medium wide", "angle": "eye level", "lens": "wide", "lightSource": "daylight", "timeOfDay": "day",
"interior": "exterior", "setting": "playground in a paved square" },
"people": [ { "label": "A", "description": "A child of about four in a red T-shirt.", "size": "medium", "position": "center",
"facesCamera": true, "eyeline": "camera", "speaking": true },
{ "label": "B", "description": "A woman in a grey hoodie pushing the swing.", "size": "wide", "position": "left" } ],
"clothing": "A: red T-shirt, denim shorts. B: grey hoodie, black leggings.",
"activities": [ { "s": 0, "e": 48.21, "name": "swinging", "who": [ "A", "B" ], "text": "B pushes A on the swing; A asks to go higher." } ],
"moments": [ { "t": 18.40, "activity": "swinging", "kind": "turns to camera", "text": "A looks straight at the phone and grins." } ],
"speech": { "language": "es", "speakers": [ { "id": 0, "person": "A" }, { "id": 1, "person": "B" } ],
"paragraphs": [ { "s": 2.10, "e": 3.40, "speaker": 0, "sentences": [ { "s": 2.10, "e": 3.40, "text": "¡Más alto!" } ] } ],
"silences": [ { "s": 8.20, "e": 11.90 } ], "loudness": { "integrated": -19.8, "truePeak": -1.2, "range": 12.1 } },
"music": { "heard": [] },
"events": [ { "t": 18.40, "kind": "moment", "text": "turns to camera: A looks straight at the phone and grins." } ]
}