What it is for
A finished piece: a reel, a tutorial, a film, an ad. Everything an edit has that a single take does not, so the one-shot fields of footage live on each shot here, and the structure of the cut sits on top.
not for
A single take, which is video/footage. A 1.0 file, which has no type, is an edit.
never has
camera, picture or form at the top level: those belong to each shot
Every video file starts with the same envelope: format, version, type, generated, source. What follows is this type's own.
MmeasuredOobservedDderivedMOmixed
Scenes and shots
The structure of the cut: shots, contiguous and covering the file, grouped into scenes. A shot carries what footage carries at the top level, measured for that stretch of frames and observed for that stretch.
| field | shape | meaning | |
|---|---|---|---|
| summary | O | string | The whole piece in one paragraph. |
| scenes[] | O | [ { id, s, e, shots [ids], place, text } ] | Contiguous runs of shots at one place, in order, every shot in exactly one scene. |
| shots[] | MO | [ { id, s, e, scene, boundary, camera, picture, form, people, clothing, composition, background, description, objects, details, quality, text } ] | One per shot, contiguous, ids from 1. scene is the scene id. |
| shots[].boundary | M | { kind, factor, likely } | How the shot begins. factor is the zoom of a punch-in or punch-out; likely marks a wipe or dissolve guessed from continuity rather than matched. |
| shots[].camera | M | { support, jitter, zoom, centreZoom, subjectScale, pan, tilt, moves [], obstructions [] } | What the camera did in the shot, as on the footage page. |
| shots[].picture | M | { palette, warmth, saturation, tonalRange, luma, contrast, sharpness, clippedHighlights, crushedBlacks, frameChange } | Colour and tone of the shot's pixels. |
| shots[].form | O | { size, angle, lens, depthOfField, lightKey, lightQuality, lightSource, timeOfDay, interior, setting, sameSceneAsPrevious } | The shot as a cinematographer names it. sameSceneAsPrevious is the model's own reading; scene is the settled answer. |
| shots[].people[] | O | [ { label, size, position, facesCamera, eyeline, speaking } ] | Who is in the shot and how they are framed. Labels are the letters of the top-level people[]. |
| shots[].clothing, composition, background, description | O | string | What the subjects wear; how the frame is built; what is behind them; what the shot shows. |
| shots[].objects[], details[], quality[] | O | [ string ] | Things worth naming; small things a viewer would miss; conditions visibly there. |
| shots[].text[] | O | [ { text, where } ] | Text legible in the shot's frames. The classified, timed list is the top-level text. |
Who is in it
| section | shape | meaning | |
|---|---|---|---|
| people[] | O | [ { label, description } ] | The subjects across the whole piece, lettered in order of first appearance; the same letter means the same person in every shot. |
| background[] | O | [ { s, e, text } ] | Unlabelled people passing through, with the times they are in frame. The footage page calls this bystanders; here the word is the 1.0 one. |
What happens
Across cuts: an activity can span many shots, and an action is a running description that does not care where the cuts fall.
| section | shape | meaning | |
|---|---|---|---|
| activities[] | O | [ { s, e, name, who [labels], text } ] | The continuous things happening, covering the piece. |
| moments[] | O | [ { t, activity, kind, text } ] | The instants an activity's state changes. |
| actions[] | O | [ { s, e, who [labels], text, camera, details [] } ] | A running description, one every few seconds, no gaps. |
Speech
Present when anyone speaks; the same block as in footage. Only silences and loudness when nobody spoke.
| field | shape | meaning | |
|---|---|---|---|
| language | M | BCP 47 | en, es-419. One per file. |
| speakers[] | MO | [ { id, person } ] | id is the voice's index in words and paragraphs, measured. person is the letter in people[]: whoever is seen (or heard) speaking, or "off camera" for a voice never seen. |
| paragraphs[] | M | [ { s, e, speaker, sentences [ { s, e, text } ] } ] | The reading view: one speaker per paragraph, punctuated sentences with their own spans. |
| words[] | M | [ { w, s, e, confidence, speaker } ] | Every word with its span, punctuation attached, confidence 0 to 1, speaker index. The alignment view. |
| fillers[] | M | [ { w, t } ] | The ums and uhs, also in words; listed again so an editor can cut them. |
| silences[] | M | [ { s, e } ] | Spans below the noise floor, 0.5 s and up. |
| loudness | M | { integrated, truePeak, range } | Of the track: LUFS, dBTP, LU, as broadcast meters report them. |
Text
An edit adds text to the picture, so text is sorted by what it is: burned-in subtitles collapse into one run, measured against the transcript; the rest are items with their kind.
| field | shape | meaning | |
|---|---|---|---|
| text.subtitles | M | { s, e, count, x, y, where, matched } | null | The run of burned-in captions: how many cards, where on the frame, and the share of their words found in the transcript around their time. null when the piece has none. |
| text.items[] | MO | [ { s, e, text, kind, x, y, style, matched } ] | Everything else written on the picture, read exactly, with the time it is legible and its position. kind is measured for subtitle and graphic, observed otherwise. |
Music
The soundtrack, and any music that was in the room.
| section | shape | meaning | |
|---|---|---|---|
| heard[] | O | [ { s, e, text } ] | Where music plays and what it sounds like there. |
| songs[] | M | [ { label, pieces [ { s, e, at } ], seconds, tempo, key, keyConfidence, recognition } ] | One per distinct recording, lettered. pieces are the spans of the file where it plays, each with at: the position in the recognised recording that plays at s, so a reader subtracts to know which part of the song any moment is. seconds is the total played. tempo in BPM, key as "F# minor" with keyConfidence 0 to 1. |
| songs[].recognition | M | { match, title, artists [], album, releaseDate, label, duration, language, genres [], isrc, upc, score, confidence, matchedAt, spotify, youtube, deezer, musicbrainz } | null | The recording as a catalogue names it. match is recording when the file holds that very recording, cover when another performance of the same song. duration is the recognised recording's full length. confidence is identified (score ≥ 70) or likely (40 to 69); null means not in the catalogue, common for library music. It names the recording; it says nothing about rights. |
| beats[] | M | [ t, … ] | Every beat, whole file. |
Events
| field | shape | meaning | |
|---|---|---|---|
| events[] | D | [ { t, e, kind, text } ] | Everything with a time, flat and time-ordered, including every cut and every change of setting. |
An edit answers "how is this cut, and what is in each piece". An editor's assistant reads shots[] with their boundaries and text.subtitles to rebuild the timeline; a library reads summary, scenes[].place and music.songs[] to file it.
Vocabularies
Closed lists. Each includes other; a reader that meets an unknown value treats it as other and keeps going.
shots[].boundary.kind
start (the first shot), cut, punch-in, punch-out, wipe, dissolve, scene (a frame-difference cut the geometry did not see).
shots[].camera
As footage: support locked off, handheld, moving, unknown; moves[].kind pan left, pan right, tilt up, tilt down, zoom in, zoom out, push in, pull out, other; obstructions[].kind occlusion, foreground pass, other.
shots[].form and people[]
size: extreme close-up, close-up, medium close-up, medium, medium wide, wide, extreme wide, no subject. angle: eye level, high, low, overhead, dutch. lens: wide, normal, long, unknown. depthOfField: shallow, deep, unknown. lightKey: front, side, back, top, flat, unknown. lightQuality: hard, soft, mixed, unknown. lightSource: daylight, artificial, mixed, screen, unknown. timeOfDay: day, golden hour, night, indoor unknown, unknown. interior: interior, exterior, unknown. People: size: close-up, medium, wide. position: left, center, right. eyeline: camera, left, right, down, up, away, unknown.
moments[].kind
begins, ends, attempt, reaches position, completes, fails, enters, leaves, picks up, puts down, turns to camera, turns away, transition, other.
text.items[].kind
subtitle, title, lower third, graphic, scene text, other. A subtitle item outside the collapsed run is one the transcript did not confirm.
shots[].quality[]
blurry, shaky, dark, overexposed, subject cut off, obstructed, other.
events[].kind
cut, scene, move, activity, moment, music, speech, pause, silence, action, text, person, quality, other.
Example
Illustrative and abbreviated.
{
"format": "smartmedia", "version": "1.1", "type": "video/edit",
"generated": { "at": "2026-09-14T10:02:11Z", "by": "smartmedia-api 0.1", "models": { "vision": "gemini-3.7-flash", "speech": "deepgram nova-3" } },
"source": { "name": "handstand.mp4", "bytes": 48120311, "sha256": "9e3a…", "quickHash": "77b0…", "duration": 71.20,
"video": { "codec": "h264", "width": 1080, "height": 1920, "fps": 30 }, "audio": { "codec": "aac", "channels": 2, "sampleRate": 48000 } },
"summary": "A coach demonstrates a wall handstand in a gym, then a client tries it while the coach spots her; captions repeat the cues.",
"scenes": [ { "id": 1, "s": 0, "e": 38.10, "shots": [ 1, 2, 3 ], "place": "gym floor by a padded wall", "text": "The coach demonstrates against the wall." },
{ "id": 2, "s": 38.10, "e": 71.20, "shots": [ 4, 5 ], "place": "the same wall, closer", "text": "The client attempts it with a spot." } ],
"shots": [
{ "id": 1, "s": 0, "e": 12.40, "scene": 1, "boundary": { "kind": "start" },
"camera": { "support": "locked off", "jitter": 0.001, "zoom": 1.0, "pan": 0.0, "tilt": 0.0, "moves": [], "obstructions": [] },
"picture": { "palette": [ "#2B2B2B", "#C9C2B6", "#E8402A" ], "luma": 0.41, "contrast": 0.30 },
"form": { "size": "wide", "angle": "eye level", "lightSource": "artificial", "interior": "interior", "setting": "gym floor" },
"people": [ { "label": "A", "size": "wide", "position": "center", "facesCamera": true, "eyeline": "camera", "speaking": true } ],
"description": "A stands facing the camera and talks, then turns to the wall.", "objects": [ "a padded wall", "a rack of kettlebells" ] },
{ "id": 2, "s": 12.40, "e": 24.00, "scene": 1, "boundary": { "kind": "punch-in", "factor": 1.5 } }
],
"people": [ { "label": "A", "description": "A man in a black T-shirt who demonstrates and gives cues." },
{ "label": "B", "description": "A woman in a green top who attempts the handstand." } ],
"activities": [ { "s": 0, "e": 38.10, "name": "demonstrating a wall handstand", "who": [ "A" ], "text": "A kicks up against the wall and holds while naming the cues." } ],
"speech": { "language": "en", "speakers": [ { "id": 0, "person": "A" }, { "id": 1, "person": "B" } ], "paragraphs": [ ], "silences": [ { "s": 30.10, "e": 33.00 } ] },
"text": { "subtitles": { "s": 0.50, "e": 70.80, "count": 19, "x": 0.50, "y": 0.86, "where": "bottom centre", "matched": 0.93 },
"items": [ { "s": 0, "e": 3.00, "text": "WALL HANDSTAND", "kind": "title", "x": 0.50, "y": 0.20, "style": "bold white, condensed" } ] },
"music": { "heard": [ { "s": 0, "e": 71.20, "text": "medium-tempo electronic track under the voice, no vocals" } ],
"songs": [ { "label": "A", "pieces": [ { "s": 0, "e": 71.20 } ], "seconds": 71.20, "tempo": 124, "key": "A minor", "keyConfidence": 0.71, "recognition": null } ] },
"events": [ { "t": 0, "e": 12.40, "kind": "cut", "text": "shot 1: wide, locked off, 1 person" }, { "t": 12.40, "e": 24.00, "kind": "cut", "text": "shot 2 (punch-in 1.5x): medium, locked off, 1 person" } ]
}