Timbre, onset & luma:
the hidden signals of a cricket shot

Ask a person to spot a cricket shot in a video and they don't think about it — they just know. The crack sounds right, the batter's body did the thing batters do, the ball flew. Teaching a phone to do the same means naming every one of those instincts and turning it into a number it can compute in milliseconds. This piece is a tour of those ideas: the physics of the sound, the geometry of the motion, and the small set of concepts that let a device tell a middled cover drive from a shout, a passing car, or the net two lanes over.

It's a concepts piece, not a manual. You'll leave knowing what timbre, onset and the sound envelope are; what a luma grid is and why brightness beats colour; how a motion heatmap learns where a recording's cricket lives; and how the stance → trigger → swing → follow-through of a stroke leaves a signature a machine can recognise. What the app does with those signals — the exact recipe — is the part we keep in the kitchen.

Sound is a wave, and a wave has two knobs

Everything you hear is air pressure wobbling up and down. Draw that wobble against time and you get a waveform. A microphone doesn't record "a bat hitting a ball" — it records thousands of tiny pressure readings every second, and everything the app knows about the audio, it has to squeeze out of that stream of numbers.

A pure wave really only has two knobs. Amplitude is how far the wave swings from the centre — that's loudness. Frequency is how fast it wobbles, measured in hertz (wobbles per second) — that's pitch. A low rumble is a slow, big wave; a sharp tink is a fast, small one.

Why loudness alone lies

The obvious idea — "a shot is loud, so watch for loud" — falls apart in about ten seconds of real footage. A teammate shouting "catch it!", a gate clanging, a car door, applause: all loud, none of them your shot. And a soft nudge to third man can be quieter than the chatter around it. Loudness is a real clue, but on its own it's a liar. The pressure that makes a phone able to separate these is that a bat-on-ball doesn't just have a loudness — it has a character. Two ideas unlock that character: onset and timbre.

Onset & attack: the fingerprint of a hit

Every sound has a life story in time — it begins, it swells or spikes, it fades. That story is the envelope: the outline of the sound's loudness over its lifetime. Classic synthesis breaks it into four stages — attack (silence up to the peak), decay (the drop after the peak), sustain (a held level), and release (the fade to silence). The onset is simply the instant the sound begins — and onset detection is the whole art of finding those instants in a stream.

Here's the thing that makes cricket tractable: a bat striking a ball is almost all attack. It goes from nothing to peak in a flash and is basically gone again — a short, violent burst engineers call a transient. A voice is the opposite: it eases in and holds. Same peak loudness, utterly different shape. That sharpness of the onset — how abruptly the sound arrives — is one of the most honest ways to tell a strike from a sustained noise.

Timbre: the colour of a sound

Play a middle C on a piano and on a violin at exactly the same loudness. You still know instantly which is which. The quality that lets you do that — everything about a sound other than its pitch and loudness — is called timbre (say "TAM-ber"). It's the "colour" or texture of a sound, and it is the single richest clue for telling one kind of event from another.

Where does timbre come from? Almost no real sound is a single pure frequency. It's a stack of frequencies blended together, and the recipe of that stack is the timbre. A mathematical tool called the Fourier transform unmixes any sound into that recipe — its spectrum: how much energy sits at each frequency, from deep bass to airy treble.

One handy summary of a spectrum is its spectral centroid — the "centre of gravity" of all that energy. When the centroid sits high, the sound is bright and sharp (a clean crack, a snare); when it sits low, the sound is dark and dull (a thud, a rumble). Brightness is exactly the sort of thing a well-timed shot and a dead-bat block differ on, and it's blind to how loud the mic happened to make either one.

Listening one band at a time

Because different events live in different parts of the spectrum, a classic move is the band-pass filter: keep one slice of frequencies and throw the rest away. Want the sharp "crack" band? Filter to the highs and the rumble of wind and footsteps mostly vanishes. Want to know if someone's talking? Look at the voice band. Slicing the sound into bands lets a phone ask several small, clean questions instead of one muddy one — "is there a sharp high-frequency spike?", "is there a low build-up?", "is there voice energy?" — and weigh the answers.

There's one more trick worth naming, because it's a lovely example of using the right tool for the smallest question. Telling apart a shout from a crack when both are loud in the voice band is genuinely hard with hand-written rules — so a tiny, purpose-built neural voice-activity detector handles exactly that one call: "is this human speech, yes or no?" It's a beautiful bit of restraint — the smallest possible model, asked the smallest possible question. We meet it, and the pose and speech models, in the interactive deep dive.

Seeing in brightness: the luma grid

Now the eyes. Video is a firehose — millions of coloured pixels, dozens of times a second. Trying to reason over every pixel of every frame on a phone battery is a losing game. The first, freeing realisation is that for motion you barely need detail, and you almost never need colour.

Every colour pixel can be split into luma — its brightness — and chroma — its colour. For spotting movement, brightness carries nearly all the useful information: a bat sweeping through a patch of the frame changes how bright that patch is, whatever colour anyone's wearing. So the app throws colour away and keeps luma. Then it goes further and shrinks the frame to a coarse grid — a small mosaic of brightness cells, each one the average glow of a little tile of the picture. A whole video frame collapses to a tidy little grid of numbers.

Motion energy: what changed between two frames

Once each frame is a little grid of brightness numbers, motion becomes subtraction. Line up this frame's grid with the last one and see which cells changed. Add up all that change and you get a single number for the moment — motion energy: high when a lot of the picture is moving, near zero when the scene is still. Keep the per-cell changes and you also know where the movement was.

This is gloriously cheap and hard to fool by noise, but on its own it's naïve — it only knows how much moved, not what. A batter driving and a spectator wandering across the frame can produce the same amount of motion energy. Sound familiar? It's the visual twin of the loudness problem. The fix is the prettiest idea in the whole system.

The heatmap: learning where the cricket lives

Most cricket footage is shot on a phone propped against a bag or clamped to a fence — a fixed camera. And a fixed camera hands you a gift: on a fixed frame, real cricket is astonishingly repetitive in space. Ball after ball, the bowler runs through the same strip of the picture, the batter sets up in the same box, the swing sweeps the same arc. The important things keep happening in the same places.

So let the recording teach you where its own action is. Accumulate motion, cell by cell, across the whole session, and you get a heatmap — a map where the squares that light up again and again are the cricket. The random stuff — someone strolling through during a changeover, a stray ball from the next net, a dog on the outfield — is a novelty: it happens rarely, and it happens somewhere else. Now every candidate moment gets a new question asked of it: did this happen where cricket keeps happening here, or off in a corner the pattern says it shouldn't? That single reframe — from "how much moved?" to "did it move where the action lives?" — was one of the biggest accuracy leaps in the whole project. It has its own story: the idea the AI missed.

The choreography of a stroke: stance, trigger, swing, follow-through

The heatmap tells you where. The last piece is what shape a real shot has in time — because a batting stroke isn't one motion, it's a little dance with a fixed running order, and that order is a signature.

• Stance — the still, set position before the ball. On the motion trace it's a calm, quiet stretch: the baseline the rest is measured against.
• Trigger movement — the small, preparatory shift a batter makes just as the bowler releases (a back-and-across step, a forward press). A subtle stir after the calm — the tell that something is about to happen.
• Backlift & downswing — the bat is lifted and brought down to meet the ball. This is the bat swing: a fast, concentrated burst of motion, and the moment the audio onset — the crack — should line up with.
• Follow-through — the swing doesn't stop at contact; it carries on and the body rotates. That trailing sweep of motion after the peak is part of the signature, and it helps confirm a genuine stroke even when the strike itself was quiet.

Put those in a row and a real delivery has an unmistakable arc: quiet → stir → sharp burst → trailing sweep, with a crisp audio onset sitting right on the burst. A spectator walking across the frame has none of that structure — just a flat, wandering smear of motion with no crack on top. Reading the shape and order of the motion, not merely its size, is what separates a stroke from ambient bustle.

Two senses beat one

Neither sense is enough alone, and that's the point. Audio is brilliant at when — the crack pins the exact instant of contact to a fraction of a second, far sharper than any camera running at a handful of frames a second could. Motion is brilliant at what and where — it knows whether that instant sits inside the session's real action and whether a body actually did the thing a batter does. Each one covers the other's blind spot.

Line the two up — a sharp onset, of the right timbre, landing on a motion burst, in a hot cell of the heatmap, with a stance-and-swing shape around it — and you have something a single shout or a single stroller simply cannot fake. The confidence comes from agreement across independent senses, which is exactly how you tell a real thing from a coincidence.

And all of it, on your phone

Here's the quietly remarkable part: none of this needs a data centre. A spectrum, a band-pass filter, a brightness grid, a frame subtraction, a running heatmap — these are cheap. The one or two places that truly benefit from a neural network use a tiny, purpose-built one that answers a single narrow question. So the whole analysis — listening, watching, scoring, cutting — runs in the seconds after you tap, offline, on a phone that's years old, with your footage never leaving your hand. That's not a compromise forced by the phone; it's the whole design philosophy, and it has its own piece: why the future of AI is small models on the edge.

Next time you hear that unmistakable tok of a middled shot, you'll know what a phone hears too: a razor-sharp onset, a bright broadband timbre, landing dead on a burst of motion, right where the action always lives. A hundred-year-old game, read through a century of signal processing — and cut into a highlight reel before you've put the bat down.

Keep going: the interactive deep dive lets you poke live demos of these ideas · the idea the AI missed is the full story of the heatmap · small models on the edge is why it all runs on your phone. More on the CricCuts blog.