The gust that
fixed the engine.

The feature request was frivolous, and I want to be honest about that up front. Not "improve recall on quiet contacts." Not "reduce false positives on noisy net footage." It was: when the bat hits the ball, draw a cartoon whoosh — a comic gust of wind, streaking away in the direction the shot went, with a little burst of impact lines on the frame of contact.

It is decoration. It exists because a fourteen-year-old sending a net session to their mates wants it to look like something happened. There is no accuracy metric anywhere in the app that this improves.

And it broke everything open.

A gust asks two questions

Here is the thing about drawing a gust of wind. Unlike almost every other thing we build, it cannot be approximately right. It has to answer two questions with hard, specific, non-negotiable answers:

• Where? A gust streams away from the bat, in the direction of the stroke. Point it the wrong way and you've drawn a cover drive going backwards. To know the direction, the app has to know — precisely, in pixels — which part of the picture is the batter, and which way their arms swept.
• When? The burst has to pop on the frame of contact. Not near it. On it. And it has to keep popping on it when the clip is played back in slow motion, where every error is stretched out and magnified.

Both questions sound trivial. The app already finds cricket shots — that's the whole product. Surely it knows when and where they happened?

It did not. Not nearly as well as we thought. And the reason it had got away with it for so long is the most interesting part of this story.

Question one: when, exactly?

Our engine's oldest and strongest instinct is sound. A cricket ball meeting the middle of a bat makes a crack with a very particular shape, and hunting for that shape is how the whole thing started. It works. It has always worked.

So we asked it for the frame of contact, drew the gust there, and watched the results. The gust was firing before the swing had started.

Not by a frame or two. By the better part of a second. To find out how badly, we did the unglamorous thing: I sat down and hand-marked the true moment of contact across several sessions — dozens and dozens of deliveries, scrubbing frame by frame — and compared them to where the engine thought contact was. The answer came back brutally consistent. It was landing early, every time, on every kind of footage, and the gap was big enough that a viewer would never need it pointed out to them.

Then the diagnosis, which is the genuinely counter-intuitive bit:

The loudest sound anywhere near a cricket shot is very often not the cricket shot.

Think about what a microphone twenty metres away actually receives. The bowler's release. Their feet landing. The clunk of a bowling machine. The ball thumping into the net after the shot. Someone's voice. And the contact itself? A well-timed shot in the middle of the bat, at that distance, through open air, can be quieter than all of it. On one net session, the overwhelming majority of the contacts I had marked by hand had no loud spike anywhere near them at all. On the bowling-machine session, not a single one did.

And an engine built by people who trusted their ears will reach for the loudest thing nearby. It could therefore never, by construction, arrive at a moment quieter than its own surroundings. It would step confidently onto the release, or the machine, or the net-thud, and plant its flag there.

The scoreboard could not see the problem

This had been true for months. We had never noticed. Why not?

Because every number we measure ourselves against is relative. Did we pick the right moments? How many real shots did we catch, how many false alarms did we let through, does this build rank the good clips above the bad ones? Those are ranking questions. And a clip window built around an anchor that's most of a second early still contains the shot. It still scores as a hit. Every accuracy metric we had was completely blind to the error, because the error doesn't change any ordering. It only changes where things sit inside the frame of time.

There had been a signal, in hindsight. When we looked at what people did after the app handed them their clips, there was a persistent little habit: everyone nudged their trims earlier. Every time. We had even quietly tuned some of our default clip padding to compensate for it — treating the symptom as a preference, because a preference is what it looks like when your instruments can't see the cause.

A metric asked us whether we were better. The gust asked whether we were right. Those turn out to be extremely different questions, and only one of them has a correct answer.

Question two: where, exactly?

Meanwhile the other half of the gust — the direction — was applying its own pressure, from the opposite side.

You cannot draw wind coming off a bat without knowing where the bat is. Which means knowing where the batter is: not "somewhere in this general area", but a tight region of the picture that contains this person's arms and their swing, and nothing else. And it has to keep being right when the camera gets repositioned between overs, or the batter changes ends, or the phone gets picked up and put down somewhere new — all of which happen constantly in real footage, because real footage is filmed by a friend who also wants to bat.

Two threads of work had already been running on this — sharpening the region the app watches, and letting one recording carry several different batting positions as the camera moves through a session. They had been treated as separate housekeeping. The gust is what made them the same thread: it turned "know exactly which pixels are the batter" from a nice-to-have into a hard requirement, because if you don't know that, there is literally nothing to draw.

And chasing that turned up something counter-intuitive enough that I'll state it as a principle and leave it there. Every instinct says be generous with a region of interest — give it room, don't clip anything off, better safe than sorry. On real footage that instinct is wrong, and it's wrong for a reason worth carrying into any problem of this shape.

So we stopped only listening

The fix, at the altitude I'm willing to describe it, is the least surprising sentence in this post: if the sound can't tell you when the bat met the ball, look. The batter is right there in the picture, doing something extremely distinctive, and unlike the sound they are not competing with a bowling machine. The catch is that all of it has to happen on a phone, on battery, on a video you imported thirty seconds ago — which quietly disqualifies most of the obvious answers before you've finished writing them down.

I'm going to skip the part where I explain exactly how we read it. That's where most of the work went, and it's the piece I'd rather keep. But the two ideas that made it work are general ones, they each cost me a wrong turn first, and they're the useful part anyway.

The most measurable moment is not the decisive one

The instinct is to reach for the biggest, most obvious landmark in the batter's movement and call that contact. It's wrong, in a way that becomes obvious the second you say it out loud: the follow-through is a bigger movement than the strike. The bat travels furthest and fastest after the ball has already gone. Aim at the most conspicuous thing and you land consistently late, out in the finish of the shot — which, played back in slow motion, looks exactly as daft as it sounds.

Contact happens while the swing is still building, and finding it means deliberately declining the most eye-catching thing in the picture — in precisely the same way we had just learned to stop trusting the loudest thing in the sound. Twice in one week, the conspicuous answer was the wrong one. I've stopped thinking that's a coincidence. Conspicuous is what survives being measured carelessly, so conspicuous is what you find first.

The first fix was wrong. The second one taught us something.

I'll include the mistake, because the mistake is where the real idea came from.

The first rule we wrote was the natural one: if the movement peaks much later than the sound, the sound fired early — override it. Simple, and it worked on the clean sessions.

Then it fell over on a play-and-miss. When a batter is beaten, they don't stop — they complete the entire stroke, gloriously, through thin air. So the movement runs on long after the sound, our rule read that gap as proof the sound had been wrong, and it went and dragged a perfectly good answer out into a follow-through of nothing. It was doing its worst damage on exactly the moments — the edges, the beaten shots, the near-misses — that are the most fun to watch.

The rule that replaced it inverts the whole relationship, and this is the idea I'd carry to any other problem:

Don't let one sense override the other. Let it corroborate.

Treat the picture as a witness rather than a replacement. The sound keeps its answer whenever the picture is willing to back it up — and sound, when it is right, is far more precise than anything we can recover from frames, so we have no business touching it. It only loses the argument when the picture flatly contradicts it. Agreement, not arbitration.

Scored against my hand-marked contacts, that version beat the override rule on clean footage and on noisy footage — the two regimes that normally force you to pick a side and sacrifice the other. It is rare to get a change that wins on both. In my experience it only happens when you've stopped tuning and actually understood something.

And then the core algorithm got better

Here's the payoff, and the reason this is a story about the engine and not about a cartoon.

Once you have a trustworthy answer to "was the batter swinging at this moment?", you don't have to only use it for drawing wind. You can ask it of every candidate the engine has picked.

Our hardest remaining false positives were loud things that weren't shots. A shout. A bag dropped. A ball thumped into a net by hand. Something clattering off the stumps. Acoustically they can be indistinguishable from a middled drive — that's exactly why they survived every audio-side fix we ever tried. But they share one property: nobody swung a bat.

So the engine stopped taking loudness at its word. Scored against the clips a real person had sat down and curated by hand, it removed a genuinely meaningful share of the false positives that had survived every previous attempt — and the recall we paid for it was small enough that I checked the number twice. That is a trade we will take every day of the week, and it exists because a cartoon gust of wind needed to know which way to blow.

Two design principles kept it safe, and they're worth stating plainly because they're what stops a clever filter from becoming a disaster:

• "Unknown" must never mean "no." If the swing evidence couldn't be computed for a moment — the region wasn't covered, the footage didn't allow it — the shot is spared, not dropped. An absence of evidence is not evidence of absence, and in a recall-sensitive product that distinction is the whole ballgame.
• It can only ever un-tick, never tick. The filter removes; it never adds. And anything it removes is one tap away from coming back, permanently, for good. The machine gets an opinion. The human gets the vote.
• It knows where it doesn't apply. On footage from a row of practice nets, a loud crack with no swing from your batter is very often a real shot from the next net over. An inference that is sound on one kind of footage can be actively wrong on another, and a filter that doesn't know the difference will confidently delete somebody else's cover drive.

And the gust? The gust is fine now. It fires on the frame it should. It also, along the way, fixed where our clips start and end, where the slow-motion ramp lands, and what the thumbnail shows you.

Art has been doing this for six hundred years

I want to widen the lens, because after this landed I couldn't stop noticing the pattern. It is not a quirk of our little app. The history of technology is full of moments where someone chasing something beautiful — or merely something that looked right — forced a piece of engineering into existence that the engineering itself would never have demanded.

And two more that don't fit in boxes, because they're about something subtler than invention.

One voice in a diner

When Karlheinz Brandenburg's team was perfecting the MP3, they needed a test signal that would expose every lie the codec might tell. They found it in Suzanne Vega singing "Tom's Diner" — a single unaccompanied human voice. Nothing to hide behind. No drums to mask the artefacts. The story goes that they tuned the encoder against that track until it stopped sounding wrong, and the format that reshaped the entire music industry was calibrated on the hardest possible piece of art they could find.

That is exactly what happened to us, in miniature. Precision and recall were our drums — busy enough to hide the artefact. The gust was our unaccompanied voice.

The current runs the other way too

None of this is a claim that art is upstream and technology downstream. The traffic is constant and it runs both directions. Collapsible metal paint tubes — a genuinely boring bit of nineteenth-century manufacturing — let painters leave the studio and work outdoors, and Impressionism happened; Renoir is supposed to have said that without tubes of paint there'd have been no Impressionism at all. Auto-Tune came out of signal processing built for oil prospecting, and reshaped popular music. Photography freed painting from having to be accurate, which is arguably how we got modern art.

It's a loop, not an arrow. But the direction people notice less is the one this post is about: the demands of making something beautiful are a first-rate engineering specification.

Why art keeps doing this to engineering

Having lived through a very small version of it, I think there are three reasons, and they're all about the shape of the requirement rather than anything mystical about creativity.

• Art sets absolute standards, not relative ones. Engineering metrics are almost always comparative: better than baseline, better than last week, top of the leaderboard. Art asks a different question — is it right? — and it will not accept "better than before" as an answer. There's no partial credit on a whoosh that fires at the wrong time.
• The human perceptual system is an absurdly good instrument. We are talking about detectors that notice audio latency of a few thousandths of a second, that catch a single dropped frame, that can tell a face is subtly wrong without being able to say why. Point that instrument at your system and it will find errors your purpose-built measurements were never designed to detect. It's free, it's fast, and it does not care about your confusion matrix.
• Art delivers the whole artefact, not the average. You cannot ship a painting that is correct in aggregate. Every square inch is in the thing you hand over. That forces you to fix the tail cases — the beaten shot, the edge, the odd camera angle — that a mean score lets you round away.

There's a fourth, quieter one. Aesthetic requirements are cheap to judge and impossible to fake. Getting an honest read on whether a detection change helped takes hours of careful measurement and a curated ground truth. Getting an honest read on whether the gust looks stupid takes one glance, from anyone, including people who have never heard of a false positive. That is a fast feedback loop, and fast feedback loops are where engineering quality actually comes from.

Give an engineer a metric and they'll optimise it. Give them something that has to look right, and they'll find out what the metric was hiding.

The frivolous feature

I've written before about the biggest accuracy idea in this project coming from a human hunch rather than a model, and about using an AI agent to build a deterministic engine. This is a third character in the same story, and the one I least expected: the feature that was just for fun turned out to be the most demanding client the engine ever had.

It didn't care how we scored. It didn't accept an average. It asked one question — when and where, exactly? — and it would not stop looking stupid until we answered properly. So we went and learned to watch the batter instead of only listening for them, and the whole thing got better.

Next time someone on your team wants to add the silly visual flourish, let them. Then watch very carefully what it breaks.

Related reading: timbre, onset & luma — the hidden signals of a cricket shot, the idea the AI missed, and designing a cricket video editor around the bat-ball moment. More on the CricCuts blog.