Inside Statchecker: fast searches and automated data recovery

The search starts before you press Search.

One of the best ways to make a search fast is to give it less work to do.

Statchecker preloads common searches. If people regularly ask for the same thing, there’s value in having that work ready ahead of time.

The form itself also helps. As someone fills it out, the system starts trimming the area of the database that needs to be searched. Each choice tells us more about what they’re looking for, so we can narrow the relevant data before they submit the final search.

By the time they press Search, we’ve already had a head start. Common searches have been prepared, and the search has a smaller amount of relevant data to work through. That’s a big part of what makes it feel so quick.

1. PreparePreload common searches.
2. NarrowUse form choices to reduce the search space.
3. SearchWork through the relevant data.

There’s established research behind these ideas. Work on semantic data caching explores reusing data from previous queries to reduce repeated work. The classic Selinger paper on query optimisation explains how a database can use filters, selectivity and access paths to choose less costly ways of answering a query.

AFL: automate the recovery, too.

For AFL, data ingestion is automated. The system brings in the data without me having to manually run each update.

But running a process automatically is only part of the job. I also wanted it to respond when something breaks.

I’ve built automated debugging into the workflow. When an ingestion error comes up, AI agents investigate the problem, change the code to address it and rerun ingestion. The recovery process can move from finding the issue to trying the fix without waiting for me to write the patch.

That’s the bit I find useful. An alert tells me something needs attention. This workflow can act on the problem and run the process again.

NBA: wait until the whole game is ready.

The NBA system goes a step further because it uses multiple data sources.

That creates a different problem: one source arriving doesn’t mean we have everything needed for a game. Some of the data can be there while other pieces are still missing.

So the system tracks whether it has all the required pieces for each game. A game only joins the searchable database once those requirements are met. Until then, it stays out of search.

1. CollectBring in data from multiple sources.
2. CheckTrack the required pieces for each game.
3. PublishMake the game searchable when complete.

This gives the search system a clear rule for when a game is ready. It also means that arriving data and searchable data are treated as separate stages.

Research on automated data-quality verification supports this approach: define the requirements explicitly, then check them automatically as data moves through the pipeline. Completeness is one of those requirements. It doesn’t prove every value is correct, but it does stop “we’ve received something” from being treated as “we have everything we need”.

The NBA pipeline also has automated error handling and AI debugging. Agents can investigate ingestion failures, code a fix and rerun the work. The completeness requirement still determines when a game becomes searchable.

Give the agents a feedback loop.

The useful part of an AI coding agent is what it can do with the result of its own work. It needs somewhere to investigate the failure, make a change and run the process again.

That’s the pattern in Statchecker’s ingestion recovery: an error gives the agent a problem to investigate; the agent changes the code; rerunning ingestion provides feedback on whether the problem has been resolved.

SWE-agent, a research project on automated software engineering, explores agents that navigate code, edit files and execute programs and tests. It provides evidence for this kind of tool-based workflow, while also showing why the tools and feedback available to an agent matter.

In Statchecker, the recovery loop runs without a human approval step in the middle. Reports are emailed to me so I can review what happened. I still have visibility, but I don’t have to be the person moving every step along.

The rerun matters: it gives the recovery process feedback on whether the ingestion problem has actually been resolved.

What I take from this build.

The pieces work together. Preparing common searches avoids repeated work. Narrowing the search early reduces what the database needs to consider. Checking completeness keeps partially assembled games out of search. Automated recovery gives the ingestion process a way to respond when something goes wrong.

For someone using Statchecker, the aim is straightforward: find the stats quickly, with the data ready behind them. Getting there means paying attention to the work that happens before the search, and the work that happens when a background process fails.

That’s the kind of system I like building. A simple experience on the surface, with the detail underneath thought through.