October 5, 2026 - 6 min read

Stopping a point-forecast API from collapsing under peak load

A point forecast cut from gridded output took 10–15 s and crashed at peak. A gateway and on-demand store took 5xx to zero, without a backend rewrite.

Stopping a point-forecast API from collapsing under peak load

Peak hours, before and after

How do you stop a spatial API from collapsing under load when you can't rewrite the backend behind it?

We had to answer that on one platform we run. A forecast for an arbitrary point took 10–15 s to compute. At peak hours those requests piled up until the backend fell over and took the system down with it, and these outages kept coming back.

Since 1 October, peak hours no longer bring it down, and 5xx errors are at zero. The backend was not rewritten. Two things in front of it did this together. The first is an edge gateway that now sees every call, per API key, and gives us traffic monitoring we did not have before. The second is an on-demand store that keeps every valid forecast answer until a new model run makes it obsolete.

The store did not exist before 1 October. It started empty that day, and by 4 October it was answering 43% of the calls on a path that had never had a cache.

This has been running for a week. The people who use the API haven't noticed yet, which is fair: a week is not long enough to notice an outage that stopped happening. In time they might.

Why each answer was expensive

A forecast for an arbitrary coordinate is not stored anywhere. It is cut from gridded model output on every request. The backend opens the latest runs in Zarr, where a single run can be over a gigabyte and a half. It selects the cells around the point and builds the time series across models and variables. That is where the 10–15 s went, and at peak it was more work than the backend could absorb.

The platform already had a store for this: a precalculated list of locations whose forecasts a pipeline computes after each model run and writes to object storage. It works for the places someone thought of in advance. But people don't ask for a list. They ask for points: a site, an address, a coordinate from a device. Every point outside the list went down the slow path, and that is the path that crashed.

Store what people ask for, not what you guess

The on-demand store turns this around. It doesn't try to predict which points matter. When the backend computes a forecast, the gateway keeps the answer, and the next request for that point is served from storage.

The numbers from the first days show why this works. On the path that had no cache before 1 October, the share of calls answered by the new store went:

  • 1 October, its first day: 6%
  • 2 October: 17%
  • 3 October: 41%
  • 4 October: 43%

On 4 October the precalculated list answered 845 calls on that path. The on-demand store, three days old and filled only by real requests, answered 2,325. A list built on guesses covers the places someone expected. A store filled by demand covers the places people actually ask about, and it learns them on its own.

An answer lives until the next model run

Keeping answers is the easy part. The hard part is knowing when a stored forecast stops being true, and this is the idea we would carry to any system that serves answers derived from gridded data.

The usual tool is a fixed lifetime, and for forecast data it forces a bad choice. Make it short to stay fresh, and you recompute answers that hadn't changed, which is the load you were trying to avoid. Make it long to save compute, and you keep serving the previous run's forecast after a new one is out.

The data already tells you the right moment. A stored point forecast is exactly what the backend would compute again, for as long as the model run it came from is the latest. When a new run is available, the answers built on the old one are obsolete. They are deleted, and the next request for that point is computed against the new run and stored again.

So there is no lifetime to tune. The store always serves the newest forecast. It never recomputes a point nobody asked for. Recomputation happens once per point per run, spread over the requests that actually arrive, instead of all at once whenever a timer runs out. The same holds for any product derived from a source that refreshes on a known cycle: an answer is valid for exactly as long as its input is the latest.

One front door

None of this works unless every request passes through one place. API, map and tile traffic now all go through one edge Worker (Cloudflare Workers) in front of the backends. We rolled it out in shadow mode. It resolves each caller's key, counts requests, checks rate limits and records what it would block, but blocks nothing. Every caller became visible per key before any rule was enforced. That visibility is what let us trace each remaining 5xx to its cause instead of guessing.

Two details mattered for spatial traffic:

  • The key travels as a query parameter. Map clients read PMTiles archives with HTTP range requests and fetch raster tiles from {z}/{x}/{y} templates. A query parameter goes along with both, and with plain links, where a header often can't. The gateway folds it into the same key check and strips it from the URL before caching and logging.
  • Seeing everything turns up surprises. Two internal pipeline writes, hundreds of calls a day, turned out to be using a customer's key. That is the kind of thing to find before you enforce anything, not after.

Only complete answers go in

A store in front of gridded data is only as trustworthy as the check that decides what goes into it. It writes only a complete answer, the one the pipeline would have produced for that request. On the day the store was built, partial copies of a model run on two nodes made the backend return forecasts cut to three days instead of the full horizon, with a 200 status. A store in front of that would have kept the short answer and served it to everyone until the next run. So it was deployed switched off, and turned on only after the backend stopped returning short forecasts.

Stored answers are named by a geohash of the location plus the request dimensions, so a repeated request lands on the same object. The gateway also gives these routes a 60-second timeout floor. That way the slow first computation can finish and fill the store, instead of being cut off and repeated by the next caller.

What's next

Most of the calls still computed on that path come from legacy systems that send unusual parameters. One sent its timestamp with a +00:00 offset that the store's validator did not accept, so every one of those requests was computed and none was ever stored. The gateway's traffic monitoring is how we find these, one caller at a time. Then we either fix the caller or have the gateway normalise the request, so that it lands in the store too.

If your API answers points from grids

If your product answers questions about arbitrary points from gridded output and falls over when demand rises, the fix may not be a rewrite. Put one door in front of it and see who is calling. Store what people actually ask for, and only when the answer is complete. Let the next model run, not a timer, decide when that answer stops being true. On this platform that took peak hours from recurring outages to ordinary traffic, with 5xx errors at zero for the week it has been running.

If you have an expensive spatial endpoint that buckles under load, we'd be glad to talk it through.

Keep reading