Our API Took 14 Seconds to Respond. Here Is How We Got It to 2ms

Our marketplace listing reported an average latency of 12,191ms. Not a p99. The average.

Nobody had complained, because nobody had stayed long enough to complain. A developer clicking "Test Endpoint", waiting twelve seconds, and closing the tab does not file a bug report.

This is what was wrong, what we changed, and the two bugs that almost shipped with the fix.

Where fourteen seconds comes from

Every request ran an upstream scraper job synchronously:

const run = await client.actor(ACTOR_ID).call(input);
const { items } = await client.dataset(run.defaultDatasetId).listItems();

That .call() starts a container, waits for it to boot, waits for the scrape, and returns. Cold start dominates. Measuring the same request repeatedly gave 14.7s, then 28.0s on an identical call minutes later. Not just slow: unpredictable, which is worse to design around.

We had a cache already. It was an in-memory LRU with a 500 entry ceiling and a two hour TTL, and it did almost nothing, for two reasons.

It died on every deploy. In-memory means the process restart that ships your code also throws away everything you have paid to fetch.

The keys were too specific. The cache key included every query parameter:

profile:username:20:latest::

So a request for 10 posts missed entirely even when 20 posts for that exact account were sitting in memory. Every limit and sort variant forked into its own entry. The cache was fragmented into near-uselessness by its own key design.

The fix, in three parts

1. Move it to Postgres

Nothing exotic: one table, key, JSONB payload, timestamp.

CREATE TABLE cache_entries (
  key              TEXT PRIMARY KEY,
  data             JSONB NOT NULL,
  created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
  refreshing_until TIMESTAMPTZ
);

The important decision was making the database a soft dependency. If Postgres is unreachable, the service falls back to the in-memory LRU and keeps serving. A cache is an optimisation. It must never be able to take the API down, and a new hard dependency in the request path would have made availability worse while making latency better.

2. Cache supersets, slice on read

Keys no longer include limit. Each entry records the width it was fetched at:

{ coveredLimit: 20, payload: { ...} }

A read is a hit when the stored width covers the request, and the payload gets sliced down. One entry now serves limit=20, limit=10 and limit=5.

The subtle part: coveredLimit stores the limit that was requested, not the number of items that came back. A creator with 5 videos fetched at limit=20 covers every request up to 20. Store 5 and the count can never reach the limit asked for, so that account refetches forever.

3. Stale-while-revalidate with expiring leases

Two TTLs per data class. Past the soft TTL we serve the stale entry immediately and refresh in the background. Past the hard TTL we treat it as absent.

Profile data gets 24 hours soft and 30 days hard. Follower counts do not move meaningfully within a day. Trending hashtag data gets one hour and three hours, because it genuinely does.

A single global TTL either wastes money refetching profiles or serves misleading trend data. There is no correct single number.

Concurrent refreshes are prevented with a lease column rather than an in-process set, so the lease expires. The previous in-memory version had no expiry: a refresh that crashed left the key marked as refreshing until the next restart, and it would never refresh again.

Two bugs that nearly shipped

The ESM import would have crashed the process on boot. pg is CommonJS and exposes no named ESM exports, so import { Pool } from "pg" throws at load time in an ESM package. Not a subtle degradation. A dead process on deploy, caught only because we ran the thing before shipping it.

An integer overflow would have silently disabled the cache every hour. The cleanup job passed a millisecond interval to Postgres:

DELETE FROM cache_entries
 WHERE created_at < now() - ($1::int * interval '1 millisecond')

Thirty days is 2,592,000,000 milliseconds. int4 tops out at 2,147,483,647. So the hourly cleanup would throw, and because that error path marked the store unhealthy, the whole cache would drop to memory-only once an hour. The symptom would have been "the cache does not seem to help much", which is close to undiagnosable without knowing to look.

Two lessons, both boring: use double precision for millisecond spans, and do not let a janitorial failure mark your primary path unhealthy. Pruning failing says nothing about whether reads and writes work.

What it measures now

Same endpoint, same creator, back to back:

That is roughly 7,000x. More usefully, a cached account no longer waits at all, and after the soft TTL expires it still serves instantly while refreshing behind the request.

The superset change compounds it. A trace of three calls at limit=20, 10 and 5 used to mean three upstream fetches and three cache rows. It is now one of each.

What we would do differently

Measure before optimising anything else. We now record hit and miss counts and surface them on an internal dashboard. Before that, the hit rate was technically observable in logs, which in practice meant nobody observed it.

Treat the cache key as a design decision. Including every parameter felt safe and correct. It quietly fragmented the cache into single-use entries, and it never looked like a bug because every individual response was right.

Run it before shipping it. Both near-miss bugs were caught by executing the code, not by review or types. The overflow in particular would have passed any code review, because the query is obviously correct until you know the magnitude of the number going into it.

---

The API this runs on is TikTok Data Pro: public TikTok profile and hashtag data as clean JSON, with 25 free calls and no credit card.