CI from 485 s to 108 s without changing a test

A small monorepo I work on had a first-pass CI. One GitHub Actions job that installs, builds, typechecks, lints and runs every test suite in sequence.

It took four minutes. A week of adding tests later it took eight, and my theory was slow tests. Probably the database suite recreating its schema too often.

I was half right about the second part and wrong about why it mattered. The runner had two cores and I had never looked.

This is one afternoon with an AI agent, which did the measuring and the legwork while I decided what the numbers meant. 485 seconds to 108.

Where the eight minutes went

The first instruction to the agent was "pull the per-step timings off the last green run and tell me where the eight minutes go". Not "speed up the tests". I had a theory and I didn't want to act on it yet.

Inside the test step: the backend suite 41 s, the Next.js app 41 s, nine small libraries about 40 s between them, and the database package 159 s. Thirty-three test files, each printing its PASS line four or five seconds after the one before.

Jest runs test files in parallel by default and these were serial. Its default worker count is cores − 1, and a GitHub-hosted ubuntu-latest runner has four cores for a public repository. For a private one it has two. So every jest suite in the repo had been running on a single worker since the day the workflow was written, and nothing in the workflow file says so.

That one took a minute to find and I'd been staring past it for a week.

The database suite had its own problem on top. Each of its thirty-three files started a postgres:16 via testcontainers, ran every migration, ran its tests, then stopped the container. Timed apart: a test file with no container costs about 0.75 s of jest overhead, the same file with a container about 3 s. Call it 1.5 s a file, times thirty-three, in sequence. Roughly two thirds of that suite was starting Postgres.

Measuring before guessing

What each fix was worth, measured the way the runner runs. One worker, cold jest cache.

Skipping the type-check in ts-jest came back at ten percent, so I set it aside. That was a mistake and I'll come back to it. Parallelism was worth half or more, but the runner had no cores to give.

So the container churn went first.

One Postgres, one template, one database per file

Postgres can create a database as a copy of another, CREATE DATABASE … TEMPLATE …, in about a hundred milliseconds. A jest globalSetup starts one container per run and migrates the default database once. Each test file gets a copy under a random name and drops it when done.

The constraint I gave the agent: the thirty-three test files do not change. Same helper, same signature, same isolation between files.

// globalSetup: one container, migrated once, then closed so it can be a template
const container = await new PostgreSqlContainer('postgres:16').start();
await migrateToLatest(dbFor(container.getConnectionUri()));
process.env.TEST_PG_URI = container.getConnectionUri();      // workers inherit env
process.env.TEST_PG_TEMPLATE = container.getDatabase();
 
// startTestDb, called by each file's beforeAll
const name = `test_${randomUUID().replace(/-/g, '')}`;
await sql`CREATE DATABASE ${sql.id(name)} TEMPLATE ${sql.id(template)}`.execute(admin);
// … connect to `name`, hand it to the file; stopTestDb does DROP DATABASE … WITH (FORCE)

A template can't be copied while anything is connected to it, so the setup destroys its own client before it returns. With more than one worker that would otherwise fail intermittently, which is the kind of bug you find at the worst possible time.

Single worker, 69 s to 10.5 s. Four workers, 9.5 s. Coverage unchanged at 97% lines, 88% branches.

Four runners instead of four cores

If the runner won't give me cores, I can have four runners. The one job became four - typecheck, lint, the database suite, everything else - behind a fifth that only needs: them.

The fifth is named test, the old job's name, because the branch ruleset and the release script both look for a check by that name. A pipeline refactor shouldn't need a repository settings change. If it does, it's two changes and you've made the rollback harder.

test:
  needs: [typecheck, lint, test-db, test-unit]
  if: always()   # skipped ≠ failed: a skipped required check blocks the merge silently
  steps:
    - run: |
        echo '${{ toJSON(needs) }}' | jq -e 'all(.[]; .result == "success")' > /dev/null

The shared setup went into a composite action. The typecheck job also stopped running tsc --noEmit over the libraries, because the build step had just run tsc over the same files with the same config. 30 s of that 52 s step was a repeat.

Two small ones

The Next.js app's Vitest run was 41 s, and its own summary said tests 712ms, environment 16.36s, collect 10.51s. The config set environment: 'jsdom' for every file. Three of the thirty-three test files render anything. Two Vitest projects - .tsx tests in jsdom, .ts tests in node - made it a rule of the file extension. Environment time 16.4 s to 2.3 s.

The nine small libraries were nine jest processes, four of them for packages with no tests at all. One root config listing them as projects made it one process.

One gotcha there. The root collectCoverageFrom is matched relative to each project's own root, so */src/** matched nothing and coverage reported 0% until it became src/**. The test run was green the whole time. What caught it was diffing the coverage table row for row against the previous run.

Being wrong about ts-jest

After the first round the wall time was 209 s.

Every measurement the agent had taken was on my laptop, and on those the ts-jest change wasn't worth much. On the runner, the database suite I'd measured at 10 s locally took 80 s, with two files at 22 s each that take 2 s on an M-series Mac.

The runner's CPU is three to five times slower at the work ts-jest does, which is compiling and type-checking every test file against the whole program.

ts-jest 29.4 reads isolatedModules from the package's tsconfig and, when set, transpiles each file without type-checking it. Safe here, because the typecheck job does that once. Every package already passed tsc under that flag, so it was one line in eleven tsconfigs.

On the runner: backend suite 46 s to 18 s, small libraries 25 s to 7 s, database suite 80 s to 54 s.

The fix I set aside at ten percent was worth half.

Caching what doesn't change

Setup was now about 90 s of every job, four times over. Two caches.

The built libraries (dist/), keyed on a hash of their sources, tsconfigs and lockfile. Not with actions/cache, which saves in a post-step even when the job failed. A broken build would be cached, and since the build doubles as the libraries' typecheck, every later run with the same sources would skip both and go green. That one would have been genuinely nasty to debug. actions/cache/restore and save as separate steps, save after the build, only on success.

Then node_modules, 1.4 GB across four directories and 240 MB compressed, keyed on the lockfile, linker config, Node version and every workspace package.json. A restore-keys prefix means a lockfile change restores the previous tree and yarn install reconciles it, rather than linking from nothing.

- id: deps
  uses: actions/cache/restore@v4
  with: { path: node_modules, key: deps-${{ hashFiles('yarn.lock', '**/package.json') }}, restore-keys: deps- }
- if: steps.deps.outputs.cache-hit != 'true'
  run: yarn install
- if: steps.deps.outputs.cache-hit != 'true'
  uses: actions/cache/save@v4
  with: { path: node_modules, key: ${{ steps.deps.outputs.cache-primary-key }} }

Restoring 240 MB takes about 17 s against roughly 40 s for the install. The dist restore is under a second.

Eight minutes to under two

A cold run with both caches missing is about 145 s. Billable minutes went from around 8 a run to 6 or 7.

Three things that didn't go the way I expected. Laptop measurements understated the CPU-bound fixes by three to five times and were about right for the I/O-bound one - which machine counts was a question for the start, not the second round. Run-to-run variance on the runner is ±10 s per step, and two runs of the same commit disagreed by that much in both directions. And the database suite is still 54 s on the runner against 10 s locally; two files are about 20 s each there, the largest tables' round-trip tests, and they're CPU-bound. I left them.

Eight minutes to under two, merged and released the same evening, no test changed. The first run on main after the merge was cold, because GitHub scopes cache reads to the base branch plus the current branch, so main couldn't see the caches the PR had saved until it had saved its own.

What the agent actually changed was the cost of measuring. Step timings out of a CI log, a test file timed with and without its container, a config flag A/B'd on a cold cache, two coverage tables diffed. Each of those is ten minutes I'd normally have talked myself out of, and it did them in seconds, so I did all of them. Most of the afternoon went on choosing the next measurement. The one I got wrong was believing a laptop number about a runner.

Still open: the two slow database files, and yarn install at 40 s a job on a cache miss. Neither is on the critical path of an ordinary PR, so neither is bothering me enough yet.

Links

• About GitHub-hosted runners — standard runners for private repositories
• Jest CLI: --maxWorkers and configuration: projects
• Testcontainers for Node
• PostgreSQL: template databases
• ts-jest: isolatedModules
• Vitest: projects and environment
• actions/cache/restore and actions/cache/save