#9282 Benchmark generated projects and time a chain of modules in CI

70da3d1bc43caa96214c942f72db9296f83a9db6..pr-9282

ea9680b96 · JonoPrest · Nov 09, 2026, 14:22 UTC

Count the build's allocations on generated projects in the benchmarks

Why

The benchmarks compile runtime and Belt files one at a time, whose dependency graphs are small, and build tests/tests, whose modules use a few small modules. Costs that grow with how many modules a build has or loads barely show: a change that avoids copying the signatures of large dependencies that a module hardly reads moved them by 0.4%, and opening the map of a namespaced package, which is quadratic in its modules per build, doesn't run at all.

With --project, measure now also builds two projects that the script generates, clean and after an edit, on one worker domain and counting allocated words like tests/tests:

- WideDeps: 240 modules that each use one or two values of four modules with 600 types and 1,200 values each. The edit adds a value to one of them, which compiles its users again. - Namespace: a package with "namespace": true and 1,000 one-line modules. The edit changes a module that nothing uses.

Their counts are larger than all the others together, so the report adds them up on a line of their own, Generated projects, and In total covers the other benchmarks as before. Each has its row and threshold like any other benchmark. Every module of a project build loads the runtime's interfaces, so the inputs of the generated projects and of tests/tests now include a hash of the runtime sources: a pull request that changes the runtime marks them as inputs changed rather than as regressions.

Locally, the signature copy pull requests take WideDeps clean from 505M to 333M words and its edit from 232M to 146M, and building the namespace map on demand takes Namespace clean from 611M to 203M; the other benchmarks' counts are unchanged. The builds of tests/tests now share countBuild and editModule with the generated projects.

66b8aa38f · JonoPrest · Nov 09, 2026, 14:27 UTC

Time a chain of modules with the base's and the head's build systems

Why

A change to the build's scheduling, such as starting a module once the interfaces it reads are final, does the same work sooner. Counting allocated words and instructions can only show what such a change costs, never what it saves.

time <base checkout> <head checkout> <out.json> builds a generated project of 150 modules in a chain, each using the one before it, and a module beside each, clean with the two build systems in turn on the default number of worker domains: one build each that isn't counted, then 10 rounds, the order alternating each round. compare --timings adds a section to the report with the base's and the head's median times and the median of the head's time over the base's in the same round, with a 95% confidence interval from a percentile bootstrap of the rounds with a fixed seed. The section never counts as a regression or an improvement; when the times' file is missing or can't be read, it says they weren't measured. In CI, the step runs after both measurements, each build may take 2 minutes and the whole command 10, and a failure only leaves the times out.

Locally, with load around 5 to 9 on 8 cores, the ratio was 0.695 (0.646 to 0.825) for the pull request that starts dependents once a module's interface is final, 0.991 (0.948 to 1.055) for one that only changes namespaced packages, and 0.965 (0.921 to 1.078) with the same build system on both sides. The command took about 45 seconds.