Semantic Diff

#9171 Compile only the modules that read a changed interface in OCaml rewatch

4 commits

ac7357822edb14ab7f695dcacff864e38205512e..pr-9171

a48a7c288 · tsnobip + Claude · Nov 02, 2026, 13:57 UTC

Number identifiers in saved interfaces from the request baseline

Why

A saved interface took its identifiers' stamps from the compilation's counter, which reading imported interfaces also advances. An interface whose signature didn't change was saved with other stamps, and so another CRC, whenever an interface it reads grew: adding a value to a module changed the CRC of the modules matching on its constructors, and a build system comparing CRCs had to rebuild their dependents.

The substitution that renames a signature's identifiers for saving now numbers them from the request baseline, then lets the counter continue after both.

modifiedInterface User.cmi after let extra = 2 is added to Shape

// Shape.restype t = A | B({x: int})let make = () => A // User.reslet value = switch Shape.make() {| A => 1| B({x}) => x}
text
Before
After
CRC recorded for Shape changedCRC recorded for Shape changedCRC of User changedCRC of User unchanged

modifiedInterface 2000-module synthetic project, after adding a value to a module with 5 direct dependents

text
Before
After
interfaces whose CRC changed 3interfaces whose CRC changed 1

Review

  • Can an identifier numbered from the baseline be confused with another? The predefined identifiers all come before the baseline, and every other identifier in a signature being saved is bound in it and renamed by this substitution. Reading the interface back renames them again, and the counter continues past both for the rest of the compilation.
9a85528f3 · tsnobip + Claude · Nov 02, 2026, 14:20 UTC

Let a compilation record only the interfaces it reads

Why

An interface records the CRCs of the interfaces its module read and, through them, of every interface below: loading an interface records all those it lists. A build system then cannot tell whether a module read an interface below it, so it rebuilds every module below a changed one. It cannot keep one either: a module left with the old CRC of an interface it never read fails the consistency check of the next module that reads both.

With the internal flag -bs-record-read-imports, loading an interface still checks the CRCs it lists against the others, but the compilation records only the interfaces it read. Without the flag nothing changes: Rust rewatch rebuilds the direct dependents of a .cmi whose bytes changed, and relies on interfaces changing whenever one below them does to reach modules further down.

modifiedInterface Top.cmi

// Base.restype t = {value: int}let make = value => {value: value} // Middle.reslet base = Base.make(1)let count = 2 // Top.reslet count = Middle.count
text
Before
After with -bs-record-read-imports
CRCs recorded Top, Middle, BaseCRCs recorded Top, Middle

Review

  • Can a module that did not read an interface depend on it? What its compiled code depends on of a type, the type checker had to look at, which reads the interface that defines it. A module that only passes a value on, as let x = Middle.base, compiles the same whatever its type.
  • Do interfaces built with and without the flag meet? Rust and OCaml rewatch record different compiler hashes in compiler-info.json, so switching between them cleans the build first.
84a5944fa · tsnobip + Claude · Nov 02, 2026, 14:20 UTC

Compile only the modules that read a changed interface, in OCaml rewatch

Why

A module's interface changed its bytes whenever an interface below it changed, and OCaml rewatch compiled the dependents of every interface whose bytes changed. Adding one value to a module therefore compiled every module below it: 349 modules of a synthetic 2000-module project, though only 6 read the interface that changed.

Publishing an interface now compares its own CRC and flags with the published one. An interface that only records other CRCs for its imports is still written, but keeps its modification time and leaves its dependents alone. Compile requests pass -bs-record-read-imports, so an interface records only what its module read.

A module that is not dirty, but has an interface below it that is newer than its .cmt, is checked on a worker when the scheduler gets to it: it compiles if an interface it read has another CRC than the one it recorded, and otherwise its .cmt gets a new time. A build also starts with the modules for which that holds, which a stopped build may not have reached.

modifiedBuild adding a value to a module with 5 direct dependents, synthetic 2000-module project, median of 7, Apple M4 Pro

text
Before
After
modules compiled 349wall time 0.371 smodules compiled 6wall time 0.160 s

Review

  • Can a module that read a changed interface be missed? That interface is below it in the dependency graph, which reaches every interface that the ones it names refer to, so it is newer than the module's .cmt and the module is checked; the module recorded its CRC because it read it. A test changes how Shape represents a constructor that User matches on through Middle's types, and User compiles again.
  • Why give a module found current a new .cmt time? The changed interface below it would otherwise make every later build check it again.
  • What does a check cost? Reading the module's .cmi, and for one with an interface file its .cmt, about 0.15 ms to unmarshal. The 0.16 s above are 55 ms more than a build after editing a leaf module, for compiling the 5 direct dependents and checking the 989 other modules below the changed one.
e23be358e · tsnobip + Claude · Nov 03, 2026, 08:51 UTC

Check imports of interfaces missing at a build's start, in OCaml rewatch

Why

An import check looked a module's recorded imports up in the interfaces published when the build's first check ran, and skipped the units it didn't find there as coming from outside the build. When a module's published .cmi was missing at the start, as after a deletion, and another module's check ran before it was published again, its dependents skipped it. A dependent that reached it only through a module whose CRC stayed the same then kept its stale JavaScript, and its .cmt got a new time, so later builds kept it too.

Every scheduled module's published .cmi is now in the lookup, whether or not it exists yet, and one that can't be read counts as changed. A module whose interface is missing at the start is dirty, so it is scheduled, and it is published before any module that read it is checked.

Build after deleting lib/ocaml/Shape.cmi and changing Shape's representation, with User reaching Shape only through Middle, and Other checked from the start (new case in rewatch-ocaml/tests/run.sh):

text
Before
After
modules compiled 2 (Shape, Middle)User.js stale, 3 runs out of 3modules compiled 3 (Shape, Middle, User)User.js rebuilt, 3 runs out of 3