When you build a language with two ways to run a program, comparing them catches disagreement but not shared mistakes. For Chuks v0.2.0-rc.3, we added tests that hold programs to a known answer, generated about 7,800 of them from every combination of three language features, and committed to shipping with no further release candidate.
The problem with checking a thing against its twin
Chuks runs programs two ways. A bytecode VM gives instant startup while you develop. A native compiler produces the binary you ship. The promise is simple: the same program prints the same thing in both.
The natural test is to run everything both ways and compare. It is also a trap. Imagine the two implementations share a front end, the part that reads your code and decides what it means. If that part makes a mistake, both back ends faithfully execute the mistake, agree with each other, and the test passes.
We hit this for real. Both back ends once agreed on a value that should have been a typed record and was not. Agreement told us nothing, because both were wrong the same way.
Test against the answer, not the twin
The fix is to give each test a right answer that does not come from the thing being tested. Chuks v0.2.0-rc.3 does this three ways.
Borrow answers from elsewhere. We translated real compiler regression tests into Chuks. Each carries the output it should produce. The corpus now stands at 450 programs, and every one passes on both the VM and the native build with identical output.
Generate the combinations a person would not write. Thirteen generators, one per area of the language: generics, tasks, records, closures, errors, classes, unions, collections, modules, narrowing, return inference, type agreement and name collisions. A generator takes the features of its area and builds programs from every combination of three. That is about 7,800 programs, far more than anyone writes by hand, and every one is held to a rule instead of only to the other back end.
Build real software. Two large projects, Valet and CodeStudio, about 46,000 lines, were compiled natively as part of the work. Real programs lean on awkward seams in a way tidy generated code can miss.
What the new tests found
A lot, and not trivial things. Cancelled tasks that kept running to the end of their body. A lock that stayed held after an error inside it, so the next attempt to take it waited forever. a || b returning one of its operands on the VM and a plain bool in the native build. An HTTP parser that accepted framing the specification forbids, which is the kind of gap request smuggling lives in.
Each of those is now fixed, and each fix became a permanent check, so the same mistake cannot come back unnoticed.
When is a candidate the last one?
After the fixes, the thirteen generators passed on the full tree. A nightly job runs them at higher strength than the commit gate, and it has already caught a real regression within a day of it appearing.
That is the evidence we used for a decision most projects avoid: rc.3 is the last candidate. There will be no rc.4. If testing finds something now, it is fixed in place, and v0.2.0 is published from this build.
One clean run is a strong signal, not a guarantee. No gate proves the absence of bugs, only the absence of the bugs it knows how to look for. That is why the release asks for the one thing we cannot generate ourselves: programs we did not write.
Try to prove us wrong
chuks upgrade --prerelease
chuks run main.chuks
chuks build main.chuks && ./build/main
Run your own program both ways. If the output differs, open an issue with chuks --version and a minimal reproduction.