Add ci/selftest.sh: prove the checks can still fail

Three rounds of fixes to this kit, and each round found the previous round's
blind spot: a check that examined nothing, then a check that examined
everything with a pattern that could not match, then a flag that made one rule
skip binary files without a word. None of them were visible from a green run,
and each was found by a sibling plugin rather than by anything here.

So the battery stops being a shell loop somebody retypes and becomes a script
CI runs. 26 assertions: 13 credential shapes that must be flagged, 6 lookalikes
that must stay quiet, the direct-access count, the four text-domain verdicts,
the annotated path being a real file, and a plugin.conf with its floors
stripped failing to load.

🔴 It asserts on the thing that has CONSEQUENCES — the exit status of the
script CI runs, and the specific failure message. Never a filename, a count, or
a log line that merely sits near the signal. The previous ad-hoc battery
asserted "the fixture's filename appears in the output", which passes on grep's
own `binary file matches` diagnostic, because that diagnostic prints the
filename in exactly the place a real failure would. A sibling plugin recorded
two binary cases as PASS on that basis. The probe had the same defect as the
check it was testing.

The self-test was then checked to go red: making the AKIA pattern inert fails
exactly one assertion, naming the shape.

Its first run failed an assertion, and the fixture was at fault rather than the
product. This script sources ci/lib.sh, so the floors are in the environment
and a plain subshell inherits them — ci_load_conf then passed over a
plugin.conf that does not define them. Fixed with an explicit unset, and the
reason is inline so nobody weakens the assertion instead. A failing probe is
evidence about the probe until you have looked at it.

Wired into the static job in BOTH workflow roots, byte-identical: .gitea hides
.github completely rather than merging it, so a step added to one root only
would not run on Gitea and would look exactly like a runner that had not picked
the job up. ci/workflows-in-sync.sh passes.

Excluded from the release zip, and it plants nothing in the real tree — the
fixtures go into a throwaway copy under mktemp, several being credential-shaped
by definition.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
kristianwind
2026-09-17 00:30:52 +02:00
co-authored by Claude Opus 5
parent 3f85642235
commit 0a58661864
3 changed files with 179 additions and 0 deletions
+8
View File
@@ -30,6 +30,14 @@ jobs:
- name: Lint, hygiene, versions and compatibility headers
run: ci/static.sh
# Runs static.sh again against planted fixtures and asserts it goes RED.
# Every check in ci/ has been silently hollow at least once — reporting
# success having examined nothing, then examining everything with a
# pattern that could not match. Neither is visible from a green run, so
# the only defence is a thing that must be found.
- name: The checks can actually fail
run: ci/selftest.sh
- name: The release zip is buildable and complete
run: ci/build-zip.sh