Kodlo/ue-toolchain
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(skills): ship ue-design-skills bundle, licensing and delivery gate

Browse Source 
Phase 0 of the handoff plan, as a marketplace rather than a flat skills/
directory. Content moved out of the LyraResearch archive and depersonalised:
addresses stay in the archive, recipes ship.

- plugins/ue-design-skills: 17 skills, 232 failure-mode entries, each with the
  six required fields; catalog.json as the harness-neutral source of truth and
  .claude-plugin/ as one adapter over it.
- _gate: 16 rules, one poisoned fixture per rule, plus surface coverage so a
  declared file cannot silently miss the line rules.
- ADR-0002 (harness-neutral bundle behind a marketplace) and ADR-0003 (split
  licensing: CC BY-ND 4.0 prose, Apache-2.0 code and metadata).
- LICENSE files at both levels, CONTRIBUTING.md, docs/licensing-options.md as
  the material the licence decision grew from.

Verified: gate.py 0 violations; test_gate.py 16/16 rules redden on their
fixtures with a clean baseline and 2 root files reaching the line rules.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
MagentaDolphin
2026-09-05 23:48:55 +07:00
parent 80ea7b03a9
commit ecd87ac96d
67 changed files with 21630 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plugins/ue-design-skills/skills/ue-evidence-discipline/SKILL.md
+242
View File
@@ -0,0 +1,242 @@
---
name: ue-evidence-discipline
description: >-
How to produce and check claims about an Unreal Engine codebase so they can be
trusted later: separating measured from derived from open, why an empty search
result proves nothing, verifying a headline claim yourself before repeating
it, treating a delegated report as raw material, and writing detection recipes
that have been run. Use when auditing a codebase, reviewing someone else's
audit, adopting a reference project, or writing any document that will be
quoted back at you.
---
# Evidence discipline for codebase work
The invariant this whole bundle rests on:
> A claim without a source you can point at is not a result. A skill that is
> confident and wrong is worse than a missing one.
Every other skill here is normative — it tells you how to design something. This
one is methodological: it tells you how to know whether what you just wrote is
true. Read it once before using the others, and again before writing an audit
someone else will act on.
Ten ways this discipline fails in practice, each with a detection recipe:
[failure modes](references/failure-modes.md). They are all mistakes made during
the work that produced this bundle, which is the only reason they are specific
enough to be useful.
---
## 1. Three kinds of statement, never mixed in one sentence
| Marker | Meaning | Test |
|---|---|---|
| **measured** | read in the source, config or a live object | someone else can open the same place and see it |
| **derived** | a conclusion from measured facts plus engine semantics | the facts are cited and the inference is stated |
| **open** | not answerable with the tools used | says what would answer it |
**Do not mix two of these in one sentence.** The overwhelming majority of false
conclusions in codebase work are born exactly there: a measured fact and an
inference joined by a comma, where the reader inherits the confidence of the
first half for the second.
Concretely, this is wrong:
> The pool is only configured for one tier, so the other tiers are broken.
and this is right:
> The pool is configured for one tier **[measured]**. The other tiers therefore
> inherit engine values **[derived]** — whether that is wrong for this product is
> **[open]** until someone measures the target hardware.
The second version is longer and it is the only one you can act on.
### Corollary: an absence has a marker too
"There is no lag compensation in this project" is a measured claim **only** if
you say which searches you ran. Otherwise it is derived from your search
strategy, which brings us to the next rule.
---
## 2. An empty search result proves the pattern, not the codebase
This is the single most expensive mistake available in this work, because it
produces confident negative claims that survive review.
A search returns nothing when the code is absent — and also when the pattern was
wrong, when the path was wrong, when the file type was excluded, when ignore
rules filtered the directory, or when the identifier lives in a binary asset.
The tool reports the same empty result for all six.
**Before writing "this does not exist":**
1. widen the pattern and confirm it can match something you know is there;
2. check the path and include plugins, not only the primary source directory;
3. check file-type filters in both directions — too narrow hides code, too wide
counts compiled debug symbols as source;
4. for anything that could live in a binary asset, state that the search cannot
see it.
Two real corrections from the audit behind this bundle, both caught by re-running:
- a search for one declaration form found 99 entries; the correct pattern found
**212 across five files**, because plugin configuration uses a syntax that
differs by one character;
- a search for a suspected dead field returned four hits; three were compiled
debug symbol files and the source truth was **one**.
Both would have shipped as facts.
---
## 3. Verify a headline claim yourself before repeating it
Any statement that will lead a section, appear in a summary, or be quoted gets
re-read at its source **by the person publishing it** — not accepted from a
teammate, a tool, or an earlier version of the same document.
This is not distrust. It is that a claim changes shape as it travels: a specific
finding about one function becomes a general statement about a subsystem in one
retelling, and nobody notices because each step was small.
Three corrections that came from exactly this re-reading, during the work behind
this bundle:
- "a field that is never read" was actually **written once in a constructor and
never read** — the precise version names a real defect class; the loose version
hides it;
- "the data graph has no compiler" was wrong: structural validation existed, in
quantity. The true statement was narrower — **semantic cross-family validation
is absent** — and the narrow version is the one that tells you what to build;
- a count of definitions "spread over 40+ files" was **31 files**, and the
original number had never been measured.
Note that in all three the correction made the claim *more* useful, not less. A
weaker true statement beats a stronger false one, and usually points at the fix.
---
## 4. A delegated report is raw material
Output from a subagent, a script, or a search tool is an input to your judgement,
not a result. Re-verify every headline claim by reading the source directly
before it reaches a document or a conclusion.
The same rule with a sharper edge: **a report that agrees with what you expected
deserves more scrutiny, not less.** During the work behind this bundle a delegated
search reported three instances of a defect; direct measurement found nineteen.
The report was not wrong in kind, only in scale, which is the hardest error to
notice because nothing about it looks incorrect.
---
## 5. A recipe you have not run is a hypothesis
Every detection recipe in this bundle was executed against a real codebase before
being written down. That is not thoroughness for its own sake — it is the only
way to find the recipes that are confidently wrong.
Two examples from this bundle, both caught by running them:
- a search for "is this field ever assigned" matched the **declaration**
`double LastFireTime = 0.0;`, so it reported a writer where none exists — a
false negative, in the exact case the recipe was written to catch;
- a search for "is this ordering value ever compared" matched the arrow operator
in `Entry->Priority`, so two assignments were counted as comparisons.
Both recipes looked correct. Both would have told the reader "no problem here" at
precisely the moment there was one.
**A detector that fails silently in the case it exists for is worse than no
detector**, because it converts an open question into a wrong answer. Run the
recipe against a codebase where you already know the answer, both when the answer
is yes and when it is no.
---
## 6. A check that has never failed may not be a check
Write the failing case for every gate you add, and confirm it fails.
The worked example is from this bundle's own tooling. A validator ran green for
months. It checked exactly one condition: that skill documents did not contain an
absolute path from **a different workspace than the one it ran in** — a literal
that could never appear. The check existed, the consumer existed, the report said
zero errors, and nineteen real leaks were present.
That is category C2 from `ue-reference-project-adoption` — a mechanism whose
consumer exists and whose condition can never become true — found in our own
code rather than in someone else's. It is the reason the delivery gate for this
bundle ships with one poisoned fixture per rule and refuses to pass if any rule
has never reddened.
Apply the same rule to content-validation commandlets, asserts, editor
validation and CI steps: **break something on purpose and confirm the pipeline
notices.** A gate whose failure path has never executed is a log line with
ambitions.
---
## 7. Write findings down where they survive
A finding that exists only in a conversation is lost. Context ends; files do not.
- write the observation at the moment it is made, not at the end of the session;
- put the citation next to the claim, not in a separate index;
- record open questions **as open**, rather than closing them with a plausible
guess;
- when a later measurement contradicts an earlier note, add the correction and
keep both — the fact that the number changed is itself information about how
it was obtained.
That last point matters more than it looks. Two of the corrections in §3 were
only findable because the original number had been written down precisely enough
to be checked.
---
## 8. Publication checklist
Before an audit, a review or a skill leaves your hands:
- [ ] every headline claim re-read at its source by you;
- [ ] measured, derived and open marked, and never mixed in one sentence;
- [ ] every negative claim states the searches that produced it;
- [ ] every recipe executed against a real codebase, in both directions;
- [ ] every gate has a failing case that has been observed to fail;
- [ ] open questions listed as open;
- [ ] counts re-measured rather than carried over from an earlier draft;
- [ ] corrections recorded rather than silently applied.
---
## 9. What this does not give you
None of the above makes a claim true. It makes a false claim **findable** — by a
reader, by a later measurement, by the person who inherits the document. That is
the whole ambition, and it is worth stating plainly because the alternative
belief is dangerous: a document full of markers and citations can still be
wrong, and a green gate means the form is right, not the content.
The one thing that reliably catches the rest is the habit in §3: read it yourself
before you say it.
---
## Provenance
Every rule here was written after violating it. The examples are real, they come
from the audit that produced this bundle, and they are included specifically
because rules stated without the failure that motivated them do not survive
contact with a deadline.
## Evidence boundary
This skill contains no claims about any codebase. Its examples describe mistakes
made during one research project and the corrections that followed; they are
offered as illustrations of a class, not as findings about the project they came
from.
plugins/ue-design-skills/skills/ue-evidence-discipline/references/failure-modes.md
+327
View File
@@ -0,0 +1,327 @@
# Failure modes: evidence discipline
Ten ways an audit produces a confident wrong answer.
Every entry below happened during the work that produced this skill bundle. Not
"could happen" — happened, was caught, and cost real time. That is the only
provenance this document has and the only one it needs: a method document written
from imagination describes a method nobody has used.
The shared property is uncomfortable: **each of these produces output that looks
like evidence.** A count, a file list, a quoted line, a green check. The failure
is never a missing answer; it is a well-formed answer to a question you did not
ask.
Identifiers (`ED-01` and up) are stable.
---
## Searching
### ED-01 - An empty result read as absence
**Mechanism.** A search returns nothing, and "nothing" is recorded as "this does
not exist in the codebase".
**Why it is silent.** Zero hits is a definite-looking answer. It has no error
state, no partial result, and nothing that suggests the pattern rather than the
tree was at fault.
**Why the obvious check misses it.** The obvious check *is* the search. Verifying
it requires a second, differently-shaped search — which feels redundant precisely
when it is most needed, because the first one was so clear.
**Symptom.** A claim of the form "there is no X here" that turns out to be "my
pattern did not match X". In this project it happened three times: a structure
assertion missed because the namespace prefix was omitted; a subsystem file
missed because the path was guessed rather than found; and — the one that would
have shipped — a leaked-path audit whose pattern used doubled backslashes and
returned exactly one hit, which was nearly reported as "only one leak" when the
real count was nineteen across two donors.
**Detect.** Before recording an absence, widen deliberately:
```bash
rg -n "Scalability::FQualityLevels" . # first attempt: 0 hits
rg -n "FQualityLevels" . # widened: 5 hits
rg -c "PatternPart" . ; rg -c "OtherPart" . # both halves separately
```
Then check what your tool excludes by default — ignore files, binary files,
hidden directories — and whether the path you searched is the path that exists.
**Guardrail.** **An empty result proves the pattern, not the absence.** Record
absences only after a widened search, a path check, and a statement of what the
search could not see.
---
### ED-02 - Build output counted as source
**Mechanism.** An unrestricted search matches compiled artefacts — debug symbols,
binaries, intermediate files — and their hits are counted alongside source.
**Why it is silent.** The matches are real. The count is arithmetically correct.
Nothing marks a hit as coming from a file that is generated rather than written.
**Why the obvious check misses it.** The result list is usually long enough that
nobody reads every line, and the summary count is what gets quoted. The tool even
labels binary matches — in a line most people skip.
**Symptom.** A field described as "appearing four times" when it appears once. In
this project exactly that: three of four hits were debug symbol files, and the
source truth was a single declaration — which is a much stronger finding, since a
field declared once and never read is a cleaner defect than one used four times.
**Detect.** Restrict to source globs, always, and check the difference:
```bash
rg -c "Symbol" . # everything
rg -c "Symbol" --glob "*.cpp" --glob "*.h" . # source only
```
If the two numbers differ, the first one was never a fact about your code.
**Guardrail.** Default to source globs. When a count matters, state which file
types it covers.
---
### ED-03 - A count taken over the wrong root
**Mechanism.** A census runs over the primary content directory in a project whose
plugins mount their own roots, and reports a total.
**Why it is silent.** The number is real and internally consistent. Nothing in
the result indicates which roots were not visited.
**Why the obvious check misses it.** The primary root is the obvious root, and it
is where nearly all hand-authored content lives in a small project. The error only
appears at a scale where checking is expensive.
**Symptom.** Budgets, dependency graphs and audits that are individually correct
and collectively wrong. In this project the primary root held 2 838 assets while
the full set of 105 mount roots held 17 256 — a factor of six. Separately, a
native-symbol count restricted to the main source directory undercounted by
eighteen for the same reason.
**Detect.** Enumerate roots before counting within them:
```python
roots = ar.get_sub_paths("/", False) # returns all mount roots
```
```bash
rg -c "PATTERN" Source/ # one root
rg -c "PATTERN" Source/ Plugins/ # all of them
```
**Guardrail.** A count states its scope in the same sentence as its value. "3 876
project-owned assets across 105 mount roots" is a fact; "3 876 assets" is a number
waiting to be misused.
---
## Reasoning
### ED-04 - Measured and inferred mixed in one sentence
**Mechanism.** A sentence contains something read in source and something
concluded from it, with no marker separating them.
**Why it is silent.** The sentence is true. Both halves are defensible. The reader
inherits the conclusion with the same confidence as the observation, which is one
level of confidence too many.
**Why the obvious check misses it.** Review checks whether claims are correct, not
whether their epistemic status is labelled. A correct inference passes.
**Symptom.** An inference propagating into other documents as a measurement,
where it can no longer be traced back and questioned. Most false conclusions in
this project's audit originated at exactly this seam.
**Detect.** This one is answered by format, not by search. Require a marker per
claim — measured, derived, open — and treat an unmarked claim as unreviewed.
Then check the derived ones for the strongest available failure: what would have
to be true for this inference to be wrong, and did anyone check?
**Guardrail.** Three categories, never mixed within a sentence. An open question
stays written as open rather than closed with a plausible guess — a plausible
guess is indistinguishable from a finding six months later.
---
### ED-05 - A subagent's headline accepted without re-reading
**Mechanism.** Delegated investigation returns a confident summary. The summary is
used directly.
**Why it is silent.** Reports are fluent, structured and usually mostly right.
Errors arrive in the same register as correct findings.
**Why the obvious check misses it.** Reading the report *is* the check, and the
report is internally coherent. Detecting the error requires re-reading the source
the report was derived from — that is, redoing the delegated work at the points
that matter.
**Symptom.** In this project: an exploration agent reported three sites carrying
absolute workstation paths; the real count was nineteen. Separately, a claim that
the game-phase system lived in a feature plugin, when it lives in the main module.
**Detect.** For each headline claim in a report, open the cited location and read
it. If the report cites no location, the claim is unverifiable and is dropped
rather than softened.
**Guardrail.** Delegated output is **raw material**, not a result. Every headline
claim is re-read at the source before it reaches a document or a conclusion.
---
### ED-06 - A number carried forward without re-measurement
**Mechanism.** A count measured once is quoted in later documents. The tree
changes, or the original measurement was scoped differently, and the number
persists.
**Why it is silent.** Numbers do not expire visibly. A stale count looks exactly
like a fresh one, and having a number at all suppresses the impulse to take one.
**Why the obvious check misses it.** Review checks whether the document is
coherent, and it is. The original measurement was correct when taken.
**Symptom.** In this project, an audit stated "59 native definitions across 40+
files". Re-measured during depersonalization: **94 definitions across 31 files**,
with the discrepancy caused by ED-03 in the original. The correction was recorded
alongside the original rather than replacing it, because which of the two readings
was direct is itself information.
**Detect.** Re-run the measurement when quoting it in a new context, and diff:
```bash
rg -c "PATTERN" Source/ Plugins/ | awk -F: '{s+=$2} END {print s}'
```
**Guardrail.** A quoted number carries the command that produced it, so the next
reader can re-run it in one paste. Corrections are recorded as corrections — an
archive that silently self-heals loses the trail of which reading was direct.
---
## Recording
### ED-07 - A finding that never left the conversation
**Mechanism.** Something real is discovered, discussed, and not written down.
**Why it is silent.** At the moment of discovery it feels known. The cost arrives
later, when the context holding it is gone.
**Why the obvious check misses it.** There is nothing to check. The absence of a
note is not visible from anywhere except the future.
**Symptom.** The same investigation performed twice. A conclusion remembered
without its evidence, which then cannot be defended or corrected.
**Detect.** At the end of a session, diff what was concluded against what was
written. Anything in the first list and not the second is already lost — writing
it down later is a reconstruction, not a record.
**Guardrail.** **Found and not written down is lost.** Write the observation
immediately, with its address, before continuing. A conversation is not a store.
---
### ED-08 - A check whose condition cannot become true
**Mechanism.** A validator tests for a literal that could never appear — a path
from a different workspace, a name from a previous project, a case the current
tree cannot produce.
**Why it is silent.** The check runs, passes, and reports success. Green is the
outcome it was built to produce, and it produces it forever.
**Why the obvious check misses it.** The validator exists, is invoked, and is
listed in the process documentation. Everything about it says "this is checked".
Nobody re-reads a passing check.
**Symptom.** In this project: a skill validator searched for one hardcoded
absolute path from a *different* project, and only inside one file type. It ran
green for months while nineteen leaked paths from two donors sat in the tree.
That validator became this bundle's worked example — our own code, not the
reference's.
**Detect.** For every check, produce the input that makes it fail. If you cannot
construct one, the check does not exist:
```bash
python validate.py # green
# now poison a fixture with the exact thing the rule forbids
python validate.py # must be red
```
**Guardrail.** **A rule with no fixture that reddens it is not a rule.** Keep one
poisoned fixture per rule, plus a clean baseline, and assert the rule count so a
rule cannot be dropped silently.
---
### ED-09 - A namespace collision that collapses two sets
**Mechanism.** Two independent collections use the same identifier prefix. A tool
that merges them by key silently overwrites, and reports success on the survivors.
**Why it is silent.** Every remaining entry resolves. The tool's output is a
consistent, complete-looking mapping — of a smaller set than it was given.
**Why the obvious check misses it.** The check verifies that everything present
resolves, which is true. Nothing verifies that everything given is still present.
The count is the only tell, and only if someone compares it against the source.
**Symptom.** In this project, two skills adopted the same two-letter prefix. Nine
entries vanished from the resolver, which reported "173 shipped, 173 mapped, 0
unresolved" — a perfectly green run against a set that had lost nine members. It
was caught only because the total failed to grow after adding nine entries.
**Detect.** Count both sides independently and compare:
```bash
rg -c "^### [A-Z]{2,4}-" skills/*/references/*.md | awk -F: '{s+=$2} END {print s}'
# compare against what the merging tool reports
```
Then check prefix uniqueness directly, and make it a rule rather than a habit.
**Guardrail.** Any tool that merges by key asserts that its output cardinality
equals its input cardinality. A resolver that cannot lose entries is worth more
than one that reports zero failures.
---
### ED-10 - A recipe published without being run
**Mechanism.** A detection recipe is written from understanding of the defect
rather than from executing it against a real tree.
**Why it is silent.** The recipe is plausible, well-formed, and would work if the
world were slightly simpler. It is published in a document whose whole purpose is
to be trusted.
**Why the obvious check misses it.** Reading the recipe confirms it expresses the
right idea. Only running it reveals that the pattern matches something it should
not, or misses something it should catch.
**Symptom.** In this project, three recipes were wrong on first draft and each was
wrong in a way that produced a **false negative** — the worst direction. A search
for a field's writer matched the declaration's own initializer and reported "there
is a writer" in exactly the case the recipe exists to catch. A character class
intended to find comparisons matched the arrow operator and reported assignments
as comparisons. A search for a namespace matched nothing because real names carry
a prefix — and zero hits would have read as "this problem is absent here".
**Detect.** Run every recipe against a tree where you already know the answer, and
check both directions: does it find the known instance, and does it stay quiet
where there is none?
**Guardrail.** **A recipe that has not been run is a hypothesis.** Publish the
corrected form, and where the first draft failed in an instructive way, publish
that too — the trap is often more useful than the recipe.