My Extraction Score Was 0. 08 and the Model Was Innocent: Rebuilding the Ruler

CauterRule’s v0.3.1 release corrects a misleading extraction‑accuracy metric that reported a 0.08 score. The update introduces a trigger‑only semantic agreement, separates directive scoring, and fixes reporting bugs that conflated zero with missing ground truth. The new metrics reveal that models a…

Sun, 13 Sep 2026 23:08:45 GMT

In the latest release of the open‑source sidecar CauterRule, developers have addressed a critical flaw in the extraction‑accuracy metric that had been reporting a disconcertingly low score of 0.08. The new version, v0.3.1, introduces a trigger‑only semantic agreement, separates directive scoring, and fixes several reporting bugs that conflated zero with missing ground truth. The result is a clearer picture of how well cloud models are extracting rules from agent failures.

What Happened?

CauterRule is designed to learn standing rules from repeated agent failures. It extracts lessons from trajectories, replay‑tests them, and attempts to separate reusable guidance from noisy over‑generalization. In a recent field test, the tool evaluated two cloud models—gpt‑4o‑mini and llama‑3.1‑8b—across 40 corpora and 4,742 trajectory runs. The first reading of the extraction‑accuracy metric was a shocking 0.08, suggesting that the models were performing poorly.

However, a deeper dive revealed that the metric was penalizing the models for re‑wording the directive portion of a rule. A directive such as "pull latest changes before pushing" was re‑phrased to "run a rebase or fetch‑merge first"—a change that shares almost no literal tokens but is semantically equivalent. The metric’s reliance on literal token F1 for the directive caused the score to collapse, masking the fact that the models were correctly identifying the trigger.

How the Metric Was Fixed

Three key changes were introduced in v0.3.1:

• Trigger‑only agreement: The metric now compares only the trigger portion of the rule using a semantic similarity score. This prevents re‑wording of the directive from penalizing the overall score.
• Directive reporting: The directive is no longer part of the headline agreement. Instead, it is reported separately as directive_f1, allowing users to see how well the model captured the action without affecting the composite score.
• Null handling: When a corpus lacks an expected_rule field, the metric now returns null instead of 0.0. This distinction prevents the report from misrepresenting unmeasured cases as failures.

These changes are backed by regression tests that ensure the metric behaves as expected. The test suite now includes test_extraction_accuracy.py and test_agreement_is_trigger_only_directive_not_gated.py, which guard against future regressions.

What the Data Reveals

The updated metrics show a wide spread across different comparators:

• Agreement (semantic trigger): 0.74–0.93—models are consistently capturing the correct trigger.
• Token F1 (literal wording): 0.42–0.65—literal similarity is moderate, reflecting re‑wording.
• Directive F1: 0.20–0.39—directive wording is the weakest link, which is expected given the new scoring approach.

For example, in the golden corpus (60 trajectories), gpt‑4o‑mini achieved an agreement of 0.783 and a directive F1 of 0.205, while llama‑3.1‑8b scored 0.850 agreement and 0.220 directive F1. Across the reference‑expansion corpus (303 trajectories), both models achieved agreement scores above 0.9, indicating that they are reproducing reference phrasings accurately.

What Still Needs Investigation

Several open questions remain:

• Is trigger‑only agreement too generous? A correct trigger with a nonsensical directive might still be considered a good extraction.
• How does the high agreement in reference‑expansion reflect on generalization to unseen rules?
• What is the impact of small corpora, such as failures/positive (23 trajectories) and paraphrase‑diversity (15 trajectories), where statistical confidence is low?

Future releases will aim to address these questions by refining the metric, expanding the corpus, and adding more robust statistical analysis.

Why This Matters

Accurate extraction metrics are essential for building reliable AI agents. By separating trigger and directive evaluation, CauterRule provides developers with a clearer diagnostic tool, enabling faster iteration and more trustworthy rule extraction.

Key Takeaways

• CauterRule v0.3.1 corrects a misleading 0.08 extraction score.
• The new metric uses trigger‑only semantic agreement and reports directives separately.
• Null handling prevents misinterpretation of unmeasured cases.
• Field‑test data shows models capture triggers well but re‑word directives.
• Open questions remain about directive quality and small‑sample confidence.

FAQ

• What is CauterRule? An open‑source sidecar that learns standing rules from agent failures, extracting lessons, replay‑testing, and promoting reusable guidance.
• How do I install it? Run pip install cauterule to get the CLI, adapters, and official rule packs.
• Which models were tested? gpt‑4o‑mini and llama‑3.1‑8b were evaluated across 40 corpora.
• What does a 0.08 score mean? It was a metric artifact caused by penalizing re‑worded directives; the new metric shows higher agreement.
• Where can I find the test data? The full field‑test report is available on GitHub under the v0.3.1 release notes.

Why it matters

By refining the extraction‑accuracy metric, CauterRule now offers developers a truthful assessment of model performance, enabling more effective debugging and faster deployment of reliable AI agents.

Key points

• CauterRule v0.3.1 corrects misleading extraction metrics
• Trigger‑only semantic agreement separates directive re‑wording
• Null handling distinguishes missing ground truth from failures
• Field‑test data shows strong trigger extraction but weaker directive capture
• Open questions remain about directive quality and small‑sample confidence

Frequently asked questions

What is CauterRule?

An open‑source sidecar that learns standing rules from agent failures, extracting lessons, replay‑testing, and promoting reusable guidance.

How do I install it?

Run <code>pip install cauterule</code> to get the CLI, adapters, and official rule packs.

Which models were tested?

gpt‑4o‑mini and llama‑3.1‑8b were evaluated across 40 corpora.

What does a 0.08 score mean?

It was a metric artifact caused by penalizing re‑worded directives; the new metric shows higher agreement.

Where can I find the test data?

The full field‑test report is available on GitHub under the v0.3.1 release notes.

Reporting drawn from

• dev.to