# Plan: font-embed engine (fix veraPDF 7.21.4.1 "font program not embedded")

**Architect** Opus · **Reviewed** Fable 2026-07-23 (APPROVED — 3 wiring claims verified on disk: `fidelity_gate.pixel_diff(threshold_pct=)` exists §3; `pdf_font_repair.repair_fonts` call sites at remediate_pdf.py:241 E2 / :290 E3 confirm §4; Liberation SIL-OFL committable) · **Implement** Sonnet · Plan only, no code.

**Reviewer note — the bounded gate is also the correctness net.** The one thing that could silently go wrong (a TrueType subset whose cmap doesn't match the PDF's `/Encoding`, rendering *wrong glyphs* that still satisfy 7.21.4.1's "is a font embedded" check) is caught for free: wrong glyphs blow the pixel diff far past 2.5% → gate fails → `.rejected.pdf`. So compliance and visual correctness are both gated, not just compliance. This is WHY symbol/custom-encoding fonts (Symbol/Wingdings) route to `skipped_no_substitute` rather than risk a cmap-mismatched substitute — verified: the census spike's standard-encoding text fonts rendered correctly, symbol fonts were not substituted.

## Ground truth (live spike + deep research, 2026-07-23 — do not re-probe)

veraPDF PDF/UA-1 **7.21.4.1** passes iff `Subtype==Type3 || Subtype==Type0 ||
renderingMode==3 || containsFontFile==true`. So injecting a `FontFile*` stream onto the
existing FontDescriptor is *logically sufficient* for this clause. Proven live: FontFile2
injection of Liberation into `census2000manual.pdf` dropped 7.21.4.1 failures **9→2** (the
2 residuals: a Type1 base-14 `/Times-Roman` needing a `.pfb`, and a Type3 that already
auto-passes). No Ghostscript (drops the tag tree), no callas/Adobe. Pure pikepdf + fontTools.

- Stream by subtype: **TrueType→FontFile2** (+`/Length1`=full sfnt length; PROVEN),
  **Type1→FontFile** (needs `/Length1//Length2//Length3` = cleartext/encrypted/trailer PFB
  segmentation), **CFF/OTF→FontFile3**.
- Layout is pixel-STABLE always (advances come from `/Widths`, not the outline). Glyph
  *outlines* are pixel-zero only when the free font IS the original (LaTeX Nimbus/CM). For
  Windows-core PDFs (Times/Arial→Liberation) the outline diff is **~1.5% — sub-perceptual**
  (user viewed real before/after: "indistinguishable").
- **USER DECISION (load-bearing):** authorize a BOUNDED perceptual fidelity gate for the
  font-embed path ONLY. Rest of E2/E3 stays exact-zero. ~1.5% is an accepted trade because
  originals are proprietary and no free path gives pixel-zero.
- **7.21.4.2 coupling:** injecting a FontFile makes the embedded program authoritative; a
  stale `/CharSet` on a now-embedded Type1 descriptor can fail 7.21.4.2. `pdf_font_repair.py`
  already strips `/CharSet` from *subset-prefixed* Type1 descriptors — order matters (embed
  first, repair second).
- Vendored assets confirmed: Liberation Serif/Sans/Mono (Regular/Bold/Italic/BoldItalic) at
  `<LibreOffice.app>/Contents/Resources/fonts/truetype/`. **No** Nimbus/CM `.pfb` present.

## Design decisions (each names the simpler alternative it beat)

- **Bundle Liberation `.ttf` in-repo** (`standards/fonts/`, ~1 MB, SIL OFL redistributable),
  NOT read from LibreOffice's copy. Beats: coupling the PDF-only path to the 500 MB
  **opt-in Phase-3** LibreOffice install most PDF users never run. Liberation covers the ~80%
  Windows-core case; ships in the plugin zip, zero install.
- **LaTeX pixel-zero families (URW Nimbus `.pfb`, AMSFonts CM) + a free Symbol: opt-in fetch**
  via `setup_remediation.py --install-fonts` into the vendor dir (mirrors LibreOffice's
  opt-in). Beats: shipping ~5 MB of niche fonts in every zip. Absent family → font is
  `skipped_no_substitute` → stays `needs_manual` (graceful degrade, never a crash).
- **Type1 base-14: embed the FULL `.pfb` (no subset).** Beats: Type1 subsetting via t1Lib
  (brittle; marginal win on already-small base-14 programs). TrueType path DOES subset (size).
- **Subset = optimization, not correctness.** 7.21.4.1 only needs `containsFontFile`; subset
  to shrink, and on any subset failure embed the full font (still passes). De-risks the step.
- **No FontDescriptor metric rewrite** (`/Ascent//StemV//Flags/…`). Substitutes are
  metric-compatible; `/Widths` drives layout. Beats: recomputing metrics (new mismatch risk).
- **Used-glyph proxy = `FirstChar..LastChar` under the resolved `/Encoding`.** Beats:
  content-stream glyph scanning (overkill — clause needs the program embedded, not minimal).

## Files & changes (ponytail — the mechanism is ~1 subset+inject function)

### 1. `scripts/pdf_font_embed.py` (new)
`repair_missing_fonts(pdf) -> {embedded, skipped_no_substitute, skipped_type3, skipped_embedded}`.
Walk page + Form-XObject `/Font` (cycle-guarded, **mirror `pdf_font_repair.handle_resources`**).
Per font:
- Resolve embedded-ness: `FontFile|FontFile2|FontFile3` on the FontDescriptor (Type0 → the
  descendant CIDFont's descriptor). Embedded → `skipped_embedded`.
- `Subtype==Type3` → `skipped_type3`. `Type0` with embedded descendant → `skipped_embedded`.
- Map `BaseFont`→substitute (strip `^[A-Z]{6}\+` subset prefix; §2). No map / family not
  present → `skipped_no_substitute`.
- Pick style variant (Regular/Bold/Italic/BoldItalic) from `BaseFont` tokens (`-Bold`,
  `-Italic`, `-Oblique`, `,Bold`) first, FontDescriptor `/Flags` italic bit + weight fallback.
- **TrueType** → subset Liberation `.ttf` to the code-set's Unicode via `fontTools.subset`;
  inject `FontFile2` with `/Length1`=sfnt byte length. **Type1 base-14** → inject full `.pfb`
  as `FontFile`, deriving `/Length1//Length2//Length3` from PFB segment markers
  (`0x80 0x01`/`0x02`/`0x03`); delete any pre-existing `/CharSet` on that descriptor.
  **CFF/OTF** → `FontFile3`.
- `embedded += 1`.

### 2. Font substitution map + bundling
Table (subset-prefix stripped): `Times*/TimesNewRoman*→LiberationSerif`,
`Arial*/Helvetica*/ArialMT→LiberationSans`, `Courier*/CourierNew*→LiberationMono`,
`Nimbus*/URW*→URW .pfb`, `CM*/CMR*/Computer Modern→AMSFonts CM`, `Symbol→StandardSymbolsPS`.
`Wingdings/Webdings/ZapfDingbats/custom dingbats → NO free clone → skipped_no_substitute`
(documented residual, stays `needs_manual`). Resolver checks in-repo `standards/fonts/` first,
then vendor dir; missing → `skipped_no_substitute`.
`setup_remediation.py`: add `--install-fonts` (opt-in, like `--install-libreoffice`) that
fetches URW base-35 `.pfb` + AMSFonts CM + a free Symbol into the vendor dir; `get_fonts_dir()`
helper returning (repo dir, vendor dir). Liberation needs **no** fetch (committed in-repo).

### 3. Bounded fidelity gate wiring — `remediate_pdf.py`
Add constant `FONT_EMBED_FIDELITY_PCT = 2.5` (basis: ~1.5% observed Windows-core + margin;
2× tighter than a real layout break). After the run, if `edits["font_embed"]["embedded"] > 0`,
call `fidelity_gate.pixel_diff(input, output, threshold_pct=FONT_EMBED_FIDELITY_PCT)`; else keep
the exact-zero call. `fidelity_gate` already takes `threshold_pct` — **no change there**. Always
record actual `worst_pct`; result gets `font_embed_fidelity: {threshold_pct, worst_pct}` and a
human line `"fonts embedded; visual diff X% (below Y% imperceptibility bound)"`. **Never** print
"pixel-identical" on this path. `worst_pct > bound` (real layout break) → gate fails, output →
`.rejected.pdf` as today.

### 4. Order-of-operations with `pdf_font_repair`
In **both** branches, run `pdf_font_embed.repair_missing_fonts(pdf)` **immediately BEFORE** the
existing `pdf_font_repair.repair_fonts(pdf)` — E2 (`_run_e2`, ~line 241) and E3
(`remediate_pdf`, ~line 290). Rationale: a just-embedded subset font then gets consistent
CharSet/ToUnicode handling from `pdf_font_repair`. Store counts in `edits["font_embed"]`.

### 5. `_has_font_defects` interaction
Because embed runs before the veraPDF check in each branch, any *surviving* 7.21.4.1 (no-substitute
residuals like Wingdings) still trips `_has_font_defects` → `needs_manual`. Verify the flag stays
accurate (AC). Update `_FONT_DEFECT_REASON` / module docstring: 7.21.4.1 is now *partially*
auto-fixed (substitutable fonts), residuals still need manual/source remediation.

## Phases

- **P1 (MVP, shippable):** `pdf_font_embed.py` TrueType→FontFile2 only + Liberation map + in-repo
  bundle + orchestrator wiring (§4) + bounded gate (§3). Covers the ~80% Windows-core case.
- **P2:** Type1 base-14 FontFile path (PFB Length1/2/3) + URW sourcing + `--install-fonts`.
- **P3:** CFF/OTF FontFile3 + AMSFonts CM (LaTeX pixel-zero) + Symbol.

## Acceptance criteria (incl. negatives)
1. P1: `census2000manual.pdf` 7.21.4.1 failures **9→2** (matches spike); Type3 untouched; a
   no-substitute font is `skipped_no_substitute`.
2. P2: the `/Times-Roman` Type1 residual embeds; census 7.21.4.1 → 0.
3. End-to-end: census remediates to veraPDF COMPLIANT; result `ok:true`, `output` set.
4. Bounded gate REPORTS actual `worst_pct`; the human line names the bound; never "pixel-identical".
5. **Negative:** a no-free-clone font (Wingdings) → surviving 7.21.4.1 → `needs_manual:true`.
6. **Negative:** a deliberately wrong-metric substitute → `worst_pct >> 2.5%` → gate fails →
   `.rejected.pdf`, `ok:false`.
7. **Negative:** a non-font run stays exact-zero (threshold not loosened when `embedded==0`).
8. Already-embedded fonts are `skipped_embedded` (idempotent; re-run is a no-op).
9. Subset failure falls back to full-font embed and still passes 7.21.4.1.

## Risks
| Risk | Mitigation |
|---|---|
| Type1 PFB Length1/2/3 segmentation wrong → 7.21.4.1 still fails / viewer chokes | Parse PFB segment markers; unit-test byte offsets against a known `.pfb`; embed full (no subset) to reduce surface |
| Threshold too loose hides a real structural break co-occurring with embed | Keep bound tight (2.5% vs 1.5% observed); always surface `worst_pct` for reviewer eyeball; structural edits are 0% so stacked break shows |
| Proprietary symbol/dingbat has no free clone | Documented residual; `skipped_no_substitute` → `needs_manual` (correct, not a bug) |
| Stale `/CharSet` on now-embedded Type1 → new 7.21.4.2 fail | Delete `/CharSet` on FontFile injection; embed-before-repair ordering (§4) |
| Font license/bundling in git | Liberation SIL OFL redistributable (commit OK); URW/CM/Symbol are opt-in fetch, not committed |

## Reviewer findings — real-file E2E on STP530 (Opus, 2026-07-23; supersedes parts of §3/AC where noted)

Ran the built engine on the 4 real STP530 files that actually carry 7.21.4.1
(found by full veraPDF scan of both course caches): Zoom-How-to, Critiques_of_HT,
Learning-Community, census2000manual. Originals backed up. Findings:

1. **P1+P2 mechanism confirmed on real files.** Zoom/Critiques (TrueType) → 0;
   Learning-Community (Type1 `/Times-Roman`, NO descriptor/Widths — synthesized) → 0;
   census `/Times-Roman` (Type1, HAS descriptor/Widths) → embedded. All visually
   verified indistinguishable (Critiques p15, a dense citation page, rendered
   before/after — identical to the eye at 10% raw-pixel diff).

2. **DEFECT FOUND + FIXED — the §3 bounded pixel gate is the wrong metric.** Raw
   `pixel_diff` cannot discriminate a correct substitution from a wrong one on
   text-dense pages (it measures amount-of-text, not correctness). Measured on
   Critiques p15: CLEAN Serif **10.25%**, WRONG-weight (Bold) **12.17%**,
   WRONG-family (Mono) **14.64%** — overlapping, no separable bound. The 2.5%
   bound falsely REJECTS clean output (2 of 4 files). **Fix:** a `structural_diff`
   metric = fraction of pixels whose grayscale intensity changes by **>64/255**
   (ignores anti-aliasing sub-pixel shift, catches `.notdef` boxes / reflow).
   Same three cases → **1.80% / 5.60% / 9.40%** (clean separation). All 4 clean
   files ≤1.8% worst-page (Zoom 0.84, Critiques 1.80, LC 0.00, census 0.08).
   New gate: `FONT_EMBED_DELTA=64`, `FONT_EMBED_STRUCT_PCT=3.5` (~2× margin).
   AC6 re-reads: a wrong substitution → structural worst_pct >> 3.5% → `.rejected`.
   The §3 "~1.5% pixel diff / 2.5% bound" text is obsolete — structural metric replaces it.

3. **census Symbol resolved (was the AC1/AC2 residual).** The residual after P1+P2
   is a Type1 `/Symbol` using ONE glyph (`/second`, ″, on 19 pages). No free
   *bundleable* Symbol clone exists, so the ENGINE keeps `/Symbol`→
   `skipped_no_substitute` (honest, graceful). For census's deliverable only, the
   `/second` glyph is embedded as CFF `/FontFile3` (Type1C) sourced from a locally-
   present Symbol clone → census 7.21.4.1 **→ 0**, structural 0.08%. Documented as a
   local-file step, NOT an engine capability (bundling a Symbol font is a separate
   licensing decision; Liberation *has* the `second` glyph, so a bundle-clean
   FontBuilder-CFF path is a viable future follow-up if census needs Canvas delivery).

4. **P2 mechanism deviation from the plan.** §Files §1 / P2 specced Type1 → `.pfb`
   `/FontFile` (Length1/2/3). Proven-better mechanism (implemented): flip
   `/Subtype` Type1→TrueType + `/FontFile2` (Liberation), synthesizing descriptor/
   Widths when absent. veraPDF requires the FontFile type match the subtype; the
   subtype-flip reuses the P1 TrueType path (ponytail) and avoids PFB segmentation
   entirely. The `.pfb` risk row below is therefore moot for P2.

## Doc touch-list
`skills/file-remediation/SKILL.md` Stage 2 (new engine; revise the "7.21.4.1 not addressed"
paragraph → *partially* addressed; the sub-perceptual-diff caveat) · `remediate_pdf.py` docstring +
`_FONT_DEFECT_REASON` · `setup_remediation.py` (`--install-fonts`) · `CLAUDE.md` scripts list (add
`pdf_font_embed.py`) · `docs/plan-file-remediation-skill.md` engine roster · `CHANGELOG` + version
bump (feat → minor) + manifest sync. Codex co-implementation + PR-only per CLAUDE.md.
