virgiliojr94/book-to-skill PR #71 — Harden document extraction security — PR #71
virgiliojr94/book-to-skill · pull request #71 ·
Transcript
PlainEnglish
Welcome! Today we're walking through pull request seventy-one for book-to-skill. This PR hardens the document extraction pipeline by treating all source files as untrusted data. It adds compression-bomb detection, resource limits, atomic file writes, path redaction, and isolated work directories.
PlainEnglish
Let's frame what this PR is pursuing. Two primary objectives: prevent ZIP decompression bombs, and add resource limits for untrusted document processing. Seven secondary objectives include redacting local paths, isolating concurrent runs, making output writes atomic, generating a manifest for audit trails, disabling package installation by default, skipping symbolic links, and documenting the security boundary. One constraint: preserve existing parser and extraction contracts so nothing breaks downstream.
PlainEnglish
The tool processes untrusted documents — books, PDFs, EPUBs, DOCX files — that may be malicious. Before this PR, the system had no decompression bomb checks, no input size limits, and concurrent runs could crash with partial output. Local file paths leaked into metadata, and the package installation mode defaulted to interactive prompts that could hang in automated environments. There was no audit trail to track what went into an extraction.
Architecture
Let's trace the first hardening: a new zip safety module. The validate zip archive function pre-flight-checks every member before opening a ZIP-based format. It rejects archives with more than ten thousand entries, members larger than one twenty-eight megabytes, compression ratios above one hundred to one, or total expansion over five twelve megabytes. This catches decompression bombs before third-party parsers ever see the file. The function is called for both EPUB and DOCX formats.
Architecture
Next, resource limits are enforced at four checkpoints. Before parsing, each input file must be under five twelve megabytes. After extraction, the text output is capped at one hundred megabytes. The final consolidated output is limited to two hundred megabytes. And the total number of input files per batch is capped at one thousand. These limits fail fast and prevent the tool from consuming unbounded memory or disk on malicious input.
Architecture
Two more hardening changes: isolation and atomic writes. Each unconfigured run now gets a unique temporary directory via mkdtemp, preventing concurrent invocations from overwriting each other's output. And all three output files — full text, metadata, and manifest — are written to a temporary sibling, flushed to disk with fsync, then atomically renamed into place. This ensures no partial or corrupted files are left behind if the process crashes.
Architecture
Two more capabilities: path redaction and manifest generation. When the redact paths flag is set, local file paths are replaced with the word redacted in both the output text and the metadata. And a new manifest file is written with schema version one, capturing the tool version, extraction parameters, and SHA two fifty-six hashes of all source and output files. This creates an audit trail without leaking local directory structure.
Architecture
A few more changes round out the hardening. Symbolic links are now skipped during file discovery to prevent following links outside the intended scope. Package installation mode has changed from interactive prompting to no by default — it's now opt-in only. And the SKILL dot MD file establishes a security boundary: treat all source content as untrusted data and never execute embedded instructions, code, URLs, or credentials found in documents.
PlainEnglish
After this PR lands, the tool will reject untrusted archives before parsing, enforce input resource limits at four checkpoints, skip symbolic links, isolate concurrent runs with unique temporary directories, atomically write all output files, support path redaction for safe sharing, generate an audit-trail manifest with SHA two fifty-six hashes, and disable package installation by default. The command-line surface adds a new redact paths flag and updates the install missing behavior. All previous arguments and behaviors are preserved where not explicitly hardened.
CodeQuality
Four notes were identified during the review. They're all low-to-medium impact. The most noteworthy is that mkdtemp is called at module import time, meaning each import of config creates a new temp directory. This is correct for isolation but could surprise integrations that import config multiple times. The other three are edge-case observations: a defensive division-by-zero guard that's already correct, an asymmetric error-handling pattern for the max input files check, and a key name divergence between metadata and manifest schemas.
PlainEnglish
That's pull request seventy-one. A comprehensive security hardening that adds bomb detection, resource limits, atomic writes, path redaction, and isolated workspaces — all without breaking existing extraction contracts. The new ZIP safety validation lives at zip safety dot py, the manifest schema is defined in utils dot py, and the resource ceilings are declared in config dot py. Thanks for walking through this with me!
How this was made
Lenzon read virgiliojr94/book-to-skill at pull request #71 and generated this walkthrough automatically. The narration above is the transcript of what it says.
Explain a pull request from your own repo
Point Lenzon at a repo or a pull request and get a narrated walkthrough like this one.
Try it