Skip to content

feat(python): expose strip_headers_footers and remove_page_numbers - #458

Open
ddakv wants to merge 1 commit into
firecrawl:mainfrom
ddakv:feat/python-markdown-kwargs
Open

feat(python): expose strip_headers_footers and remove_page_numbers#458
ddakv wants to merge 1 commit into
firecrawl:mainfrom
ddakv:feat/python-markdown-kwargs

Conversation

@ddakv

@ddakv ddakv commented Aug 24, 2026

Copy link
Copy Markdown

What

Exposes two existing MarkdownOptions switches as keyword-only arguments on the Python process_pdf / process_pdf_bytes bindings:

result = pdf_inspector.process_pdf(
    "document.pdf",
    strip_headers_footers=False,  # keep running headers/footers
    remove_page_numbers=False,    # keep standalone page-number text
)

Both default to True, matching MarkdownOptions::default() — existing callers see byte-identical behavior.

Why

The Rust API has had these switches all along (PdfOptions::new().markdown(...)), and the WASM binding already exposes a subset of MarkdownOptions (profile, includePageMarkers, includeImages) — but the Python bindings hardcode PdfOptions::new(), so Python callers cannot opt out of furniture stripping.

Our use case: a document-ingestion pipeline (RAG) where header/footer removal must be a per-workflow choice — some corpora need verbatim page text preserved (compliance/eval requirements), others want the noise gone. Today the default (strip) is right for most retrieval workloads, but the pipeline can't offer the choice without this exposure.

Notes on semantics

  • strip_headers_footers=False cleanly bypasses the furniture pass (markdown/mod.rs's gate), keeping running headers/footers.
  • remove_page_numbers=False disables the lexical standalone-page-number filter in postprocess. Folios resolved from positional evidence are still removed — layout analysis depends on them — and the new docstrings/stub/docs say so explicitly, so nobody reads the kwarg as "verbatim output".
  • Kwargs are keyword-only (after pages), following the process_pdf_with_ocr signature style. The shared option assembly lives in one build_process_options helper.

Testing

  • Five new tests in tests/test_python.py: repeated running furniture survives with strip_headers_footers=False (p1244-1996.pdf) and is stripped by default; Page NNN text survives with remove_page_numbers=False (multiline_indent_cell_rect_grid.pdf); explicit True kwargs produce byte-identical markdown to the bare call; kwargs are keyword-only; the bytes variant matches the file variant.
  • cargo fmt, cargo clippy --features python -- -D warnings, and cargo test all pass.
  • pdf_inspector.pyi and docs/python.md updated in step.
  • No extraction-path changes and defaults preserved, so the pdf-evals snapshot suite is unaffected by construction (I don't have access to run it).
  • Note: tests/test_python.py::TestMultipleFixtures::test_process_all_fixtures[encrypted-secret123.pdf] fails on unmodified main too (pre-existing, process_pdf has no password parameter); unrelated to this change.

Summary by cubic

Expose strip_headers_footers and remove_page_numbers as keyword-only args in pdf_inspector for process_pdf and process_pdf_bytes. This lets Python callers keep running headers/footers and standalone page-number text when needed; defaults preserve previous output.

  • Add keyword-only args after pages: process_pdf(path, pages=None, *, strip_headers_footers=True, remove_page_numbers=True) and same for process_pdf_bytes.
  • Keep behavior identical for existing callers (defaults match MarkdownOptions::default()).
  • Factor option assembly into a shared build_process_options that sets PdfOptions.markdown with these fields.
  • Semantics: strip_headers_footers=False skips the furniture pass; remove_page_numbers=False disables the standalone page-number filter; folios removed via positional evidence remain removed.
  • Update pdf_inspector.pyi and docs/python.md; add tests for defaults, keyword-only enforcement, bytes/file parity, and examples where furniture/page numbers are retained.

Written for commit 37a9572. Summary will update on new commits.

Review in cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 4 files

Shadow auto-approve: would auto-approve. Exposes existing Rust markdown options to the Python API. The changes use keyword-only arguments and preserve default behavior, following established library patterns with thorough test coverage.

Re-trigger cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant