feat: add support for tracking removed header/footer lines in PDF pro…

[Harikishan-AI]

Fixes: #459

what is done:

1. Per-page removed header/footer counts are computed in src/markdown/mod.rs.
2. Counts are threaded through MarkdownConversionOutput.
3. Counts are surfaced on PdfProcessResult.
4. Python, Node/NAPI, and WASM wrappers expose the field.
5. I fixed the visibility issue so src/lib.rs can access MarkdownConversionOutput

---

Summary by cubic

Fixes #459 by adding per-page counts of lines removed as running headers/footers to PDF processing results, now exposed across the Python, Node/NAPI, and WASM bindings. Previously callers had no visibility into how much content was stripped during markdown conversion.

New Features

• Counts are computed by diffing line counts before and after header/footer stripping.
• Makes MarkdownConversionOutput fields pub(crate) so src/lib.rs can thread the counts through.
• In analyze mode, removed_header_footer_lines is returned as a zero-filled vector per page.

^{Written for commit 40f57ca. Summary will update on new commits.}

[cubic-dev-ai]

3 issues found across 6 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/lib.rs">

<violation number="1" location="src/lib.rs:4140">
P3: Analyze mode always reports zero removed header/footer lines because it skips the Markdown conversion that computes them. Compute the counts in Analyze mode too, or document that this field is only meaningful for Full-mode results.</violation>

<violation number="2" location="src/lib.rs:4382">
P3: `conversion.markdown.clone()` copies the entire converted Markdown document before returning it. Move `conversion.markdown` directly because the result still reads only the independent `removed_header_footer_lines` field.</violation>
</file>

<file name="src/markdown/mod.rs">

<violation number="1" location="src/markdown/mod.rs:914">
P2: When conversion receives no items, the returned per-page count vector is empty instead of containing one zero for each document page. Return a zero-filled vector sized to `document_page_count` on the empty-input path so callers can safely index counts by page.</violation>
</file>

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

P2: When conversion receives no items, the returned per-page count vector is empty instead of containing one zero for each document page. Return a zero-filled vector sized to document_page_count on the empty-input path so callers can safely index counts by page.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/markdown/mod.rs, line 914:

<comment>When conversion receives no items, the returned per-page count vector is empty instead of containing one zero for each document page. Return a zero-filled vector sized to `document_page_count` on the empty-input path so callers can safely index counts by page.</comment>

<file context>
@@ -909,10 +909,11 @@ impl TableDetectionOutput {
-    markdown: String,
+pub(crate) struct MarkdownConversionOutput {
+    pub(crate) markdown: String,
+    pub(crate) removed_header_footer_lines: Vec<u32>,
     #[cfg(feature = "ocr")]
-    detected_tables: Vec<(u32, crate::tables::Table)>,
</file context>

[cubic-dev-ai]

P3: Analyze mode always reports zero removed header/footer lines because it skips the Markdown conversion that computes them. Compute the counts in Analyze mode too, or document that this field is only meaningful for Full-mode results.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/lib.rs, line 4140:

<comment>Analyze mode always reports zero removed header/footer lines because it skips the Markdown conversion that computes them. Compute the counts in Analyze mode too, or document that this field is only meaningful for Full-mode results.</comment>

<file context>
@@ -4134,6 +4137,7 @@ fn process_document(
             processing_time_ms: start.elapsed_ms(),
             pages_needing_ocr,
             ocr_reasons_by_page: page_ocr_reasons_vec(detection_ocr_reasons),
+            removed_header_footer_lines: vec![0; page_count as usize],
             title,
             confidence,
</file context>

[cubic-dev-ai]

P3: conversion.markdown.clone() copies the entire converted Markdown document before returning it. Move conversion.markdown directly because the result still reads only the independent removed_header_footer_lines field.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/lib.rs, line 4382:

<comment>`conversion.markdown.clone()` copies the entire converted Markdown document before returning it. Move `conversion.markdown` directly because the result still reads only the independent `removed_header_footer_lines` field.</comment>

<file context>
@@ -4363,7 +4374,12 @@ fn process_document(
+            let md = if options.mode == ProcessMode::Analyze {
+                None
+            } else {
+                Some(conversion.markdown.clone())
             };
 
</file context>

Suggested change

	Some(conversion.markdown.clone())
	Some(conversion.markdown)