RAG Pipeline Preprocessing — Capability Boundary Description

PDF

.pdf

PDF_CONTENT_EXTRACTION / TEXTIN / AZURE_DI / ALI_OCR / LLM_CONTENT_EXTRACTION

Basic parsing only applies to text-based PDFs; scanned documents require an OCR step

Word Document

.docx

DOCX_CONTENT_EXTRACTION

Converted to Markdown via pypandoc, supports paragraphs/tables/embedded images

Legacy Word

.doc

Must first be converted via FILE_CONVERT_WITH_SPIRE or FILE_CONVERT_WITH_LIBREOFFICE

Cannot be parsed directly; must first be converted to .docx or .pdf

Markdown

.md

MD_CONTENT_EXTRACTION

Natively supported

Plain Text

.txt

TXT_CONTENT_EXTRACTION

Natively supported

Excel

.xlsx .xls

TABULAR_CONTENT_EXTRACTION

Subject to row/column limits, see §2

CSV / TSV

.csv .tsv

TABULAR_CONTENT_EXTRACTION

—

PowerPoint

.pptx .ppt

Must first be converted via FILE_CONVERT_WITH_SPIRE or FILE_CONVERT_WITH_LIBREOFFICE

Direct content extraction is not supported; must first be converted to PDF

Image

.png .jpg .jpeg .gif .webp .svg

IMAGE_CONTENT_EXTRACTION

Uses a Vision LLM (such as GPT-4o) to describe image content, up to 300 words

Video

.mp4 .avi .mkv .mov

VIDEO_CONTENT_EXTRACTION

Extract audio track → transcribe to text

Audio

.mp3 .wav .flac .aac

AUDIO_CONTENT_EXTRACTION

Split into 60-second segments and transcribe segment by segment

.exe

Hardcoded blacklist (_UNSUPPORTED_EXTENSIONS)

.zip / .rar and other archives

No decompression step; the Pipeline does not process them

.html / .xml

No dedicated parsing step

.json / .yaml

No dedicated parsing step

.eml / .msg emails

Not supported

.dwg / .dxf CAD files

Not supported

.rtf

No dedicated step (conversion via LibreOffice may be attempted, but not guaranteed)

Encrypted / password-protected files

Encrypted files of all formats cannot be processed

Character count

15,000 characters

✅ Via the FILE_LIMIT_CHECK_CONFIG environment variable

Applies to text-based files such as PDF/DOCX/TXT/MD/CSV/XLSX

Page count

25 pages

✅ Same as above

PDF by physical pages, Excel by sheet count, PPT by slide count

Parallel PDF page conversion

50 pages per batch (PDFConversionDefaultOptions.CHUNK_SIZE)

Very large PDFs (500+ pages) may consume high memory during pdf2image conversion, potentially causing OOM

Table row count

Single sheet ≤ 20,000 rows

Exceeding the limit causes direct failure and parsing rejection

Table column count

Single sheet ≤ 200 columns

Exceeding the limit causes direct failure and parsing rejection

Table images

Single file ≤ 150 images (_MAX_IMAGES)

Excess images are ignored

Vectorization batch

100 segments per batch (batch_size)

—

Segment image density

Single segment ≤ 5~8 images

Exceeding the limit triggers early splitting to avoid token overflow

Audio slicing

60 seconds / segment

Long audio/video files generate a large number of transcription API calls

TextIn OCR

API-level maximum of 1,000 pages

Quota exhaustion will cause errors

DF > 200 pages + pdf2image

🔴 High

pdf2image renders each page as an in-memory bitmap at 300 DPI; 200 pages ≈ several GB of memory

Use OCR steps (TextIn/Azure DI) instead of basic parsing

Excel single sheet > 10,000 rows

🟡 Medium

Fully loaded into memory, generating a large number of segments

Pre-split the file or increase worker memory

Video > 2 hours

🟡 Medium

ffmpeg audio extraction + 120 transcription requests of 60s each

Limit video duration or pre-split it

Single file containing 100+ embedded images

🟡 Medium

Each image requires a Vision LLM call, resulting in high time and cost

Extract only key images

Bulk import of 1,000+ files

🟡 Medium

Celery worker queue backlog, gevent concurrency limit of 100

Import in batches and monitor queue depth

Basic（pypdf / pdfplumber）

Direct extraction from text layer

Native text PDFs (exported from Word, generated by LaTeX, etc.)

Scanned documents, image PDFs, complex layouts

No external dependencies

TextIn OCR

Cloud OCR + layout analysis

Scanned documents, receipts, mixed-layout PDFs

—

TEXTIN_APP_ID / TEXTIN_APP_SECRET

Azure Document Intelligence

Cloud Layout/Read models

Table structure preservation, multi-column layouts

—

azure_ocr_endpoint / azure_ocr_key

Ali OCR

Alibaba Cloud OCR

Chinese scanned-document scenarios

—

Alibaba Cloud API credentials

LLM Parsing（PDF LLM）

Page screenshots → Vision LLM recognition

Extremely complex layouts, mixed text-image layouts

Large files (high cost, slow speed)

Vision LLM (GPT-4o, etc.)

pypandoc

Pandoc format conversion

DOCX → Markdown

Complex macros, ActiveX controls

Pandoc binary

LibreOffice / Spire

Format conversion engine

.doc→.docx, .ppt→.pdf

—

LibreOffice or Spire runtime

Plain text extraction

✅

✅

✅

✅

Scanned / image PDF

❌ Returns empty

✅

✅

✅

Table structure preservation

❌ Formatting lost

✅

✅（Layout mode）

✅

Multi-column layout

❌ Text disorder

✅

✅

✅

Hyperlink extraction

✅（pdfplumber）

❌

❌

❌

Embedded image extraction

✅（converted to PNG）

✅

✅

✅

Processing speed

⚡ Fast

🐢 Medium

🐢 Medium

🐌 Slow

Cost

Free

💰 Charged per page

💰 Charged per page

💰💰 Charged per token

Paragraph text

✅

Tables

✅（converted to Markdown tables）

Embedded images

✅（extracted as separate files + Markdown markers）

TOC / bookmarks

⚠️ Partial (may be lost during Pandoc conversion)

Macros / VBA

❌ Ignored

Revision marks / comments

❌ Lost

Complex nested tables

⚠️ Pandoc behavior is unstable

Multiple sheets

✅ Processed sheet by sheet

Formulas

⚠️ Only calculated values are read (data_only=True)

Charts / pivot tables

❌ Ignored

Embedded images

✅（up to 150 images / file）

Merged cells

⚠️ May cause data misalignment

More than 20,000 rows

❌ Parsing rejected

More than 200 columns

❌ Parsing rejected

Speech-to-text

✅（60s segmented transcription）

Multilingual recognition

Depends on the transcription service model

Speaker diarization

❌

Background music / noise

⚠️ Affects transcription quality

Video visual content recognition

❌ Audio track only is extracted

Subtitle / CC extraction

❌

Image content description

✅ Vision LLM description（≤ 300 words）

OCR text recognition

⚠️ Depends on Vision LLM capability; not professional OCR

Chart / flowchart understanding

⚠️ Limited (determined by LLM comprehension)

SVG vector graphics

⚠️ Depends on whether the LLM can render them

Fixed character count（FIXED_SIZE）

General text

chunk_size=1024

May split in the middle of a sentence; single segment ≤ 5~8 images

By page（PAGE）

PDF documents

—

Only valid for PDF; segment quality may vary greatly when page content differs significantly

By title（TITLE）

Markdown documents

chunk_size=1024

Only recognizes # / ## / ### headings; without headings, degrades to one segment for the entire document

Tabular（TABULAR）

Excel / CSV

—

Dedicated to tabular files

LLM intelligent（LLM）

Scenarios requiring semantic coherence

—

Slow and costly; uneconomical for large files

Refine（REFINE）

Secondary optimization of coarse segmentation results

—

Must be used together with upstream coarse segmentation steps

METADATA_EXTRACTION

Document-level metadata (title/author/date, etc.)

LLM

Schema must be predefined

SEGMENT_METADATA_EXTRACTION

Segment-level metadata

LLM

One LLM call per segment; costly for large files

KEYWORDS_EXTRACTION

Keyword extraction

LLM

—

SEGMENT_SUMMARY

Segment summary

LLM

One LLM call per segment

DOCUMENT_SUMMARY

Full-document summary

LLM

Long documents require truncation or batch aggregation

TABLE_CAPTIONING

Table description generation

LLM

—

TABLE_CAPTIONING_ADVANCED

Table-level summary + row-level narrative grouping

LLM

Effectiveness on complex tables depends on LLM comprehension

IMAGE_CAPTIONING

Image description generation

Vision LLM

One LLM call per image

EMBEDDING

Text vectorization

batch_size=100

Depends on the Embedding model; image placeholders are automatically stripped

TOKENIZER

Full-text indexing tokenization

spaCy language model

PostgreSQL tsvector length limit

EMBEDDING_STORE

Write vectors into VectorDB

—

Batch writes; depends on VectorDB performance

TOKENIZER_STORE

Write tokens into PostgreSQL

—

—

.doc

.docx / .pdf

LibreOffice / Spire

Requires the corresponding runtime to be installed on the server

.ppt

.pdf

LibreOffice / Spire

Animations/transitions are lost

.pptx

.pdf

LibreOffice / Spire

Same as above

TextIn OCR

TEXTIN_APP_ID / TEXTIN_APP_SECRET / TEXTIN_APP_ENDPOINT

PDF/image OCR

The corresponding OCR step is unavailable

Azure Document Intelligence

azure_ocr_endpoint / azure_ocr_key

PDF layout analysis

The corresponding step is unavailable

Ali OCR

Alibaba Cloud credentials

Chinese OCR

The corresponding step is unavailable

Embedding model

LLM Gateway configuration

Text vectorization

The EMBEDDING step is unavailable

Vision LLM

LLM Gateway configuration

Image description / LLM PDF parsing

IMAGE/LLM steps are unavailable

General LLM

LLM Gateway configuration

Summary/keyword/metadata extraction

Field extraction steps are unavailable

LibreOffice

Server-side binary

Format conversion

.doc/.ppt cannot be processed

Pandoc

Server-side binary

DOCX → Markdown

DOCX parsing is unavailable

ffmpeg

Server-side binary

Video audio-track extraction

Video processing is unavailable

spaCy

Python package + language model

Tokenization/sentence splitting

TOKENIZER step and refinement features are unavailable

1

Scanned PDF + Basic parsing

Extracted content is empty

Switch to TextIn / Azure DI / LLM steps

2

Multi-column PDF + Basic parsing

Text order is disordered, paragraphs overlap

Use Azure DI Layout mode

3

Complex tables in PDF + Basic parsing

Table structure is lost, data becomes prose

Use TextIn / Azure DI

4

Encrypted / password-protected PDF

pypdf throws an exception

Remove password protection during preprocessing

5

Direct upload of .doc (legacy Word)

No native parser

Ensure the Pipeline includes a file conversion step

6

Excel exceeds 20,000 rows

Parsing rejected, error thrown

Split into multiple files

7

Excel merged cells

Data may be misaligned

Unmerge before upload

8

Excel formulas

Only result values are read, not the formulas themselves

Expected behavior

9

Video extracts audio track only

Visual content cannot be recognized

If visual information is needed, extract key frames and process separately

10

Image OCR accuracy

Vision LLM is not professional OCR, accuracy is limited

Use TextIn for OCR-demanding scenarios

11

Large files + LLM parsing

Extremely high cost (charged by page/token)

Use LLM parsing only for key documents

12

Markdown without heading structure + TITLE segmentation

The entire document becomes one segment

Switch to FIXED_SIZE segmentation

13

PPT animations/video embedding

Lost after conversion to PDF

No solution

14

DOCX containing macros/VBA/ActiveX

All macro content is ignored

Expected behavior

15

DOCX revision marks/comments

Lost

Accept all revisions before upload

PDF (Basic parsing)

≤ 50 pages

pdf2image memory consumption; OOM risk beyond 50 pages

PDF (OCR parsing)

≤ 100 pages

Limited by OCR service API constraints and cost

DOCX

≤ 20,000 characters

Pandoc conversion memory usage

Excel

≤ 20,000 rows × 200 columns / sheet

Hard limit

CSV

≤ 20 MB

Fully loaded into memory

Audio

≤ 60 minutes

60 transcription API calls

Video

≤ 30 minutes

ffmpeg extraction + transcription; the longer the duration, the higher the failure probability

Image

≤ 10 MB / image

Vision LLM input limit