Full working example available on GitHub: sanitize-rag-ingest-files-python

The AI platform challenge: why RAG ingest leaks metadata

AI platform and MLOps teams wire upload folders into chunkers and embedding jobs every week. The body text is what product wants searchable. The file still carries Author on DOCX, Info and XMP on PDF, and EXIF GPS on phone photos. Once those fields land in the vector store or tool traces, they show up in answers and audit exports long after the original upload is gone.

When weekly ingest volume jumps from a few hundred files to tens of thousands, desktop cleaners and ad-hoc scripts stop being a process. Teams either skip hygiene, invent fragile format-specific strips, or discover the leak during a privacy review.

What platform teams need is a single gate that inspects mixed uploads, strips recognized metadata packages, and proves the cleaned copy is quieter before chunking starts - without building a separate cleaner per format.

Industry snapshot: Privacy-aware RAG write-ups treat file-level EXIF and Office identity as a first-class preprocessing step, not an afterthought next to NLP redaction.

Why generic cleaners don’t work here

Upload pipelines are not browser tabs. Teams usually try one of these before an SDK gate:

  • Manual desktop cleaners: Fail when ingest is automated and headless.
  • Pillow / EXIF-only scripts: Miss Office authors and PDF XMP on the same folder walk.
  • Byte-level Info edits: Break on compressed PDF streams and never see XMP packets.

GroupDocs.Metadata for Python via .NET closes that gap with one API across Office, PDF, and images, including sanitize() for a full package wipe.

The solution: GroupDocs.Metadata in a RAG ingest workflow

GroupDocs.Metadata sits in front of the chunker. The worker opens each upload, lists properties for telemetry, calls sanitize(), saves a cleaned sibling, then re-opens that sibling to count what remains. Only then does embedding start. The same three functions cover DOCX, PDF, and JPEG, so the gate stays format-agnostic.

To follow the implementation below, you’ll need:

  • Python 3.13 (py -3.13)
  • pip install groupdocs-metadata-net (26.9.0+)
  • A GroupDocs.Metadata license for unrestricted save()
  • Sample uploads with authors or EXIF still present

Inspect before you embed

with Metadata(input_path) as metadata:
    info = metadata.get_document_info()
    print(f"format={info.file_type.file_format}")
    print(f"pages={info.page_count}")
    print(f"encrypted={info.is_encrypted}")

    count = 0
    for prop in metadata.find_properties(lambda p: True):
        print(f"{prop.name}={prop.value}")
        count += 1

This baseline is the evidence your privacy review will ask for. Log the count next to the job id.

Sanitize for RAG ingest

with Metadata(input_path) as metadata:
    removed = metadata.sanitize()
    metadata.save(output_path)
    print(removed)

sanitize() clears recognized packages in one pass. Always write to a new path so the original upload stays available for incident review.

Verify the cleaned copy

with Metadata(cleaned_path) as metadata:
    remaining = 0
    for prop in metadata.find_properties(lambda p: True):
        print(f"{prop.name}={prop.value}")
        remaining += 1
    print(remaining)

Fail the job when remaining identity fields exceed your policy. Do not start embedding on hope.

Business impact

A metadata gate cuts a common privacy finding: personal and device identity in retrieval indexes. It also reduces format-specific glue code - one worker handles Office, PDF, and images. Operators get before/after counts they can attach to tickets instead of screenshots from desktop tools.

Additional Resources

Conclusion

RAG hygiene is a file-level problem as much as a text-redaction problem. Inspect uploads, sanitize with GroupDocs.Metadata for Python, verify the cleaned copy, then embed. Clone the sample repo, drop it in front of your chunker, and keep Author and GPS out of the vector store.