chunking

Fully neural approach for text chunking

Adaptive Chunking: automatically select the best chunking method per document for RAG. Accepted at LREC 2026.

smart-llm-loader is a lightweight yet powerful Python package that transforms any document into LLM-ready chunks. Spend less time on preprocessing headaches and more time building what matters. From RAG systems to chatbots to document Q&A, SmartLLMLoader handles the heavy lifting so you can focus on creating exceptional AI applications.

The RAG Experiment Accelerator is a versatile tool designed to expedite and facilitate the process of conducting experiments and evaluations using Azure Cognitive Search and RAG pattern.

A package for parsing PDFs and analyzing their content using LLMs.

Pure Rust PDF library for AI/RAG: structure-aware chunking, no ML, no C deps.

Open-source toolkit for reliable RAG pipelines: convert PDFs to Markdown, clean documents, inspect chunks, compare chunking strategies, and enrich metadata for LLM applications.

PDFStract - Extract, Chunking and Embedding Layer in Your RAG Pipeline - Available as CLI - WEBUI - API

A Python CLI to test, benchmark, and find the best RAG chunking strategy for your Markdown documents.

An LLM GUI application; enables you to interact with your files, offering dynamic parameters that can modify response behavior during runtime.

Build document-native LLM applications

🧠✂️ SemanticSlicer — A smart text chunker for LLM-ready documents.

Privacy-first document intelligence engine — parse PDFs, DOCX, PPTX, XLSX & CSV into AI-ready chunks for RAG pipelines. Includes HITL review, 3-layer memory chat, and a production FastAPI server.

Experimental RAG playground for exploring retrieval quality, corpus construction, and filter-chain design. Features configurable ranking and filtering pipelines, visual document grounding, chat interfaces, web search, Open WebUI integration, and rich debugging insights. Ollama and vLLM, running in Dev Containers or natively on Linux and Windows.

LLM-free, layout-aware PDF chunking for RAG pipelines. Preserves document structure via regex and font heuristics. Written in pure Rust.

Locally fine-tuned Russian RAG models: document splitter, query expansion, retrieval embedder. Teacher distillation → GGUF → llama.cpp on AMD Vulkan. Commercial-OK licenses.

🚀 Unified NLP Pipelines for Language Models

Visual tool to compare 6 RAG chunking strategies side-by-side with grading and query selection

Docsifer is a powerful tool for converting various data formats into Markdown for applications such as indexing, text analysis, and more. It supports PDF, PowerPoint, Word, Excel, Images, Audio, HTML, and other text-based formats, and leverages LLMs to enhance performance.