nlm-ingestor
llmsherpa
| nlm-ingestor | llmsherpa | |
|---|---|---|
| 3 | 6 | |
| 858 | 1,045 | |
| 5.6% | 12.2% | |
| 7.0 | 6.6 | |
| 10 days ago | 16 days ago | |
| Python | Jupyter Notebook | |
| Apache License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
nlm-ingestor
-
Pg_vectorize: The simplest way to do vector search and RAG on Postgres
>tree-based approach to organize and summarize text data, capturing both high-level and low-level details.
https://twitter.com/parthsarthi03/status/1753199233241674040
processes documents, organizing content and improving readability by handling sections, paragraphs, links, tables, lists, page continuations, and removing redundancies, watermarks, and applying OCR, with additional support for HTML and other formats through Apache Tika:
https://github.com/nlmatics/nlm-ingestor
-
Show HN: Open-source Rule-based PDF parser for RAG
Here's another notebook from the repo with examples: https://github.com/nlmatics/nlm-ingestor/blob/main/notebooks...
llmsherpa
-
LlamaCloud and LlamaParse
To get good RAG performance you will need a good chunking strategy. Simply getting all the text is not good enough and knowing the boundaries of table, list, paragraph, section etc. is helpful.
Great work by llamaindex team. Also feel free to try https://github.com/nlmatics/llmsherpa which takes into account some of the things I mentioned.
-
Show HN: Open-source Rule-based PDF parser for RAG
I wrote about split points and the need for including section hierarchy in this post: https://ambikasukla.substack.com/p/efficient-rag-with-docume...
All this is automated in the llmsherpa parser https://github.com/nlmatics/llmsherpa which you can use as an API over this library.
What are some alternatives?
txtai - 💡 All-in-one open-source embeddings database for semantic search, LLM orchestration and language model workflows
unstructured - Open source libraries and APIs to build custom preprocessing pipelines for labeling, training, or production machine learning pipelines.
SemanticSlicer - A recursive text chunker that attempts to preserve context.
llama_parse - Parse files for optimal RAG
Parsr - Transforms PDF, Documents and Images into Enriched Structured Data
marker - Convert PDF to markdown quickly with high accuracy
paperetl - 📄 ⚙️ ETL processes for medical and scientific papers
open-webui - User-friendly WebUI for LLMs (Formerly Ollama WebUI)
llama-hub - A library of data loaders for LLMs made by the community -- to be used with LlamaIndex and/or LangChain
grobid - A machine learning software for extracting information from scholarly documents