What about native support for jsonl formatted data

[brroMonta]

is this on the roadmap, or are there workarounds in the current implementation?

[ZhenhangTung]

@brroMonta Hi could you explain more on this topic? Thanks!

[brroMonta]

I hava a ton of our internal data sources with pipelines built to export data in JSONL format, which based on our experimenting, provides the cleanest input for our systems. But Ragflow does not successfully import or parse these files.

JSONL, also known as JSON Lines or newline-delimited JSON, is a text-based format where each line is a valid JSON object. Unlike traditional JSON which typically wraps an entire data structure within a single set of curly braces, JSONL separates each JSON object into its own line. This structure makes it particularly useful for processing large datasets in a streaming fashion.

Here's a breakdown of key aspects of JSONL:
Key Features:
Line-by-line processing: Each line in a JSONL file represents a separate, valid JSON object, allowing for efficient processing of large datasets without needing to load the entire file into memory at once.
Streaming: JSONL is well-suited for streaming data, where data is processed as it arrives, making it ideal for real-time applications and log files.
Simple structure: Each line contains a single JSON object, making it easy to parse and understand, even for large files.
File extension: JSONL files are typically saved with the .jsonl or .ldjson extension.

Use Cases:

Large datasets: JSONL is preferred for handling massive datasets that exceed available memory, as it can be processed in chunks.
Streaming data: It is commonly used in data pipelines and real-time applications where data needs to be processed as it becomes available.
Machine learning:JSONL is used in various machine learning tasks, including training and fine-tuning models.
Log files:Its line-based structure makes it a natural fit for storing and processing log data.

Comparison with JSON:
JSON: Typically stores a single JSON object or array, making it suitable for smaller datasets or when you need to represent complex nested structures.
JSONL:Stores each JSON object on a separate line, making it more efficient for streaming and processing large datasets.

[Dongzhenpu]

Hi there, I want to share my opinion
RAG retrieves relevant documents from an external knowledge base and then uses that information to generate more accurate and context-aware responses.
I think a .jsonl file is suitable for RAG if each line contains natural language text that can be used for semantic retrieval
eg.

{"title": "What is RAG?", "content": "Retrieval-Augmented Generation is a method that..."}
{"question": "How does LangChain work?", "answer": "LangChain connects language models with tools..."}

while if the entries are mostly structured data or code, with no meaningful natural language for the model to understand or retrieve from, is not suitable
eg.

{"id": 1, "product_id": 1234, "price": 99.9, "stock": 20}
{"id": 2, "product_id": 5678, "price": 75.0, "stock": 15}

[brroMonta]

in this case, the entries are natural language. they are all the clauses and terms and conditions of sales agreements.

[ZhenhangTung]

Appreciate all the inputs.

[yongtenglei]

Hi, @brroMonta and @Dongzhenpu

Is that OK? Supports both .jsonl and .ldjson. However, the usage in real scenarios still needs to be validated.