Welcome to a state-of-the-art, event-driven pipeline engineered to perform deep semantic analysis of software codebases. This system discovers, analyzes, and understands the intricate relationships within code, building a queryable, high-fidelity knowledge graph using Large Language Models (LLMs) and graph database technology.

This is not just a static analysis tool; it's a dynamic, scalable, and resilient platform for building a "living" model of your code's cognitive architecture.

• Overview
• Core Concepts
  • Cognitive Triangulation
  • Event-Driven & Asynchronous Architecture
  • The Transactional Outbox Pattern
  • Polyglot Persistence
• Architectural Blueprint
  • High-Level Diagram
  • Component Breakdown
• The Data Pipeline: A Step-by-Step Journey
• Key Features
• Comparison to Existing Technologies
• Getting Started
  • Prerequisites
  • Installation
  • Configuration
  • Running the Pipeline
• API Reference
  • Endpoints
  • WebSocket Events
• Directory Structure
• Database Schema
  • SQLite Schema
  • Neo4j Graph Model
• Contributing
• License

This system is architected to deconstruct a software repository and rebuild it as a knowledge graph. It achieves this by:

1. Discovering all relevant files and directories, intelligently respecting .gitignore rules.
2. Analyzing file contents using LLMs to extract key Points of Interest (POIs), such as classes, functions, variables, and imports.
3. Triangulating Relationships through a sophisticated multi-pass analysis, combining deterministic logic with contextual LLM queries to identify how POIs connect.
4. Building a Knowledge Graph in Neo4j, representing the codebase as a network of nodes (POIs) and edges (relationships).
5. Providing Real-time Insight into the analysis process through a robust REST and WebSocket API.
6. Maintaining Itself by automatically cleaning up graph elements corresponding to deleted files.

The result is a powerful, queryable graph that enables advanced code navigation, dependency analysis, architectural validation, and impact analysis for refactoring.

The power of this system comes from the fusion of several key architectural patterns and concepts.

This is the core philosophy of the pipeline. Instead of relying on a single method to find relationships, I use multiple, independent "witnesses" to build confidence.

1. Deterministic Pass: Identifies obvious, rule-based relationships (e.g., a file CONTAINS a class). This pass is fast and highly accurate.
2. Intra-File LLM Pass: An LLM analyzes the POIs within a single file to find local relationships (e.g., function A CALLS function B in the same file).
3. Intra-Directory LLM Pass: An LLM analyzes all POIs within a directory to find connections between files (e.g., one file IMPORTS an exported class from another).
4. Global LLM Pass: After summarizing entire directories, an LLM analyzes these summaries to find high-level architectural relationships between different modules or services.

The ConfidenceScoringService then aggregates the "evidence" from these passes. A relationship reported by multiple passes receives a higher confidence score, filtering out noise and LLM hallucinations.

The system is built on BullMQ and Redis, allowing for a highly scalable and decoupled architecture. Each stage of the pipeline is a queue, and each processing unit is a worker. This design means:

• Scalability: You can run hundreds or thousands of workers across multiple machines to process massive codebases in parallel.
• Resilience: If a worker fails while processing a job, BullMQ ensures the job is automatically retried, preventing data loss.
• Decoupling: Each worker has a single responsibility. I can update, improve, or replace a worker (e.g., FileAnalysisWorker) without affecting the rest of the system.

To ensure maximum reliability, I use the transactional outbox pattern via the TransactionalOutboxPublisher.

When a worker (like FileAnalysisWorker) processes a file, it doesn't immediately publish an event to the next queue. Instead, it writes its findings and an "event to be published" into the SQLite database within a single transaction.

A separate, dedicated TransactionalOutboxPublisher service polls this outbox table and reliably publishes the events to BullMQ. This guarantees at-least-once delivery and prevents a scenario where a worker crashes after performing its work but before it could enqueue the next job, which would otherwise halt the pipeline for that file.

The system leverages two different databases for what they do best:

• SQLite (better-sqlite3): Acts as the fast, transactional, and transient data store. It's perfect for the outbox pattern, storing intermediate POIs, and tracking job states. Its file-based nature makes it easy to manage per-run.
• Neo4j (Graph Database): Serves as the final, persistent knowledge graph. It is optimized for storing and querying complex network relationships, making it the ideal destination for our analysis.

+------------------+      +--------------------+      +-----------------------+      +-------------------+
|                  |----->|  File Analysis     |----->|  Relationship         |----->|                   |
|   EntityScout    |      |  Queue & Workers   |      |  Resolution Queues    |      |   GraphBuilder    |
| (Discover Files) |      |  (Extract POIs)    |      |  & Workers (Triangulate) |      | (Persist to Neo4j)|
|                  |----->| (Chunking, LLM)    |----->|  (Confidence Score)   |----->|                   |
+------------------+      +----------+---------+      +-----------+-----------+      +-------------------+
        |                          |                        |
        v                          v                        v
+-------------------------------------------------------------------------+
|                                                                         |
|  SQLite (Intermediate Storage: POIs, Relationships, Outbox, Evidence)   |
|                                                                         |
+-------------------------------------------------------------------------+
        ^                          ^                        ^
        |                          |                        |
+-------+--------------------------+------------------------+-------------+
|                                                                         |
|             Transactional Outbox Publisher (Reliable Eventing)          |
|                                                                         |
+-------------------------------------------------------------------------+
        |
        | Publishes to...
        v
+-------------------------------------------------------------------------+
|                                                                         |
|                BullMQ / Redis (Job Queues & Caching)                    |
|                                                                         |
+-------------------------------------------------------------------------+

The pipeline is a collection of specialized agents and workers, each with a distinct role.

Here is how a single file travels through the system:

1. Initiation: A user submits a new pipeline run via the POST /api/pipeline/start endpoint, specifying a target directory.
2. Discovery: EntityScout awakens, scans the directory, and creates a "file-analysis" job for every discovered file, placing it in the file-analysis-queue.
3. POI Extraction: A FileAnalysisWorker picks up a job. It reads the file, asks the LLM to identify all POIs (functions, classes, etc.), and writes these POIs along with an event into the SQLite outbox table in a single transaction.
4. Reliable Eventing: The TransactionalOutboxPublisher polls the outbox, sees the new event, and publishes it to the relationship-resolution-queue.
5. Relationship Triangulation: A RelationshipResolutionWorker receives the POI. It orchestrates the multi-pass analysis, querying the LLM with different contexts to find potential relationships. Each piece of evidence is written to the outbox.
6. Evidence Validation: The ValidationWorker consumes these evidence events from the outbox. It stores them in the relationship_evidence table and uses Redis to track how much evidence has been gathered for a specific potential relationship.
7. Reconciliation & Scoring: Once all evidence for a relationship is collected, the ValidationWorker enqueues a job for the ReconciliationWorker. This worker fetches all evidence, calculates a final confidence_score using ConfidenceScoringService, and if the score exceeds a threshold, writes the final, validated relationship to the main relationships table in SQLite.
8. Graph Construction: After all analysis jobs are complete, the GraphBuilder agent runs. It reads all validated relationships from SQLite and bulk-inserts them into Neo4j, creating the final, queryable knowledge graph.
9. Monitoring: Throughout this entire process, status updates are broadcast via WebSocket, allowing a user to monitor the progress in real-time.
10. Cleanup: If a file is ever deleted from the source repository, the SelfCleaningAgent can be run to find and remove its corresponding data from both SQLite and Neo4j, keeping the graph synchronized with the codebase.

• Deep Semantic Analysis: Goes far beyond simple pattern matching by using LLMs to understand code intent and context.
• Highly Scalable: The distributed worker/queue architecture allows for massive parallel processing of large and complex codebases.
• Extremely Reliable: The transactional outbox pattern guarantees that no analysis work is lost, even if parts of the system crash.
• Confidence-Scored Relationships: The "Cognitive Triangulation" approach filters out LLM noise and produces a high-fidelity graph.
• Real-time Monitoring: A built-in API and WebSocket server provide immediate insight into the pipeline's status, progress, and logs.
• Automated Graph Construction: Fully automates the process of turning a directory of code into a queryable Neo4j knowledge graph.
• Language Agnostic: Because it relies on LLMs, the system can be adapted to analyze code in any programming language.

Follow these steps to get the pipeline running on your local machine.

• Node.js: v18.x or later recommended.
• Redis: An instance of Redis for BullMQ. Can be run locally or via Docker.
• Neo4j: A Neo4j instance (v5.x recommended). Can be run locally or via Docker.
• DeepSeek API Key: You need an API key from DeepSeek or another compatible LLM provider.

1. Clone the Repository:

   git clone https://github.com/yourusername/cognitive-triangulation-pipeline.git
   cd cognitive-triangulation-pipeline

2. Install Dependencies:

   npm install

1. Create an Environment File: Copy the example environment file:

   cp .env.example .env

2. Edit .env: Open the .env file and fill in the details for your local setup:

   # --- LLM API Configuration ---
   # Get your key from https://platform.deepseek.com/
   DEEPSEEK_API_KEY=sk-your_secret_api_key
   
   # --- Redis Configuration ---
   # URL for your Redis instance
   REDIS_URL=redis://127.0.0.1:6379
   
   # --- Neo4j Database Configuration ---
   # URI for your Neo4j instance
   NEO4J_URI=bolt://localhost:7687
   NEO4J_USER=neo4j
   # Use a secure password, especially in production
   NEO4J_PASSWORD=your_secure_password
   NEO4J_DATABASE=neo4j
   
   # --- SQLite Database Configuration ---
   # Path where the intermediate SQLite database will be stored
   SQLITE_DB_PATH=./database.db

The system includes a user-friendly API server for managing runs.

1. Initialize the Database Schema: This command will create the database.db file and set up the necessary tables.

   npm run init-db

2. Start the API Server: This will start the web server and the WebSocket server for real-time updates.

   npm run start-api

   You should see output indicating the server is running on http://localhost:3002.

3. Submit a Pipeline Run: Use curl or any API client to send a POST request to the /api/pipeline/start endpoint. Replace /path/to/your/codebase with the absolute path to the directory you want to analyze.

   curl -X POST http://localhost:3002/api/pipeline/start \
     -H "Content-Type: application/json" \
     -d '{"targetDirectory": "/path/to/your/codebase"}'

4. Monitor the Progress:

   • Terminal: Watch the console output from the npm run start-api command.
   • REST API: Get the status of a specific run: GET http://localhost:3002/api/pipeline/status/:pipelineId
   • WebSocket: Connect a WebSocket client to ws://localhost:3002 to receive real-time JSON patches of the pipeline status.

Once the pipeline completes, you can explore the generated knowledge graph in your Neo4j Browser.

The PipelineApiService provides the following endpoints:

Connect to ws://localhost:3002 to receive real-time updates.

src/
├── agents/             # High-level orchestrators (EntityScout, GraphBuilder)
├── workers/            # Single-responsibility, queue-driven workers
├── services/           # Core services (LLM Client, Outbox Publisher)
├── utils/              # Shared utilities (DB drivers, queue manager, logger)
├── config/             # Application configuration
├── main.js             # Main CLI entry point for the pipeline
└── ...

Defined in src/utils/schema.sql, the SQLite database serves as the pipeline's operational datastore.

• files: Tracks discovered files and their processing status.
• pois: Stores Points of Interest extracted from files.
• relationships: Stores validated relationships with their final confidence scores.
• relationship_evidence: Stores all raw "evidence" for potential relationships from different analysis passes before reconciliation.
• directory_summaries: Stores the LLM-generated summaries for each directory.
• outbox: The transactional outbox table for reliable event publishing.

The final graph in Neo4j is simple and powerful:

• Nodes:
  • :POI: All Points of Interest are stored as nodes with this label.
  • Properties: id, type, name, filePath, startLine, endLine.
• Relationships:
  • :RELATIONSHIP: A generic relationship type between two :POI nodes.
  • Properties: type (e.g., 'CALLS', 'IMPORTS', 'CONTAINS'), confidence.

Contributions are welcome! This project is ambitious, and there are many avenues for improvement, from performance tuning to enhancing the LLM prompts for even more nuanced analysis. Please open an issue to discuss your ideas or submit a pull request.

This project is licensed under the MIT License.