Funnel is a proof-of-concept metric ingestion service designed to efficiently handle and aggregate large volumes of time series data. It leverages modern, scalable technologies to provide reliable ingestion, storage, and real-time aggregation of sensor metrics.

• Funnel - High Volume Data Aggregator (PoC)
  • Table of Contents
  • Overview
  • Architecture
  • Features
  • Tech Stack
  • Setup & Usage
    • Prerequisites
    • Steps
    • Project Structure
    • Testing & Load Generation

Funnel simulates a high-throughput metric ingestion pipeline. Sensor data is published to Kafka, batched, and stored in a MongoDB time series collection. Aggregation queries can be run on demand to summarize data over configurable time windows.

Description:

• Producers send sensor data to Kafka.
• Go-based consumers batch and flush data to MongoDB every few seconds.
• MongoDB stores data in a time series collection, optimized for fast inserts and aggregations.
• REST API endpoints allow for data ingestion and aggregate queries.
• Load testing tool simulates high-volume data ingestion and validates aggregation performance.

• High-throughput ingestion using Kafka
• Efficient batch processing and storage in MongoDB time series collections
• Flexible aggregation queries (minute, hour, day granularity)
• Scalable, concurrent Go workers
• Load testing utility for benchmarking
• Easy deployment with Docker Compose

• Golang: Concurrency, batch processing, REST API
• Kafka: Message queue for decoupled, scalable ingestion
• MongoDB: Time series storage and aggregation
• Docker Compose: Simplified multi-service orchestration
• Prometheus: (Coming soon) Real-time metrics and monitoring

• Docker & Docker Compose

1. Clone the repository

   git clone https://github.com/yourusername/funnel.git
   cd funnel

2. Start core services

   docker compose up -d

3. Start the main application (with Kafka profile)

   docker compose --profile kafka up -d

4. Start the load testing tool (optional)

   docker compose --profile loadtest up -d

.
├── internal/
│   ├── broker/         # Kafka integration
│   ├── db/             # MongoDB storage and aggregation logic
│   ├── domain/         # Core domain models and interfaces
│   └── worker/         # Batch processing workers
├── load-testing/       # Load test tool (Go)
├── scripts/            # MongoDB initialization scripts
├── Dockerfile          # Main app Dockerfile
├──   # Multi-service orchestration
└── 

• The load-testing service simulates concurrent users sending sensor data and running aggregate queries.
• Configuration (users, duration, etc.) can be set via environment variables in docker-compose.yml.