Veloctra Data Platform

Enterprise-grade, vectorized, multi-tenant ETL/ELT platform engineered with PyArrow C++ columnar engines, dynamic MongoDB state management, and double envelope AEAD security.

⚡ 120,000+ rows/sec 🛡️ MemoryGuard (75% Cap) 🔐 Double Envelope AEAD 🔄 Zero Data Loss DLQ

🚀 2-Minute Quickstart 🏛️ Explore Architecture

💡 Why Enterprises Choose Veloctra

⚡

Vectorized C++ Engine

Zero-copy streaming using PyArrow RecordBatches and Polars transforms, processing over 120,000 rows/second with sub-second latency.

🛡️

Intelligent MemoryGuard

Enforces a strict 75% RAM/CPU ceiling. Dynamically shards chunk sizes down to 1 record for multi-megabyte payloads, preventing OOM crashes.

🔐

Double Envelope AEAD

Double-layer encryption (AES-128-CBC + ChaCha20-Poly1305) with dynamic zero-downtime key rotation for all credentials and database DSNs.

🎯

Zero Data Loss (DLQ)

Corrupt rows or poison-pill payloads are automatically isolated to the Dead Letter Queue (DLQ) without aborting multi-million row pipelines.

🖥️

Visual Studio & 1-Click Publish

Design, edit, and validate pipelines visually or via YAML in the web studio, and deploy directly to the engine with automated connection extraction.

📊

Live Telemetry Gauges

Sub-second hardware gauges (CPU, RAM, Threads, GC) and interactive SVG throughput sparklines powered by real-time WebSockets.

🔄

Change Data Capture (CDC)

High-watermark deltas, zero-timestamp Checksum-Diff SHA-256 state hashing, and native MongoDB Change Streams with deterministic vector upsert/delete splitting.

📈

Intelligent KEDA Autoscaling

Automatic source catalog volume discovery, dynamic Prometheus workload gauges, and 2-tier scaling (MemoryGuard + KEDA horizontal elasticity 1 → 16 pods).

📊 Veloctra vs Legacy Stacks

Platform Capability	Legacy Stacks (Spark / Airflow)	Custom Python / Pandas	⚡ Veloctra Data Platform
🚀 Memory Footprint	4 GB – 16 GB JVM Heap per worker	Unbounded memory growth (OOM)	< 250 MB Process RSS
🛡️ Memory Governance	Static partition sizing	None (Manual GC)	Intelligent MemoryGuard (75% Cap)
⚡ Execution Speed	JVM / PySpark serialization overhead	Slow row-by-row iteration	PyArrow C++ Vectors (120k+ rows/s)
🔐 Credential Security	Plaintext files / Environment vars	Plaintext .env files	Double Envelope AEAD + Rotation
🎯 Fault Isolation	Entire job fails on 1 corrupt row	Script crashes on exception	Per-row DLQ Isolation (Zero Loss)
🔄 State Backend	Heavy RDBMS metadata store	None / Flat state files	MongoDB (veloctra_system) & SQLite
🖥️ Visual Modeler	Fragmented 3rd party tools	None	Built-in Studio + 1-Click Publish

🏛️ Architecture Blueprint

1. Client & Management Experience Layer

React 18 • TypeScript • Tailwind

Visual Pipeline Studio

apps/management-ui

Visual DAG Builder with 1-Click Validation
Live YAML Editor with Schema Autocomplete
Interactive Connection Credential Vault

Monaco Editor Live Canvas No-Code / Low-Code

Real-Time Observability Center

WebSocket Live Feed

Sub-second Throughput Sparklines & Gauges
Live MemoryGuard RAM / CPU Pressure Telemetry
Instant Row Counter & Execution Heartbeat

WebSockets Dynamic Gauges Alerting

CLI & Headless Orchestrator

ctl.sh & Python SDK

Single-command platform lifecycle control
CI/CD GitOps pipeline triggers & imports
Docker Compose container auto-bootstrap

Bash CLI REST Client Docker Compose

REST API (FastAPI) & Bi-Directional WebSockets

2. Enterprise Control Plane & Security Engine

FastAPI • JWT RBAC • Double Envelope KMS

API Gateway & Auth

packages/veloctra-api

JWT Multi-Tenant Auth with 5-Role RBAC
Dynamic YAML Pipeline Compiler & Validator
High-Frequency WebSocket Broadcast Hub

FastAPI ASGI Pydantic v2 Bearer Auth

11-State Deterministic FSM

packages/veloctra-state

Deterministic State Machine (CREATED → COMPLETED)
Atomic Checkpointing & Offset Tracking
MongoDB (veloctra_system) & SQLite Store

Strict FSM Atomic Resume No Race Conditions

Double Envelope KMS

packages/veloctra-security

Layer 1: Fernet (AES-128-CBC + HMAC-SHA256)
Layer 2: ChaCha20-Poly1305 AEAD + Tenant AAD
Zero-Downtime KeyRotationManager (v1 → v2)

AEAD Crypto Keyring Versioning Zero-Plaintext

Non-Blocking Stream Dispatch & In-Memory Shared Buffer

3. Vectorized Data Plane & Execution Engine

Apache Arrow • Polars • MemoryGuard (75% Cap)

Stream Orchestrator

packages/veloctra-orchestrator

Intelligent MemoryGuard (75% RAM / CPU Ceiling)
Dynamic Chunk Sizing: 10,000 → 50 → 1 row
Circuit Breaker with AWS Full Jitter Backoff

Zero Memory Leak Adaptive Backpressure

Vectorized Transform Engine

packages/veloctra-transformers

PyArrow & Polars SIMD Columnar Transforms
Field-Level Column Cipher (AES-256-GCM)
WeakRef Plugin Registry & Sanitized Sandboxing

120,000+ rows/sec Zero-Copy Memory

Fault-Isolated DLQ Router

Dead Letter Queue Engine

Row-by-Row Fallback on Poison Pill batches
Corrupt records quarantined to DLQ sink
Non-blocking pipeline execution guarantee

Zero Data Loss Audit Trail

Parallelized Bulk Sinks (asyncpg Copy / Bulk Write / Parquet S3)

4. Universal Connectors & Storage Lakehouse Plane

SQL • NoSQL • Parquet • S3 / GCS

Relational SQL Drivers

PostgreSQL • MySQL • SQLite

High-speed asyncpg binary copy streams
Auto-partitioning cursor pagination queries
Transactional WAL mode with zero table lock

PostgreSQL MySQL SQLite

NoSQL & Document Stores

MongoDB • Cassandra • Redis

MongoDB Bulk Write unordered batches
Cassandra token-aware partition ranges
Redis Streams low-latency queue buffers

MongoDB Cassandra Redis

Lakehouse & Object Stores

Parquet • S3 • GCS • Partitioner

Snappy / Zstandard columnar Parquet writing
Auto-rotating FilePartitioner (Size / Row limits)
Direct streaming to AWS S3 & Google Cloud Storage

Apache Parquet AWS S3 GCS

📈 Benchmark Performance

Workload	Dataset Volume	Throughput Rate	Execution Time	Memory Footprint
CSV (Zip) → PostgreSQL	10,000,000 Rows	~28,500 rows/sec	~5.8 minutes	< 250 MB
PostgreSQL → CSV Lakehouse	1,620,000 Rows (431 MB)	~38,000 rows/sec	~42 seconds	< 180 MB
In-Memory PyArrow Vector Engine	1,000,000 Rows	~120,000 rows/sec	~8.3 seconds	< 120 MB