🏛️ System Architecture Blueprint

The Veloctra Data Platform is engineered as an enterprise-grade, zero-memory-leak, high-throughput data streaming platform built on top of modular Python/FastAPI packages and React 18 frontend consoles.

1. Client & Management Experience Layer

React 18 • TypeScript • Tailwind

Visual Pipeline Studio

apps/management-ui

Visual DAG Builder with 1-Click Validation
Live YAML Editor with Schema Autocomplete
Interactive Connection Credential Vault

Monaco Editor Live Canvas No-Code / Low-Code

Real-Time Observability Center

WebSocket Live Feed

Sub-second Throughput Sparklines & Gauges
Live MemoryGuard RAM / CPU Pressure Telemetry
Instant Row Counter & Execution Heartbeat

WebSockets Dynamic Gauges Alerting

CLI & Headless Orchestrator

ctl.sh & Python SDK

Single-command platform lifecycle control
CI/CD GitOps pipeline triggers & imports
Docker Compose container auto-bootstrap

Bash CLI REST Client Docker Compose

REST API (FastAPI) & Bi-Directional WebSockets

2. Enterprise Control Plane & Security Engine

FastAPI • JWT RBAC • Double Envelope KMS

API Gateway & Auth

packages/veloctra-api

JWT Multi-Tenant Auth with 5-Role RBAC
Dynamic YAML Pipeline Compiler & Validator
High-Frequency WebSocket Broadcast Hub

FastAPI ASGI Pydantic v2 Bearer Auth

11-State Deterministic FSM

packages/veloctra-state

Deterministic State Machine (CREATED → COMPLETED)
Atomic Checkpointing & Offset Tracking
MongoDB (veloctra_system) & SQLite Store

Strict FSM Atomic Resume No Race Conditions

Double Envelope KMS

packages/veloctra-security

Layer 1: Fernet (AES-128-CBC + HMAC-SHA256)
Layer 2: ChaCha20-Poly1305 AEAD + Tenant AAD
Zero-Downtime KeyRotationManager (v1 → v2)

AEAD Crypto Keyring Versioning Zero-Plaintext

Non-Blocking Stream Dispatch & In-Memory Shared Buffer

3. Vectorized Data Plane & Execution Engine

Apache Arrow • Polars • MemoryGuard (75% Cap)

Stream Orchestrator

packages/veloctra-orchestrator

Intelligent MemoryGuard (75% RAM / CPU Ceiling)
Dynamic Chunk Sizing: 10,000 → 50 → 1 row
Circuit Breaker with AWS Full Jitter Backoff

Zero Memory Leak Adaptive Backpressure

Vectorized Transform Engine

packages/veloctra-transformers

PyArrow & Polars SIMD Columnar Transforms
Field-Level Column Cipher (AES-256-GCM)
WeakRef Plugin Registry & Sanitized Sandboxing

120,000+ rows/sec Zero-Copy Memory

Fault-Isolated DLQ Router

Dead Letter Queue Engine

Row-by-Row Fallback on Poison Pill batches
Corrupt records quarantined to DLQ sink
Non-blocking pipeline execution guarantee

Zero Data Loss Audit Trail

Parallelized Bulk Sinks (asyncpg Copy / Bulk Write / Parquet S3)

4. Universal Connectors & Storage Lakehouse Plane

SQL • NoSQL • Parquet • S3 / GCS

Relational SQL Drivers

PostgreSQL • MySQL • SQLite

High-speed asyncpg binary copy streams
Auto-partitioning cursor pagination queries
Transactional WAL mode with zero table lock

PostgreSQL MySQL SQLite

NoSQL & Document Stores

MongoDB • Cassandra • Redis

MongoDB Bulk Write unordered batches
Cassandra token-aware partition ranges
Redis Streams low-latency queue buffers

MongoDB Cassandra Redis

Lakehouse & Object Stores

Parquet • S3 • GCS • Partitioner

Snappy / Zstandard columnar Parquet writing
Auto-rotating FilePartitioner (Size / Row limits)
Direct streaming to AWS S3 & Google Cloud Storage

Apache Parquet AWS S3 GCS

📦 Monorepo Package Directory

The core platform is strictly divided into 8 isolated packages inside packages/:

Package	Location	Core Responsibility
`veloctra-core`	`packages/veloctra-core`	Settings management (Pydantic), logging formats, and core protocol schemas.
`veloctra-security`	`packages/veloctra-security`	Double Envelope Encryption (Fernet + ChaCha20-Poly1305), KeyRotationManager, and 5-role RBAC.
`veloctra-state`	`packages/veloctra-state`	Deterministic 11-State FSM, checkpoint engine, and MongoDB (`veloctra_system`) / SQLite store.
`veloctra-resilience`	`packages/veloctra-resilience`	Circuit Breaker automaton (CLOSED/OPEN/HALF_OPEN) and AWS Full Jitter retry backoff.
`veloctra-connectors`	`packages/veloctra-connectors`	Streaming drivers for PostgreSQL, MySQL, SQLite, MongoDB, Cassandra, Redis, and Universal File System.
`veloctra-transformers`	`packages/veloctra-transformers`	PyArrow & Polars vectorized transforms, AES-256-GCM cipher engine, and auto-sized `FilePartitioner`.
`veloctra-orchestrator`	`packages/veloctra-orchestrator`	Stream execution coordinator, `MemoryGuard` resource governor, and DLQ row-by-row fallback handler.
`veloctra-api`	`packages/veloctra-api`	FastAPI application gateway, JWT auth middleware, and sub-second WebSocket telemetry broadcaster.

🔄 Deterministic 11-State Finite State Machine (FSM) Workflow

Pipeline lifecycles are governed by a strictly validated state machine preventing race conditions, resource starvation, and inconsistent progress:

CREATED

Initialization

Pipeline definition registered, YAML configuration validated, and runtime job allocated.

VALIDATING

Connection Check

Credentials decrypted via KMS, source/sink handshakes verified, and schemas aligned.

EXTRACTING

Sharded Extraction

High-throughput cursor pagination and async connection pooling streaming batches.

TRANSFORMING

Vector Processing

PyArrow zero-copy column batching, field transforms, and AES-256 encryption.

Resilient Load

Bulk load into target database or Parquet partitions with Circuit Breaker guards.

CHECKPOINTING

Atomic Commit

State persistence to MongoDB/SQLite. Increments offsets, loops or marks COMPLETED.

Automated Self-Healing & Exception Recovery Routes

PAUSED (User Intervention): Operator pauses execution via UI; state engine preserves exact stream offsets without data loss. Resumes cleanly to EXTRACTING.

RETRYING (Network Glitch): Transient connection dropped; circuit breaker engages AWS Full Jitter backoff (100ms → 2s → 8s) and recovers to TRANSFORMING.

DLQ_ROUTED (Poison Pill Record): Corrupted row detected; orchestrator splits batch row-by-row, routing poison record to DLQ and saving all valid rows.

🔐 Double Envelope AEAD Encryption & Key Rotation

To eliminate credential leakage from static configuration files, database connections and API tokens are double-encrypted with zero-plaintext exposure:

Step 1 • Plaintext Ingestion

Raw Database Credentials / API Secrets

Input JSON secrets received via HTTPS from authenticated Studio UI or encrypted CI/CD pipeline payload.

▼

Step 2 • Layer 1 Encryption

Fernet Cipher Envelope (Master Key)

AES-128-CBC payload encryption combined with HMAC-SHA256 authenticated integrity signature using the platform Root Master Key.

▼

Step 3 • Layer 2 AEAD Encryption

ChaCha20-Poly1305 AEAD + Tenant AAD

Second-layer authenticated cryptographic envelope binding the payload to unique Tenant ID context headers (AAD) to prevent cross-tenant token replay attacks.

▼

Step 4 • Safe Persistence

Versioned Ciphertext Persisted to Database

Stored in format: enc:v1:<nonce_b64>:<ciphertext_b64>. Zero plaintext ever touches disk or logs.

Zero-Downtime Key Rotation: The KeyRotationManager maintains versioned keyrings. When keys are rotated to v2, existing tokens under v1 remain decryptable, while all new or updated credentials are encrypted under v2 seamlessly.

🛡️ In-Memory Field-Level Column Encryption (AES-256-GCM)

For sensitive PII, PHI, and financial attributes, Veloctra provides in-memory vector column encryption before writing batches to disk or cloud destinations:

Vector Column Ciphers

Operates directly on PyArrow chunks using SIMD instructions, encrypting column records without decompressing surrounding dataset rows.

AES-256-GCM Authenticated Cipher

Generates fresh 96-bit cryptographic nonces per cell, appending 128-bit authentication tags to prevent data tampering.

Declarative YAML Policies

Easily define encrypted fields via configuration: fields_to_encrypt: ["ssn", "claim_amount", "card_number"].

👥 5-Role Multi-Tenant Access Control (RBAC)

Control plane endpoints enforce tenant-bound Role-Based Access Control to ensure strict isolation:

Role	Allowed Operations	Authorization Scope
`SUPER_ADMIN`	Full administrative control, global key rotation, server telemetry, tenant provisioning.	Platform-wide
`TENANT_ADMIN`	Create pipelines, manage tenant secrets, invite users, configure alert webhooks.	Tenant-isolated
`DATA_ENGINEER`	Author and test pipelines, configure transforms, manage schemas, inspect DLQ.	Tenant-isolated
`OPERATOR`	Trigger pipeline runs, pause/resume execution, view live progress and logs.	Tenant-isolated
`AUDITOR`	Read-only inspection of audit events, FSM transitions, and compliance checkpoints.	Tenant-isolated

⚡ Universal Pluggable Streaming & Messaging Architecture

Veloctra is built around an open, extensible plugin model: streaming services are never hardcoded or limited to specific vendors. Through BaseStreamingConnector and the dynamic StreamingConnectorRegistry, users can plug in any streaming broker (Kafka, RabbitMQ, AWS SQS, Redis Streams, NATS, GCP Pub/Sub, Azure Event Hubs, MQTT, or custom internal queue systems):

Open Plugin Contract

Implement 4 simple async methods (connect, close, stream_read, publish_batch) to integrate any message broker without modifying platform core code.

Dynamic Discovery & Loading

Reference custom connectors in pipeline YAML via plugin_file: "plugins/custom_nats.py" or plugin_module: "my_org.solace" with runtime hot-loading.

Ultra-Lightweight Footprint (< 40MB RAM)

Core platform has zero heavy broker dependencies. Client libraries (e.g. aiokafka, redis, aiobotocore) load lazily on-demand, allowing Veloctra to run in lightweight micro-containers and edge pods.

🐍 Custom Script Transformation Engine (UI & CI/CD Import)

For complex business logic, risk modeling, and advanced feature engineering beyond standard schema mapping, users can execute custom Python scripts:

Inline UI Scripting

Define custom Python code directly in the pipeline designer with live dry-run validation via the /scripts/validate REST API.

CI/CD Module Import

Import external scripts, shared organizational packages, and Git repository modules seamlessly via script_path or module_name.

Multi-Framework Adapters

Author logic with your choice of framework: def transform(batch: pa.RecordBatch), def transform_df(df: pd.DataFrame), or def transform_polars(df: pl.DataFrame).

🔄 Universal Change Data Capture (CDC) Architecture & Engine

The Veloctra platform delivers an enterprise-grade Change Data Capture (CDC) framework that both implements its own vectorized delta/hash-diff engines in-app and integrates with external real-time log streams. It enables zero-data-loss synchronization, sub-second delta streaming, and legacy database replication without demanding heavy external agents such as Debezium or Kafka Connect.

🌊

1. High-Watermark Delta Sync

For tables with timestamp columns (e.g. updated_at, created_at) or monotonically increasing IDs. Injects dynamic SQL/NoSQL filter clauses at extraction time and updates checkpoints automatically.

Timestamp / Sequence Atomic StateStore Checkpoints

🔍

2. Checksum Hash-Diff CDC

Built for legacy databases with zero timestamp columns, no triggers, and no replication log access. Maintains persistent SHA-256 row-level hash states to detect INSERT, UPDATE, and DELETE events.

Zero-Column / No-Timestamp SHA-256 State Map

📜

3. Oplog Change Stream CDC

Real-time log-based CDC listening to database change feeds (e.g. MongoDB coll.watch() with resume_after tokens, Redis Streams, or Kafka log topics) converted to standardized PyArrow batches.

MongoDB Oplog Resume Tokens

⚡ CDC Stream Pipeline & PyArrow Vector Reconciliation

How change events are extracted, standardized into unified PyArrow vector batches, split via SIMD operations, and reconciled into target databases:

Phase 1 • Extraction & Delta Detection

Source Change Capture (Watermark, Checksum Hash-Diff, or Oplog Stream)

Extracts deltas using either dynamic WHERE watermark clauses (WHERE updated_at > 'last_wm'), SHA-256 hash comparison across non-key fields, or MongoDB change streams.

▼

Phase 2 • Standardized CDC Contract

Unified Vector Metadata Injection (_cdc_op, _cdc_ts, _cdc_key)

Every change batch is normalized into an Apache Arrow RecordBatch with operation tags: INSERT, UPDATE, or DELETE, alongside epoch timestamps and compound primary key values.

▼

Phase 3 • SIMD Vector Split & Target Reconciliation

Zero-Copy Filter & Dual-Channel Destination Dispatch

pyarrow.compute.equal(_cdc_op, "DELETE") splits the batch into:
• Deletes: Targeted SQLConnector.bulk_delete() using primary/match keys.
• Upserts: High-throughput bulk_upsert() (ON CONFLICT DO UPDATE) or MongoDB ReplaceOne(upsert=True).

📊 CDC Mode Capabilities & Trade-Offs

CDC Mode	Supported Databases	Schema Changes Required?	Deletion Handling	Throughput Profile
High-Watermark Delta Sync	PostgreSQL, MySQL, SQLite, MongoDB, Cassandra, DynamoDB	Requires timestamp or sequential integer column (e.g. `updated_at`, `id`)	Soft deletes only (e.g. `is_deleted = true`)	⚡ Ultra-Fast (120,000+ rows/sec)
Checksum Hash-Diff CDC	Any SQL / NoSQL / Flat File / CSV source without timestamps	Zero modifications. Works on legacy & read-only replicas	Full hard-delete capture (missing key detection)	⚡ High (50,000+ rows/sec with SHA-256 SIMD)
MongoDB Change Streams / Oplog	MongoDB Replica Sets / Atlas, Redis Streams, Kafka	None (Native database binlog / oplog stream)	Full hard & soft delete capture via change event tags	⚡ Real-time continuous streaming (< 50ms latency)

⚙️ Offset Governance & Conflict Policies (UI & API)

When updating an active CDC streaming pipeline, Veloctra enforces atomic checkpoint safety to avoid silent duplication or skipped records:

CDC Offset Update Strategies

replay_from_start: Clears stored high-watermarks and state hash tables, executing a fresh initial backfill before seamlessly transitioning into continuous delta mode.

process_new_only: Preserves existing watermark checkpoints, ensuring zero duplicate processing and picking up strictly newly generated changes from the exact cursor point.

🧪 Example: Hybrid High-Watermark CDC Pipeline YAML

# Veloctra CDC Streaming Configuration
pipeline_id: postgres_to_mongo_cdc
project_id: healthcare_prod_workspace
tenant_id: healthcare_prod_workspace
version: 2

settings:
  chunk_size: 5000
  max_memory_percent: 75.0
  dlq_enabled: true

sources:
  - name: pg_healthcare_claims
    type: database
    connection_string: "enc:v1:..."
    query: "SELECT * FROM raw_claims"
    chunk_size: 5000
    delta:
      watermark_column: updated_at
      watermark_type: timestamp
      initial_watermark: "2026-01-01T00:00:00"

destinations:
  - name: mongo_claims_collection
    type: nosql
    db_type: mongodb
    connection_string: "enc:v1:..."
    database: healthcare_dw
    collection: claims_stream
    upsert_key: ClaimId
    batch_size: 5000

⚡ 5. Intelligent Migration Sizing & KEDA Elastic Autoscaling

Veloctra combines in-pod micro-level protection with Kubernetes cluster macro-level elasticity through its integrated Migration Sizing Engine and KEDA (Kubernetes Event-Driven Autoscaling) controller:

🔍

1. Source Volume Discovery

Scans SQL table catalogs (pg_class.reltuples, SELECT COUNT(*)), MongoDB collection metadata (count_documents), and file metadata to calculate exact pending rows and payload size.

📊

2. Prometheus Metric Pipeline

Emits real-time gauges: veloctra_migration_workload_demand_replicas and veloctra_migration_pending_rows on /metrics for KEDA to scrape with zero broker dependencies.

🚀

3. Two-Tier Scale Up & Down

MemoryGuard prevents OOM crashes within < 1ms, while KEDA horizontally scales worker pods (1 → 16) and smoothly drains them upon completion with a 5-minute cooldown.

# Production KEDA ScaledObject Manifest (deploy/k8s/keda_scaledobject.yaml)
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: veloctra-engine-autoscaler
  namespace: veloctra
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: veloctra-engine
  pollingInterval: 15
  cooldownPeriod: 300
  minReplicaCount: 1
  maxReplicaCount: 16
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-k8s.monitoring.svc:9090
        metricName: veloctra_migration_workload_demand_replicas
        query: sum(veloctra_migration_workload_demand_replicas)
        threshold: "1"