Architecture & Execution Model

Architecture Overview

VaakLoom separates the core execution engine from protocol integrations and the control plane:

┌─────────────────────────────────────────────────────────┐
│  Hub Control Plane (apps/hub)                           │
│  Builder · Traces · Metrics · RBAC · Themes · Deploy UI │
└──────────────────────────┬──────────────────────────────┘
                           │ DAG JSON & Traces
┌──────────────────────────▼──────────────────────────────┐
│  Engine (@vaakloom/engine / vaakloom Python)            │
│  Context · Parallel/Batch · Cache Iface · Telemetry     │
└───────┬─────────────┬──────────────┬────────────────────┘
        │             │              │
   adapter-*     compute-*         etl
   (REST,gRPC…)  (PDF,zip,media)   (warehouse)
        │             │
   App drivers    Worker queue

Execution Model & Hardening Guardrails

1. DAG Construction: Workflows are defined via the Fluent DSL or loaded from the Hub JSON export. The spec is statically validated for DAG acyclicity, step dependencies, and JSON schemas before registration.
2. Resource Governor (75% Limit & Headroom): Prior to and during execution, the ResourceGovernor constantly monitors system and process CPU/memory usage. Usage is strictly regulated under 75%, preserving 25% of compute capacity for garbage collection and host OS responsiveness. Proactive GC is triggered at 70% memory load.
3. Adaptive Chunk Sizing: Batch items are dynamically sampled for payload byte weight. Heavy payloads (> 500 KB) automatically drop chunk sizes down to 1–2 to prevent out-of-memory crashes, while lightweight records scale up to 100 items per chunk.
4. Failure Isolation & Dead-Letter Queue (DLQ): If an individual record fails in a batch or stream, the engine isolates the error, logs the payload and diagnostic trace to the DeadLetterLedger, and proceeds with the remaining records without failing the workflow.
5. Zero-Missed Observability Event Stream: 100% of engine lifecycle events, transitions, and failures are recorded in the EventHub ring buffer and streamed via Server-Sent Events (SSE) to connected Hub dashboards.
6. Zero-Leak Memory Post-Processing: After workflow completion, onPostProcess persists telemetry asynchronously into SQLite/Prometheus, and context buffers are immediately pruned to prevent memory bloat under high concurrency.
7. Context Spill Safety: When payloads exceed contextSpillBytes, large attributes are spilled to disk via ContextStore and replaced with light memory references.

Engine vs. Platform Separation

Component	Package	Responsibilities
Engine	@vaakloom/engine	Pure execution runtime, DAG validation, in-memory cache interface, fluent DSL, and telemetry hooks. Zero database or heavy runtime dependencies.
Platform	@vaakloom/platform	Full product distribution: embeds Engine, SQLite persistence, Hub Control Plane REST APIs (/hub/*), Prometheus exporter (/metrics), and RBAC authorization.
Hub UI	apps/hub	Single-page React control plane for visual workflow editing, live trace stream inspection, Snapline test execution, and RBAC administration.

Non-Goals of the Engine Core

To guarantee minimal footprint, ultra-low latency, and portability:

• No bundled heavy protocol drivers: Drivers (e.g. pg, @grpc/grpc-js, aws-sdk) remain in application userland.
• No default distributed cache runtime: Engine defaults to in-memory caching for local development, allowing seamless injection of Redis/Memcached in production.
• No blocking in-process compute: CPU-heavy workloads (video encoding, PDF manipulation, archive compression) are offloaded to asynchronous worker queues.

Monorepo Layout

packages/
  engine/         @vaakloom/engine       Core DAG + fluent DSL + Application
  platform/       @vaakloom/platform     Unified server (Hub API + SQLite)
  rbac/           @vaakloom/rbac         Role-based access control engine
  metrics/        @vaakloom/metrics      Prometheus registry & collectors
  snapline/       @vaakloom/snapline     Snapshot regression test suite
  deploy/         @vaakloom/deploy       Deploy bundling & plan exporter
  cli/            vaakloom               CLI binary (serve, test, validate)
  adapter-*       Protocol adapters      REST (runnable), GraphQL, gRPC, etc.
  compute-*       Compute offloaders     PDF, Zip, Media transcoding
apps/
  hub/            Visual control plane   Vite + React UI for operators
  demo-ts/        Golden-path demo       E2E sample app and smoke runner
python/
  vaakloom/       Python asyncio engine  Asyncio DAG runtime with Python parity