Adapters, Compute & ETL

Protocol Adapters

VaakLoom adapters provide standardized interfaces to integrate external data sources and transport protocols into DAG steps:

Universal Resilient Retries & Custom Logic

All 8 protocol adapters and engine workflow steps feature unified, enterprise-grade retry mechanics. Retry rules can be configured declaratively or programmatically using custom predicates to handle transient errors, network anomalies, rate limits, and database deadlocks:

// Example: Custom programmatic retry predicate in @vaakloom/adapter-rest
import { restStep } from "@vaakloom/adapter-rest";

const step = restStep({
  url: "https://api.partner.example.com/v1/orders",
  method: "POST",
  retry: {
    maxAttempts: 4,
    delayMs: 250,
    backoff: "exponential",
    jitter: true,
    statusCodes: [429, 502, 503, 504],
    // Custom logic: inspect response body or custom business error codes
    shouldRetry: async (ctx) => {
      // Retry if downstream rate limit or temporary lock condition
      if (ctx.statusCode === 429) return true;
      if (ctx.result?.data?.lockAcquisitionFailed) return true;
      if (ctx.error?.message?.includes("ECONNRESET")) return true;
      return false; // Do not retry fatal 4xx business validation errors
    },
    onRetry: (ctx) => {
      console.warn(`[Retry #${ctx.attempt}] Pausing for ${ctx.delayMs}ms before retrying request`);
    }
  }
});

Retry configuration is also supported directly in declarative workflow definitions via when expressions or explicit retry policies:

// Example: SQL Adapter with Postgres Deadlock (40P01) Retry
import { sqlStep } from "@vaakloom/adapter-sql";

const transferFunds = sqlStep({
  client: pgPool,
  query: "UPDATE accounts SET balance = balance - $1 WHERE id = $2",
  retry: {
    maxAttempts: 5,
    delayMs: 50,
    backoff: "exponential",
    errorCodes: ["40P01", "40001"], // PostgreSQL deadlock_detected & serialization_failure
    shouldRetry: (ctx) => ctx.errorCode === "40P01" || ctx.errorCode === "40001",
  }
});

Compute Offload Workers

To preserve low latency on the main event loop, compute-heavy tasks are isolated into modular worker packages:

ETL Workflows

The @vaakloom/etl package enables building Extract → Transform → Load pipelines using the exact same declarative DAG engine. Stream records from databases, transform rows concurrently, and batch-load into data warehouses without provisioning complex Spark or Airflow clusters for mid-sized workloads.