🚀 Setup & Deployment Guide

Follow this step-by-step guide to install, configure, and launch the Veloctra Data Platform in your local environment or production Kubernetes / VM infrastructure.

1. System Prerequisites

Python: ≥ 3.10
Node.js: ≥ 18.0 (for the React 18 Management UI)
State Store: MongoDB ≥ 5.0 (Recommended for production) or SQLite (Default local fallback)

2. Monorepo Installation

# 1. Clone the repository
git clone git@github.com:vaagatech/veloctra-data-platform.git
cd veloctra-data-platform

# 2. Set up Python virtual environment
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# 3. Install packages in editable mode
pip install -e packages/veloctra-core \
            -e packages/veloctra-security \
            -e packages/veloctra-state \
            -e packages/veloctra-resilience \
            -e packages/veloctra-connectors \
            -e packages/veloctra-transformers \
            -e packages/veloctra-orchestrator \
            -e packages/veloctra-api

# 4. Build Management Console
cd apps/management-ui
npm install
npm run build
cd ../..

3. Environment Configuration (.env)

cp .env.example .env

# Configure environment keys:
PORT=8008
STATE_STORE_TYPE=mongodb
MONGO_DSN=mongodb://localhost:27017
MONGO_SYSTEM_DB=veloctra_system
SECRET_KEY=your_super_secret_jwt_key

4. Launching the Engine Daemon

# Start the background daemon
PORT=8008 ./start.sh

# Stop the daemon gracefully
./stop.sh

Once started, access the platform at:

Management Console: http://localhost:8008
Pipeline Studio: http://localhost:8008/studio
Observability Center: http://localhost:8008/observability
Interactive API Docs: http://localhost:8008/docs

5. Kubernetes & KEDA Elastic Autoscaler Deployment

# 1. Deploy base engine and Prometheus monitoring
kubectl apply -f deploy/k8s/deployment.yaml

# 2. Deploy KEDA ScaledObject for automated migration elasticity
kubectl apply -f deploy/k8s/keda_scaledobject.yaml

# 3. Optional: Deploy ephemeral batch shard executor ScaledJob
kubectl apply -f deploy/k8s/keda_scaledjob.yaml