Features & Capabilities

Hybrid Search Engine & Semantic Highlights

Anvesh combines BM25 full-text scoring with dense vector similarity search using Reciprocal Rank Fusion (RRF). A single query returns results that are both lexically and semantically relevant:

• BM25 Full-Text — inverted index with configurable tokenizers, stemming, and stop-word filters
• Dense Vector (HNSW & SQ8) — approximate nearest neighbor with 75% vector memory compression
• Hybrid Mode — fuse keyword + semantic signals via RRF or linear score blending
• Bounded Highlighting — strict 240-character snippet truncation at sentence boundaries with semantic synonym expansion

Field Projections & Granular Selection

Optimize payload sizes and network bandwidth by retrieving only the exact fields your client application requires:

• Inclusion & Exclusion Objects — select fields with { title: 1, price: 1 } or exclude large blobs with { body: 0 }
• Nested Paths & Dot Notation — cherry-pick nested sub-documents (e.g. "specs.dimensions.weight": 1) without shape distortion
• Document ID Control — retain or suppress document identifiers with { id: 0 }
• REST & Query Shortcuts — pass concise field arrays (["title", "price"]) or URL query parameters (?select=title,price)

Automated Spider Scheduler & "Run Now" Crawling

Keep web-crawled indexes fresh with autonomous background scheduling and on-demand execution:

• 5-Part Cron & Human Intervals — define recurring schedules with standard cron (e.g. 0 2 * * *) or shorthand intervals (15m, 1h, 1d)
• On-Demand "Run Now" Triggers — manually trigger immediate crawl jobs from the REST API or Hub UI without waiting for timer expiration
• Multi-Role Post-Login Discovery — authenticates as guest, user, or admin to discover and tag role-gated routes
• Direct Stream Auto-Indexing — streams crawled pages directly into Anvesh Engine with automated dense vector embedding generation

Multi-Source Change Scanner (CDC)

Capture incremental changes across local/network folders, streaming data files, and external relational databases:

• Folder Scanner — recursive directory traversal with SHA-256 fingerprinting to detect created, modified, and deleted files
• File Stream Ingestion — line-by-line streaming ingestion of high-volume NDJSON/JSONL and CSV files with batch auto-indexing
• Database CDC — polling-based watermark tracking across SQLite, PostgreSQL, MySQL, and generic SQL databases using timestamp columns
• Dedicated Daemon & CLI — run as a background microservice (port 3853) or trigger one-off scans via the anvesh-scanner CLI

Pure Distributed State Store

Engineered for high-availability Kubernetes multi-pod clusters with zero pod-local database dependencies:

• Native System Indexes — schedules, run logs, and CDC watermarks are replicated across the search engine cluster (.anvesh_*)
• Atomic Shared DFS Storage — atomic POSIX file operations safe against OS filename limits for shared PVC storage
• Graceful Hub Degradation — the Hub UI and API gracefully detect worker offline states and display informational cards without blocking other operations

Non-AI Visual & OCR Feature Extractor

Convert images, product photos, saree weaves, and scanned PDFs into searchable text without expensive external AI APIs:

• Pure-CPU Local OCR — embedded tesseract.js worker recognizes printed words, tags, and product serials
• Textile Color Palette Analyzer — matches pixel histograms to a curated dictionary ("Gold Zari", "Royal Blue", "Maroon", "Emerald Green")
• Motif & Texture Edge Descriptor — uses gradient frequency and edge density to classify patterns ("Elephant Motif", "Temple Border", "Kattam Checks")

Config-as-Code & GitOps Automation

Manage all search infrastructure through code and version control:

• Declarative Plan & Apply — run anvesh plan and anvesh apply to evolve schemas without pod restarts
• Terraform Provider — provision and manage indexes, crawlers, and circuit breakers in .tf files
• Official Client SDK — fully typed TypeScript/JavaScript client (@vaagatech/anvesh-sdk) with automated Cognito M2M token rotation

3-Tier Protective Throttling & Circuit Breakers

Built-in defenses to prevent denial-of-service, CPU spin, and out-of-memory crashes:

• Tier 1: Rate Limiting — per-token/IP rate limits (default 120 req/min, configurable via ANVESH_RATE_LIMIT)
• Tier 2: In-Flight Slot Limits — caps simultaneous executing searches (default 32 concurrent, configurable via ANVESH_MAX_CONCURRENT_SEARCH) with instant 429 ERR_CIRCUIT_CONCURRENT shedding
• Tier 3: Memory Backpressure — proactive GC at 75% heap; automatic load-shedding at 85% heap to prevent OOM restarts

Tiered Object Storage

Anvesh automatically tiers data between fast in-memory segments and cheap cloud storage:

• Hot tier — in-memory and local SSD for frequently accessed segments
• Warm/Cold tier — S3, OCI Object Storage, or MinIO for immutable historical segments
• Cost impact — reduces cluster operational costs by up to 85% compared to monolithic search nodes

Dead-Letter Queue (DLQ)

Every failed write or corrupted document is captured in a zero-drop Dead-Letter Queue:

• Automatic retry with exponential backoff
• Manual inspection and one-click replay via the Hub UI
• Zero-document-loss guarantee