Capstone Award · Ranked #1 (Grade 9.8 / 10)·Jan 2024 — Jul 2024·Team Lead (6 Engineers)·v2: ReAct Agent Extension

CareerCompass AI

Architecting an End-to-End AI Recruitment Platform with Change Data Capture (CDC), Kafka, Vector Search, and RAG

FastAPIPostgreSQLApache KafkaDebezium (CDC)QdrantWeaviateElasticsearchOpenAI APIReAct AgentSSE StreamingMinIODocker

Role

Tech Lead / Architect

Evaluation

9.8 / 10 (Rank #1)

Precision

95% Retrieval Accuracy

Latency

< 150ms Search

1. Problem Statement

Traditional recruitment engines rely heavily on static SQL queries and keyword-based filters. When a job description asks for a "Distributed Systems Engineer familiar with Kafka," a qualified candidate who describes their experience as "architecting event-driven pipelines using publish-subscribe queues" is often completely omitted from initial search passes.

Furthermore, recruiters spend countless hours manually cross-referencing resumes against dense requirements. From a systems standpoint, performing real-time vector embedding generation and similarity calculations inside synchronous HTTP request-response cycles causes severe latency spikes and risks cascading timeouts.

2. System Architecture & Data Pipeline

To decouple transactional writes from heavy indexing workloads, I designed an event-driven Change Data Capture (CDC) architecture:

[ Client Web App ] ─── HTTP REST / JSON ───► [ FastAPI Core API Services ]
                                                        │
                                                        ▼ (ACID Transactional Writes)
                                              [ PostgreSQL Database ]
                                                        │
                                                        ▼ (Write-Ahead Log / WAL)
                                              [ Debezium CDC Connector ]
                                                        │
                                                        ▼ (Event Stream: inserts / updates)
                                              [ Apache Kafka Clusters ]
                                                        │
                                                        ▼ (Asynchronous Consumer Groups)
                                              [ Background Workers ]
                                               ├── OpenAI Embedding Pipeline
                                               └── Batch Vector Normalization
                                                        │
                                    ┌───────────────────┼───────────────────┐
                                    ▼                   ▼                   ▼
                           [ Qdrant Vector DB ]   [ Weaviate Engine ]   [ Elasticsearch ]
                           (Dense Embeddings)     (Hybrid Schemas)     (Keyword / BM25)
                                    │                   │                   │
                                    └───────────────────┴───────────────────┘
                                                        │
                                                        ▼
                                       [ Recruiter Matching & RAG Engine ]
                                                        │
                                                        ▼
                                            [ OpenAI Assistants API ]
                                                        │
                                                        ▼ (Charts & Metrics Artifacts)
                                              [ MinIO Object Storage ]

CareerCompass AI — System Architecture Diagram

Microservices, CDC Streaming, Kafka Event Bus, and Vector RAG Pipeline

View Full Resolution ↗
CareerCompass AI End-to-End System Design and Architecture Diagram

Write Path Decoupling: When an applicant updates their profile or a recruiter posts a new role, the write commits instantaneously to PostgreSQL without stalling for vector embeddings.

Debezium CDC: Debezium tails PostgreSQL's Write-Ahead Log (WAL), streaming atomic event envelopes into topic partitions in Kafka. This guarantees that no indexing events are lost, even in the event of worker crashes.

Asynchronous Indexing: Dedicated worker pools consume Kafka events, invoke the OpenAI Embedding model with exponential backoff and rate-limiting, and upsert vectors into Qdrant and Weaviate.

3. Key Engineering Highlights

1. RAG & LLM Query Expansion

Queries are expanded via lightweight LLM prompts to extract domain synonyms, required competencies, and seniority levels. The resulting dense query vectors achieved 95% retrieval precision over raw lexical search.

2. Real-Time CDC Streaming

Eliminated manual dual-write patterns and two-phase commits. By reading directly from PostgreSQL WAL logs, the system guarantees zero dual-write inconsistencies between the relational store and vector databases.

3. Recruiter Semantic Matcher

Built a multi-criteria vector scoring engine that computes cosine similarity between candidate skill vectors and job requirements, returning rank-ordered applicant shortlists in sub-150ms.

4. AI Analytics & MinIO Storage

Integrated the OpenAI Assistants API with custom code-interpreter capabilities to generate dynamic visualizations and conversion metrics, persisting generated chart artifacts safely to a private MinIO S3 cluster.

4. Engineering Challenges & Solutions

Challenge 1: Event Ordering & Race Conditions During Profile Updates

Rapid successive edits to candidate profiles caused race conditions where older updates could overwrite newer vectors if processed out of order across worker threads.

Solution: Partitioned Kafka topics by user_id, ensuring that all state mutations for a specific user were routed to the same partition and consumed strictly in FIFO order.

Challenge 2: OpenAI API Rate Limits & Cost Control

High-volume resume ingestion quickly triggered HTTP 429 rate limits and threatened to incur excessive API costs.

Solution: Built an in-memory Redis embedding cache keyed by SHA-256 text hashes, deduplicating identical job descriptions and skill summaries. Implemented a token-bucket rate limiter within the background worker service.

Challenge 3: Hybrid Search Balance (Keywords vs Semantics)

Pure vector search occasionally omitted strict requirements (e.g., exact visa status or mandatory security clearance).

Solution: Adopted a two-stage hybrid retrieval strategy: first filtering candidates using Elasticsearch BM25 / Boolean facets, then re-ranking the top candidate pool with Qdrant vector similarity scores.

v2 Platform Extension

5. Extension: Autonomous Career Intelligence Agent

Following the release of the core recruitment engine, we extended the platform with a secondary intelligent service: an autonomous Career Intelligence Agent. This feature transforms passive job matching into proactive, multi-step career guidance with iterative skill-gap roadmapping.

Architecture & Delivery Summary

“Architected a Career Intelligence Agent using OpenAI Function Calling and a ReAct loop across 6 tools (profile extraction, vector search, skill gaps, roadmap, self-evaluation); decoupled execution via Kafka and engineered real-time SSE streaming backed by Redis Pub/Sub with an event-replay buffer to eliminate race conditions.”

Interactive Architecture & Execution Flow

Career Intelligence Agent (ReAct Loop + SSE Streaming)

Multi-step LLM reasoning can take 15–40 seconds. To prevent HTTP connection timeouts and race conditions, we decouple dispatch from inference via Kafka and stream live progress back to the client using Server-Sent Events (SSE) backed by a Redis Pub/Sub engine with an Event-Replay Buffer.

Step 01< 45ms

1. Non-Blocking Ingestion

Client issues POST /api/v2/agent/career/async.

Returns { task_id: "ccp-89a" } immediately, preventing client timeouts.
Step 02Kafka

2. Decoupled Event Bus

Event is published to Kafka topic ccp_agent_task partitioned by candidate_id.

Guarantees FIFO execution order per user, absorbs burst traffic, and shields inference from spikes.
Step 0315–30s

3. Autonomous ReAct Execution

ccp-background worker executes the multi-turn ReAct loop across 6 tools.

Pushes live steps simultaneously to Redis Pub/Sub channel + Redis Replay Buffer list.
Step 04SSE Stream

4. Zero-Loss Event Replay

Client connects via EventSource to stream progress.

Replay Buffer: Flushes missed steps if client connects late, eliminating frontend race conditions.
Data & Streaming Pipeline Topology
[ Client Browser ] ────── POST /api/v2/agent/career/async ──────► [ FastAPI AI Gateway ]
        │                                                                  │
        │ (Receives task_id <45ms)                                         ▼ (Enqueues Task Event)
        │                                                     [ Kafka: ccp_agent_task ]
        │                                                                  │
        │                                                                  ▼ (Pulls Message)
        │                                                     [ ccp-background Worker ]
        │                                                     (AgentTaskRunner Process)
        │                                                                  │
        │                                                                  ▼ (Executes ReAct Loop)
        │                                                     [ CareerAgent (ccp-ai) ]
        │                                                        ├── Tool Calls (6 Tools)
        │                                                        └── Self-Evaluation
        │                                                                  │
        │                                                                  ▼ (Publishes Events)
        │                                                ┌─────────────────┴─────────────────┐
        │                                                ▼                                   ▼
        │                                     [ Redis Pub/Sub Channel ]           [ Redis Replay Buffer ]
        │                                     (Live Real-Time Streaming)          (RPUSH agent:events:{id})
        │                                                │                                   │
        │                                                └─────────────────┬─────────────────┘
        │                                                                  │
        ▼ ◄───────── GET /api/v2/agent/stream/{id} (SSE) ──────────────────┘
  [ Browser EventSource ]
  1. Replays buffered past steps (Zero Race Conditions!)
  2. Subscribes to live Pub/Sub stream until task completion

A. The 6-Tool ReAct Loop (Dynamic Reasoning vs. Rigid Chains)

Instead of hardcoding a brittle sequential chain, the agent acts as an autonomous controller powered by OpenAI Function Calling. In each iteration, the LLM emits a reasoning Thought, invokes one of 6 specialized tools (extract_candidate_profile, search_matching_jobs, get_market_skill_requirements, calculate_skill_gap, generate_learning_roadmap, and evaluate_recommendation), observes intermediate outputs, and iteratively synthesizes the roadmap.

B. Decoupled Asynchronous Execution via Apache Kafka

Multi-turn agent inference typically spans 15–40 seconds. To prevent HTTP connection timeouts (504s), the API gateway immediately responds with 202 Accepted and an opaque task_id in <45ms, enqueuing the job onto Kafka topic ccp_agent_task for consumption by background worker pools.

C. Real-Time SSE Streaming with Redis Pub/Sub & Event-Replay Buffer

To keep users engaged during reasoning cycles, events are streamed live to the client via Server-Sent Events (SSE). To eliminate race conditions where the browser connects after initial worker events have fired, events are dual-written to a live Redis Pub/Sub channel and an in-memory Redis List buffer (RPUSH agent:history:{id}). On client connection, the buffer is replayed first before subscribing to live updates, guaranteeing zero lost progress steps.

6. Results & Key Learnings

The capstone project was presented to faculty and industry reviewers, receiving a score of 9.8 / 10 and ranking #1 across all graduate capstones.

Leadership Takeaways: Serving as the technical lead for 6 engineers reinforced the value of strict interface contracts and early schema governance. Setting up Kafka topic definitions and Protobuf / Pydantic schemas upfront allowed the frontend, backend, and data pipelines to be developed concurrently without blocking dependencies.

Extending to Autonomous Workflows: Augmenting the established data foundation with the v2 Career Intelligence Agent demonstrated the scalability of the event-driven architecture: new agentic capabilities could be plugged in through dedicated Kafka topics and worker services without touching or destabilizing existing transactional pipelines.

Project Repositories

Explore the multi-repository architecture on GitHub: