Agentflow-AI

Problem Statement

Enterprise operations teams waste time toggling between static documentation and internal transactional tools. Existing chatbots either hallucinate policy answers or lack the ability to safely execute multi-step operational actions.

Dataset

26+ enterprise policy markdown and PDF documents ingested via a dual PyMuPDF + PyPDF pipeline using 300–500 token chunking with ~15% overlap, combined with transactional order and support ticket schemas.

Architecture

LangGraph pattern-driven state machine with Planner, Retriever, Tool Executor (FastMCP async client), Responder, and Clarify nodes. Coupled with a FastAPI SSE streaming backend, Redis query-hash caching (10m TTL), MongoDB Atlas audit logging, and a Next.js 14 chat interface with collapsible step-trace transparency.

Model Selection

LLM planner dynamically routes queries into retrieve_only, tool_only, retrieve_and_tool, or answer_directly via structured JSON reasoning. A calibrated 0.65 cosine similarity gate routes low-confidence retrieval to the Clarify node.

Training Process

Strict human-in-the-loop confirmation checkpoints for state-altering actions (e.g., ticket creation) and resilient JSON extraction prompts to eliminate fragile keyword routing.

Evaluation Metrics

Results

An enterprise copilot capable of resolving complex multi-step queries (e.g. policy checks combined with order lookups), citing authoritative documentation, executing MCP tools safely, and rendering full step-level audit trails.

Key Learnings

• 1A calibrated relevance threshold (0.65) is crucial for enterprise agents to gracefully decline queries rather than hallucinating answers.
• 2Separating tool execution behind the Model Context Protocol (MCP) decouples the agent core from specific operational APIs.
• 3Human-in-the-loop confirmation dialogs for destructive actions provide enterprise security without degrading user experience.