Medical RAG Chatbot Platform
Two-system medical AI platform · LibreChat RAG chatbot (pgvector + LangChain) + LoRA fine-tuning app (vLLM serving) + Langfuse LLMOps
2025 – 2026
5
Docker Services
LoRA · vLLM
Fine-tune
Langfuse
LLMOps
6+
AI Providers
Problem
A healthcare client needed two complementary AI tools: (1) a trustworthy medical Q&A chatbot where every answer is grounded in curated clinical documents — not hallucinated from model weights — with full citation traceability; (2) a secure internal platform letting medical staff fine-tune smaller LLMs (Qwen3, Phi-3) on proprietary datasets (genetic counseling, Fukushima case studies) without exposing data to external APIs. Both systems had to run on-premise or Azure with strict security: no credentials in Git, no exposed ports, and observable pipelines that could be improved continuously.
Approach
Designed and deployed a 5-service RAG infrastructure using Docker Compose: LibreChat (Node.js/Express) as the chat frontend, a dedicated RAG API service (LangChain) for retrieval, MongoDB for conversation history, Meilisearch for full-text search, and PostgreSQL + pgvector for dense vector storage — all networked internally with no public port exposure.
Built a high-fidelity document ingestion pipeline: Docling extracts structured text from clinical PDFs → semantic chunking with overlap → OpenAI / local embedding model → pgvector storage → retrieved at query time via LangChain hybrid search (BM25 sparse + dense cosine) → cross-encoder reranking — every response includes cited source chunks.
Developed a separate LoRA fine-tuning web app (FastAPI + React + PostgreSQL): users upload medical CSV datasets (prompt/response pairs), configure quantization (4-bit / 8-bit via bitsandbytes), select base model (Qwen3-0.6B, Phi-3-mini), submit background training jobs, and immediately chat with the fine-tuned adapter served by vLLM — LoRA weights composed on-the-fly at inference.
Fine-tuned on domain-specific medical data: Fukushima genetic counseling dataset (English + Japanese), custom QA pairs from clinical documents — using PEFT LoRA with TRL (supports SFT and DPO) on single GPU with 4-bit quantization, reducing VRAM requirement to under 12 GB.
Implemented Langfuse-managed prompt engineering: system prompt (premedi-medical-assistant) and RAG context injection prompt (premedi-rag-context) are version-controlled and served from Langfuse — fetched at runtime via API, never hardcoded. Clinical teams can iterate, A/B test, and promote new prompt versions directly in the Langfuse UI with zero code changes and zero redeploys.
Instrumented full LLMOps observability with Langfuse: every LLM call automatically linked to the exact prompt version used, model endpoint, latency, token cost, and retrieval metadata — giving complete traceability from prompt commit to production response.
Integrated 6+ AI providers into a single chat interface: OpenAI, Anthropic Claude, Google Gemini, AWS Bedrock, Ollama (local), and vLLM (self-hosted fine-tuned models) — all switchable per conversation, with per-user token balance tracking.
Hardened deployment security: credentials injected exclusively via environment variables, never written to image layers or committed to Git; Nginx reverse proxy terminates TLS; all internal services communicate over a private Docker bridge network.
Deployed to Azure Linux VM via Docker Compose with Nginx TLS termination — delivering a production HTTPS endpoint; used GitHub for code reviews and CI, Jira for sprint planning and release tracking.
Results
Delivered two production AI systems — RAG chatbot and fine-tuning platform — both fully containerized and serving real clinical workflows.
RAG pipeline grounds every response in cited source documents, eliminating hallucination from model weights and enabling clinical teams to verify each answer against original PDFs.
LoRA fine-tuning pipeline adapts Qwen3-0.6B to genetic counseling domain on a single consumer GPU (4-bit quantization via bitsandbytes) — vLLM serves the fine-tuned adapter with production-grade throughput.
Externalized prompt management via Langfuse: clinical teams update system prompts (premedi-medical-assistant, premedi-rag-context) live in the Langfuse UI — no code change, no redeploy, full version history and per-call traceability.
Zero credential exposure: all secrets managed via environment variables, excluded from Docker image layers and Git history across both systems.
Architecture Overview
System 1 — RAG Chatbot: PDF
Docling
Chunking
pgvector (dense) + Meilisearch (BM25)
LangChain hybrid retrieval
Reranking
LibreChat LLM
Citation-grounded response
Langfuse traces | System 2 — Fine-tuning: CSV upload
FastAPI job queue
PyTorch + PEFT LoRA training (4-bit quant)
LoRA adapter saved
vLLM serves adapter on-demand | Infra: Docker Compose (5 services)
Nginx TLS
Azure Linux VM
How to Run
System 1 (RAG Chatbot): Clone llm-safechat. Copy .env.example → .env, fill API keys. Run: docker compose up -d (starts LibreChat, MongoDB, Meilisearch, pgvector, RAG API).Ingest documents: POST /api/rag/ingest with clinical PDF files. Docling parses → chunks → embeds → stored in pgvector.Access LibreChat at http://localhost:3080 — select any AI provider (OpenAI, Claude, local vLLM, etc.) and chat with document-grounded Q&A.System 2 (Fine-tuning): Clone llm-finetune-app. Requires NVIDIA GPU (16GB+ VRAM recommended). Run: docker compose up -d.Upload a CSV dataset (columns: prompt, response) via the React UI. Select base model + quantization (4-bit recommended for single GPU). Submit training job.Monitor job status in the dashboard. Once completed, click "Serve" — vLLM loads the LoRA adapter and exposes an OpenAI-compatible endpoint.Point LibreChat at the vLLM endpoint as a custom provider to chat with the fine-tuned medical model.View all LLM traces, latency, and cost in Langfuse dashboard (http://localhost:3000 when running locally).