Description
This role will help ensure AI-powered customer experiences are accurate, reliable, measurable, and continuously improving. The analyst will monitor production LLM application performance, evaluate response quality, build dashboards, identify risks and opportunities, and synthesize insights that inform product roadmap decisions, feature requirements, backlog prioritization, and operational improvements.
Supports our AI products by analyzing performance, defining KPIs, and driving continuous improvement for generative and conversational AI applications. Monitors LLM outputs, evaluates response quality, and builds dashboards/scorecards to track product health, user behavior, and business impact. Identifies risks (hallucinations, drift, retrieval issues) and translates insights into actionable product recommendations, backlog priorities, and experiments.
The ideal candidate combines strong analytical skills, product sense, technical curiosity, and practical understandi...