Headline

AI Product Manager - LLM & Agent Evaluation · RAG Quality for Production AI

About

I work on a simple, unforgiving problem: making a production AI system safe to trust when the cost of a wrong answer is a filing error, not a bad review. At Instead - an AI-native tax research and planning platform- I own LLM output quality end-to-end: the citation-accuracy evaluation loop, a four-dimensional rubric scored via LLM-as-judge with human spot-validation, competitive and public benchmarking competitive and public benchmarking (TaxCalcBench v2, across frontier models from four providers), and the 270K+-record retrieval corpus the answers stand on. The through-line: quality isn't a vibe, it's a measurement system — evals that catch regressions before clients do. I also build in the open. CrossSource (github.com/zoeb-nomi/crosssource) is my MIT-licensed eval harness for citation-level reliability of legal RAG: a human-authored golden set, a citation-failure taxonomy, and an LLM judge blind-validated at 100% human agreement — a validation pass that caught a harness bug the judge itself scored without noticing. The project's thesis proved itself on the project. Before AI: shipped a $2.7M MRR enterprise product zero-to-one at Keka HR and helped drive India's leading HR platform into the US market. Mechanical engineer by training; PM by conviction. Open to conversations about AI product roles where evaluation and reliability ARE the product. Write-ups and the rest of my eval work live at zoebnomi.com.

Location

India

AI stack

claude code

hugging face

mcp

n8n

Supabase

Wispr Flow

Work experience

Product Manager at Instead, Sep 2025 - Present · 1 yr, Bengaluru, Karnataka, India, I own output quality, evaluation, and reliability for Instead's AI tax-research engine — a production LLM/RAG system in a domain where a wrong answer is a filing error. - Built the evaluation layer that makes quality measurable: a flag → classify → weekly-review citation loop and a four-dimensional rubric scored via LLM-as-judge with human spot-validation — regressions get caught before release, not by clients. ~95% citation accuracy on the golden eval set. - Grounded the model roadmap in evidence: structured comparative experiments vs. three competing tax-AI platforms (50+ scenarios, 5 rounds), plus a "pipeline-collapse" eval testing whether the multi-step agent workflow stays coherent and self-honest end to end. - Pressure-tested the platform on the public TaxCalcBench v2 suite — 1,200+ agent runs across frontier models from four providers, permuting extended thinking, web search, and prompt style, with full cost/latency/tool-call telemetry — to find exactly where accuracy breaks; the evidence fed 7 shipped improvements targeting those failure modes. - Designing an internal 75-case benchmark (6 entity types × 3 complexity tiers × 3 jurisdictions), volume-weighted to real filing distribution — the quality bar is what clients actually file, not what public suites test. - Built the retrieval foundation the answers stand on: a 270K+-record corpus across 100+ legal source types, fed by a multi-stage agentic ingestion pipeline (browser agents → scraping → multi-model LLM enrichment) — retrieval quality caps answer quality. - Eliminated the most dangerous failure mode — plausible-but-wrong citations — by attaching document ID + sub-type to every cited source, making outputs traceable and audit-ready. - Led legal-support research across 160+ tax strategies and shipped Source Explorer — practitioners verify any claim in seconds.

Product Manager at Multiplier, Mar 2025 - Sep 2025 · 7 mos, Multiplier is a global employment platform enabling companies to hire and manage distributed teams compliantly across 150+ countries. I owned the Value-Added Services vertical — procurement, background verification, ITSM, and product partnerships — turning ancillary capabilities into a standalone revenue-generating product line. - Generated $100K incremental revenue and 47% operational efficiency gain by building SOPs for the VAS vertical from scratch — defining the operating model for a new product area - Launched PosterElite compliance partnership in a record 21 days — owned the full cycle from partner due diligence through technical integration and go-to-market - Built and shipped the BGV module across all employee types, expanding the platform's compliance capabilities for global workforce management - Drove product-market fit for VAS by identifying high-value partner integrations and optimizing adoption through operational workflow design

Associate Product Manager 2 at Keka HR, Jul 2024 - Feb 2025 · 8 mos, On-site, Keka is one of India's leading HR technology platforms, serving enterprise clients with a $25M+ ARR CoreHR product suite. Promoted to lead product strategy across SaaS App Management, Employee Onboarding & Offboarding, Document Management, Asset Management, and Integrations. - Built the Background Verification (BGV) module from 0→1 — integrated Checkr API, onboarded 28 US enterprise clients, and generated $2.7M MRR, driving Keka's first expansion into the US market - Led the Exit Module revamp: shipped 10+ features in a single quarter, reducing employee offboarding time from 7 to 5 days (~28% faster) and increasing CSAT by 80% - Executed go-to-market for enterprise features — product training and client webinars reaching 4,400+ attendees, generating 500+ upsell requests and a 30% feature-adoption increase - Owned the CoreHR product roadmap aligned to company OKRs, driving cross-functional collaboration across engineering, design, and customer success with 100% on-time delivery - Built NPS analysis framework and customer feedback loops — identified product improvements that increased client satisfaction by 15% and reduced churn

Associate Product Manager at Keka HR, Jul 2023 - Jul 2024 · 1 yr 1 mo, Hyderabad, Telangana, India, Owned product development for Employee Onboarding & Offboarding, Document Management, Asset Management, and Integrations modules within Keka's CoreHR platform. - Led the employee login revamp — shipped 6 SSO integrations, login history mapping, and Azure AD/Google Workspace integration, driving 27% adoption increase and $121K in upsell revenue - Reduced product development cycle time by 35% through process optimization, accelerating time-to-market for key features across multiple modules

Senior Product Analyst at Keka HR, Dec 2022 - Jun 2023 · 7 mos, Hyderabad, Telangana, India, Conducted market research, competitive analysis, and user research to inform product strategy across Keka's HRMS platform. - Led competitive analysis that shaped the asset management module roadmap — delivered employee asset requests and asset reports, increasing user engagement by 37% and driving 11% upsell growth - Created PRDs and user stories aligning cross-functional teams on feature development priorities - Partnered with data teams to introduce user engagement dashboards, improving visibility into product usage and decision-making quality

Business Analyst at HurixDigital, Sep 2021 - Sep 2022 · 1 yr 1 mo, HurixDigital is an enterprise content and learning technology company serving global EdTech and corporate learning markets. - Reduced delivery times by 43% by implementing agile processes for the enterprise Learning Management System, transforming the product development workflow - Drove 31% growth in customer conversions through end-to-end deployment of video learning solutions - Owned the 2022 LMS product roadmap, driving feature parity and performance optimization across engineering and design teams

Content Writer at Ghaas, Oct 2019 - Dec 2019 · 3 mos, Pune/Pimpri-Chinchwad Area, - Developed content strategies for clients in the talent and tech domains, increasing their web traffic by 15%. - Collaborated with cross-functional teams to build effective content marketing funnels and vision documents.

Education

General Management Program, Cohort 6, Stoa

Marathwada Institute of Technology Engineering College, Satara Road.

Public links

github: https://github.com/zoeb-nomi