Headline

AI Engineer - Data Platform at Amazon Music | Generative AI & Streaming Architect | 10 YOE Distributed Systems | Scala & Python

About

I am a Software Development Engineer specializing in enterprise data platform architecture, petabyte-scale streaming, and Generative AI orchestration. Over the past decade, I have engineered data systems that eliminate operational friction, accelerate developer velocity, and deliver significant structural cost reductions—including saving over $1M/year on a single streaming architecture. My core architecture focus areas include: * Intelligent Analytics Platforms: Architecting multi-agent GenAI models (Supervisor-Sub-agent), natural language data discovery engines, and automated SQL self-correction loops using LLMs. * Petabyte-Scale Streaming Platform Engineering: Designing serverless, high-throughput ingestion pipelines capable of processing 17+ billion daily events with low event-time latency. * Enterprise Data Lake Governance: Building multi-tenant storage frameworks that decouple compute, automate data lifecycle compliance, and centralize access across globally distributed teams. Core Stack: AWS (EMR, Glue, Kinesis, Bedrock, Lambda, DynamoDB, EventBridge, CDK), Apache Spark, Scala, Python, Java, Redis/Redisson, SQL.

Location

Bengaluru

AI stack

claude code

kiro

langchain

langgraph

mcp

CrewAI

Work experience

Software Engineer at Amazon, Aug 2021 - Present · 5 yrs 2 mos, Architected Multi-Agent GenAI Platform: Deployed a Supervisor-Sub-agent LLM model using AWS Bedrock and S3 Knowledge Bases, enabling natural language data querying. Engineered Self-Correction Logic: Developed a multi-stage validation engine executing syntax checks and dry-runs on LLM SQL with up to 3 auto-regeneration attempts. Impact: Eliminated SQL bottlenecks for business teams, reducing organizational time-to-insight from hours to seconds. Petabyte-Scale Streaming Platform: Built a fully serverless near real-time streaming platform (Kinesis → Lambda → ElastiCache → Firehose → S3) handling 17+ billion daily events, slashing data latency from 24 hours to 15 minutes while maintaining 99.9% uptime. High-Performance Cache Cluster: Designed a cluster-mode ElastiCache Redis layer (30 nodes, ~17B keys) utilizing Redisson’s batch interface to execute event deduplications in a single network call. Impact: Delivered over $1M in annual cost savings (82% infrastructure cost reduction) while accelerating data availability 288x faster. Serverless Transformation Engine: Built a 100% serverless, event-driven decoupled architecture using Lambda and SQS to handle complex GenAI code conversions across legacy data warehouses. Parity & Validation Automation: Developed an LLM-based SQL translation and validation loop alongside a zero-trust verification pipeline using AWS Glue to mathematically prove GenAI code matches legacy output. Impact & Mentorship: Replaced manual dependency hunting with automated data lineage tracking via metadata catalogs, targeting an 84% boost in developer productivity. Self-Serve Data Lake Architecture: Designed and built a centrally governed S3-based data lake platform that decoupled compute from storage, serving as the storage engine for Redshift and Spectrum. Implemented Hybrid Access Control Models: Deployed a hybrid IAM and AWS Lake Formation security layer to provide granular access controls supporting heterogeneous compute clients.

Sr Engineer at Target, Feb 2020 - Sep 2021 · 1 yr 8 mos, India, Engineered highly scalable data pipelines using Hive, PySpark, Scala, and Oozie to manage a massive 40x data influx driven by the nationwide rollout of the pilot loyalty program. Spearheaded performance tuning and optimization initiatives for legacy and long-running workflows, successfully slashing a 6-hour processing window down to 1.2 hours (an 80% efficiency gain). Established streamlined DevOps practices by designing and implementing reliable automated CI/CD deployment pipelines using Jenkins. Served as a key technical liaison during a 6-month onsite deployment in Minneapolis, MN, USA, collaborating directly with business product owners to align pipeline architecture with corporate strategy. Executed legacy ETL migrations into modern Hadoop frameworks while maintaining strict adherence to Agile methodologies utilizing Jira, Git, and Confluence.

BI Engineer at Target, Jun 2016 - Feb 2020 · 3 yrs 9 mos

Education

Bachelor of Engineering, Computer Engineering, B. M. S. College of Engineering

Diploma, Computer Science, s j (Govt) polytechnic