Headline

Data Engineer 2 at Microsoft | Apache Spark | Azure Synapse | Akka | Kafka | ETL Pipelines | Data Platform | AdTech · FinTech · Privacy Platforms | 5+ Years Building Production Data Pipelines and Platforms

About

I'm a Senior Data Engineer with 5+ years of experience building and owning production-grade data infrastructure — from streaming pipelines and large-scale batch processing to cloud-native data platforms in regulated environments. Currently at Microsoft's Security org (Next Generation Privacy team), I'm responsible for end-to-end infrastructure setup in a sovereign / highly regulated cloud — designed from scratch using Infrastructure as Code (Bicep, ARM templates). My work spans secure data processing architectures, analytics and storage services, identity-first access control (managed identities, RBAC), governance and audit integration, and full production readiness including incident mitigation and environment hardening. I deploy and validate production data pipelines — covering assets, signals, and audit flows — that enable downstream business and compliance workflows. At Media.net (AdTech, 500K+ publisher network), I owned Apache Spark data pipelines powering ML teams and in-house reporting dashboards, operating over HDFS/Hive with Apache Druid for real-time analytics. At HashedIn by Deloitte (FinTech domain), I built and maintained Akka Streaming pipelines processing customer orders and payment reconciliation across heterogeneous data sources, using Akka Cluster Sharding, Kafka, and CosmosDB. I also mentored ~15 engineers on Scala, Akka, and Play frameworks. Technical strengths: Apache Spark, Akka & Akka Streams, Scala, Kafka, Python, HDFS, Hive, MySQL, Azure Synapse Analytics, Azure Databricks, Airflow, Bicep/ARM IaC, ETL/ELT pipelines, dimensional modeling, data warehousing, RBAC, and sovereign cloud infrastructure. Actively exploring Senior Data Engineer roles at product-first, data-driven organizations. Open to hybrid/remote. Let's talk.

Location

Bengaluru

AI stack

claude code

gemini

github copilot

mcp

Work experience

Software Engineer 2 at Microsoft, Jun 2026 - Present · 3 mos, Bengaluru · Hybrid

Software Engineer at Microsoft, Mar 2025 - Jun 2026 · 1 yr 4 mos, Bengaluru · Hybrid, Org: Microsoft Security Architected end-to-end cloud infrastructure from scratch for a sovereign / highly regulated cloud environment, using Bicep and ARM templates to enable repeatable, automated deployments across environments with zero configuration drift. Designed and deployed secure, multi-service cloud architectures covering data processing, analytics, storage, orchestration, and compute — fully aligned with Microsoft's strictest compliance and security requirements. Implemented identity-first security patterns across all infrastructure resources using managed identities and RBAC, enforcing least-privilege access by default and eliminating credential-based authentication vectors. Integrated cloud infrastructure with governance, audit, and monitoring systems to ensure full operational traceability in restricted regions — enabling compliance validation without manual intervention. Delivered and validated production data pipelines on sovereign cloud infrastructure, directly enabling downstream business and compliance workflows for [X internal teams]. Owned production readiness end-to-end: environment validation, infrastructure hardening, incident mitigation, and iterative reliability improvements — reducing deployment failures through systematic IaC modularization. Resolved sovereign-cloud-specific challenges including service limitations, regional constraints, and cross-tenant access issues, unblocking pipeline delivery on a timeline critical to compliance deadlines.

Data Engineer at Media.net, Jul 2023 - Mar 2025 · 1 yr 9 mos, Bengaluru, Built and maintained large-scale Apache Spark data pipelines processing 2 TB/day of advertising event data, serving both ML feature generation and in-house reporting dashboards for reporting and business downstream teams. Designed batch ETL workflows on HDFS and Hive, optimizing data partitioning and job scheduling to reduce pipeline execution time by 20% and support 50 daily active pipeline runs. Integrated Apache Druid for real-time analytics ingestion, enabling sub-second query latency on streaming advertising event data for live dashboard consumption. Collaborated closely with ML teams to design and maintain feature stores and data contracts, ensuring pipeline output consistency and reproducibility for model training and inference. Maintained MySQL-based relational data layers supporting operational reporting, and established data quality monitoring to detect schema drift and data anomalies before downstream impact. Operated in an AdTech data environment managing inventory signals for 500,000+ publisher websites, requiring strict SLA adherence and high-availability pipeline design.

Data Engineer 2 at HashedIn by Deloitte, Aug 2022 - Jul 2023 · 1 yr, Bengaluru, Redesigned state management logic for a production Akka Finite State Machine (FSM), eliminating a class of data loss bugs in the core data pipeline and improving message delivery reliability. Led technical mentorship for a cohort of ~15 junior engineers across Scala, Akka, and Play frameworks — running structured learning sessions and conducting code reviews to uplevel team capability. Drove production reliability improvements to streaming pipelines processing FinTech transaction data, reducing pipeline error rates and improving reconciliation accuracy across data sources.

Data Engineer at HashedIn by Deloitte, Jun 2021 - Aug 2022 · 1 yr 3 mos, Bengaluru, Built and owned Akka Streaming pipelines processing customer orders and payment transaction data in real time, serving downstream reconciliation and settlement workflows. Implemented multi-source data reconciliation logic, aligning transaction records across two heterogeneous data sources and generating reconciled output files for downstream finance teams. Leveraged Akka Cluster Sharding for stateful, distributed stream processing — enabling horizontal scale-out of pipeline compute without data duplication or ordering violations. Integrated Kafka as the message bus for event-driven pipeline triggering, ensuring exactly-once-semantics-compatible processing of payment events with CosmosDB as the operational data store.

SDE Intern at HashedIn by Deloitte, Mar 2021 - Jun 2021 · 4 mos, Completed a 2-month internship with Hashedin University, featuring a 4-week training program followed by a month of hands-on project work. Developed a range of technical skills, including HTML, CSS, Typescript, Java SpringBoot, Angular, data engineering principles, Amazon Athena, S3 Buckets, Python, and SQL. Contributed to the development of a dynamic and responsive website, a billing system with APIs, an Angular website, and a desktop application using Electron. Demonstrated strong problem-solving and team collaboration skills, successfully completing an end-to-end project from scratch with a self-selected team. Gained valuable hands-on experience applying learned concepts and techniques to real-world projects, enhancing practical skills and understanding of data engineering.

Data Science Intern at LeewayHertz - Software Development Company, Jun 2020 - Aug 2020 · 3 mos, India, During my internship, I worked on implementing a Face Mask Detection and Social Distancing Detection system I utilized Transfer Learning to train a model for Face Mask Detection using publicly available data. Despite the lack of benchmark datasets for this relatively new problem, I was able to achieve an accuracy of 70% on the test set of 200 images. For the Social Distancing Detection project, I collaborated with a co-intern to come up with a solution.

Summer Intern at Bennett University, Jun 2020 - Jul 2020 · 2 mos, Remote, Worked on a team of 6 members on the "Pneumonia Classification using x-ray images" project. Conducted research and explored data to identify the best approach for the project. Tried various machine learning and deep learning models, including VGG16, ResNet50, and InceptionV3, and used them as feature extractors with SVM for classification. Achieved the best results with InceptionV3, achieving an accuracy of 97% on 624 test images.

Education

Bachelor of Technology - BTech, Computer Science, ABES Engineering College