Headline

Data Engineer | Big data | Spark | Python

About

I have almost 5 years of experience working on big data projects. Currently, I work as a Data Engineer at ErosNow, where we work as a team to integrate data from multiple data sources into a data lake. I have experience in processing huge datasets and delivering solutions as per requirements. Previously, I worked with Vistaar Technologies as a Software Engineer, improving products and services for our customers and templatizing our solutions to handle different customers with millions of records. Apart from work, I love sketching and I am an avid book reader! Competencies: Iceberg | Spark | Trino | Kafka | Flink | Airflow | dbt | Python | Java | Kubernetes

Location

Mumbai

AI stack

claude

cursor

gemini

openai

claudecode

Work experience

SDE-2 at Dream11, Jul 2022 - Present · 4 y 1 mo, Mumbai, Maharashtra, India, ● Self-Service ETL Platform: Designed a secure, multi-tenant ETL scheduling platform orchestrating Spark and Trino(Presto) workloads, managing ~4,000 jobs, saving 200+ hours of engineering effort per quarter. ● Self-serve dashboarding platform: Architected and led a self-serve dashboard platform on AWS with RBAC, enabling tech and non-tech teams to independently deploy and schedule ~200 dashboards, eliminating engineering dependency and reducing reporting turnaround from 7 days to 2 hours. ● Lakehouse Migration: Led the Redshift to Iceberg ETL migration using dbt and a custom SQLGlot-based transpiler, automating dialect conversion at scale, validating integrity using Great Expectations, improving workload isolation and delivering $700/day in cost savings. ● Medallion Architecture: Architected a Medallion (Bronze–Silver–Gold) structure and centralized semantic layer (Metric Atlas), reducing dashboard load times from minutes to seconds. Designed dimensional models for the Gold layer, enabling consistent KPI definitions across 20+ dashboards and reducing p95 dashboard load latency by 50%. ● Standardizing Event Metadata: Led development of a centralized, version-controlled event metadata governance platform using a Git-like branching model, replacing manual sheets and enforcing standardized, high-quality event metadata across teams. ● DataHighway: Built a schema-gated real-time event ingestion platform processing 495M events/min (8.25M RPS, peak concurrency 16.11M) — auto-provisioning Iceberg tables on schema registration, reducing analytics latency from ~1 day to ~3 minutes, and cutting downstream job failures to near-zero via schema admission control. ● DataHighway – Scaling & Cost Governance: Scaled Kafka infrastructure to sustain 495M events/min with zero downtime during peak loads. Built a cost visibility platform attributing per-team storage costs to event owners, giving each team direct visibility into their event storage costs. ● Stream Processing Platform: Built real-time stream processing pipelines on Apache Flink — hands-on with Table API, stateful processing, checkpointing/savepoints, Kafka connectors, and parallelism tuning. ● Automation Ingestion Framework: Built an in-house ingestion framework for Apache Iceberg, automating schema evolution and backfills across ~140 tables while reducing manual effort by 80 hours/month.

Senior Software Developer at Excelra, Nov 2021 - Aug 2022 · 10 mos

Data Engineer at ErosSTX, Mar 2020 - Oct 2021 · 1 yr 8 mos, Mumbai, Maharashtra, India

Software Engineer at Vistaar Technologies, Apr 2017 - Feb 2020 · 2 yrs 11 mos

Education

Bachelor of Engineering - BE, Computer Engineering, Aldel Education Trusts St John College of Engg Vevoor Palghar Tal Palghar Dist Thane