Headline

Senior Data Engineer || Databricks || Spark || Kafka || AWS || Data Governance || PII Security || Data Reliability ||FinOps

About

I’m a Senior Data Engineer with 5+ years of experience building production lakehouse systems on AWS and Databricks. I have architected multi-workspace environments supporting thousands of users and hundreds of datasets, implementing access governance, auditability, and standardised onboarding so multiple teams can independently use data without compromising security or stability. I designed a Secure PII Data Vault using segregated workspaces, encryption, hashing, dynamic masking, RBAC authentication, and audit logging to enforce zero-trust access. This allows downstream teams to consume governed datasets for analytics and ML while preventing exposure of sensitive data. I also own the reliability of a high-throughput lakehouse ingesting 1–10TB/day across batch and streaming workloads using Spark, Delta Lake, and Kafka. I built 400+ production pipelines and implemented schema evolution handling, backfills, and observability to keep systems stable in real-world conditions. To improve trust in data, I developed a reusable PySpark + Great Expectations data quality framework and implemented cost visibility and guardrails across cloud accounts to optimise platform spend. My focus is platform ownership, governance, and reliability — building secure, observable, and self-service data foundations that analytics and ML teams can depend on. Tech: Python, SQL, PySpark, Spark Structured Streaming, Kafka, Delta Lake, Databricks, AWS (S3, IAM, KMS, Glue, EMR, Lambda, SQS).

Location

Greater Bengaluru Area

Work experience

Data Engineer II at Angel One, Jan 2024 - Present · 2 yrs 8 mos, Bengaluru, Karnataka, India

Senior Digital Engineer 1 at Tata Digital, Apr 2022 - Dec 2023 · 1 yr 9 mos, Bengaluru, Karnataka, India

Digital Engineer at Tata Digital, Oct 2020 - Apr 2022 · 1 yr 7 mos, Mumbai, Maharashtra, India

Research Intern at TCS Research & Innovation, May 2019 - Jul 2019 · 3 mos, Kolkata, West Bengal, India

Education

Bachelor of Engineering - BE, Computer Science, Jadavpur University