Headline

Lead Data Engineer | Specialist in Azure Databricks & Delta Lake | $6M Cost Avoidance | Building Scalable, High-Performance Global Data Platforms

About

Senior Data Engineer with 15+ years in IT, including 6+ years specializing in Azure-based data platforms (Azure Databricks, ADF, Delta Lake, PySpark). I build scalable, reliable data systems processing multi-terabyte workloads across Healthcare and IoT domains, with a strong focus on data reliability, performance optimization, and cost efficiency. Key Impact: • Reduced enterprise pipeline failure rate by 85% (9% → 1.3%) • Delivered ~$6M annualized cost savings through data reliability initiatives • Improved ETL performance by 37% via Lakehouse re-architecture • Built and maintained pipelines processing 3TB+ enterprise datasets • Optimized 40TB Cassandra infrastructure through schema tuning and compression • Negotiated $60K annual savings on DataStax licensing Core Expertise: • Azure Databricks, ADF, Delta Lake, ADLS Gen2 • PySpark, Spark SQL, Distributed Data Processing • Lakehouse Architecture & Data Modeling • ETL/ELT Pipelines (Batch & Incremental) • Data Reliability, Observability & Cost Optimization AI-Ready Data Engineering: Experience building data platforms that enable analytics and AI/ML use cases through high-quality, scalable datasets. Global Remote Ready: 3+ years working with distributed teams across Europe and the US with async collaboration and documentation-first delivery. Open to Senior / Lead Data Engineer roles (Remote | US/EU).

Location

Bengaluru

AI stack

claude code

claude

github copilot

Work experience

Senior Data Engineer at Wipro, Jul 2023 - Present · 3 yrs 2 mos, Designed and implemented Azure Databricks pipelines on the EADI platform processing 3+ TB of enterprise HR and Healthcare operational data with high availability and SLA compliance. Served as Service Lead for a 10-member Operations team — owning the Gold layer across 7 TEPOPs applications, ensuring data quality and SLA adherence for downstream consumers. Led delivery of a Data Product integrating SAP HRV into Azure via Databricks, processing 1+ TB of enterprise datasets from requirements through to production. Designed and launched the IDoc error platform; reduced pipeline failure rate from 9% to 1.3%, delivering approximately $6M in annualised cost avoidance. Re-architected legacy ETL to incremental Delta Lake loads using Spark SQL; cut data refresh time by 37%, improving operational reliability. Collaborated with cross-continental stakeholders (India–Netherlands) using async delivery, Agile ceremonies, and sprint tracking — increasing delivery velocity by 20%.

Data Team Lead at Wipro ltd, Jul 2023 - Present · 3 y 1 mo, Bengaluru, Lead Data Engineer | Client: Philips Electronics, Netherlands • Architected Azure Lakehouse pipelines (Databricks + Delta Lake + ADF) processing 3+ TB data • Reduced pipeline failure rate from 9% → 1.3% (85% improvement) • Delivered ~$6M annual cost savings via data reliability platform • Improved ETL performance by 37% using PySpark optimizations • Built SAP HR data product (1TB+) end-to-end • Led 10-member team, owning Gold layer across 7 applications • Improved delivery velocity by 20% via async collaboration

Senior Data Engineer at Resideo, Jan 2018 - Jun 2023 · 5 yrs 6 mos, India, • Collaborated on a $2M Azure Databricks platform as part of a 25-member team. • Served as Product Owner for a 10-member Data Migration team, successfully migrating MySQL modules from IBM Cloud to Azure. • Acted as the SPOC with DataStax, negotiating a license cost reduction from $150K to $90K annually.

Senior Data Analyst at Honeywell, Jan 2018 - Oct 2018 · 10 mos, Managed large-scale Cassandra data operations for IoT analytics platform handling 80TB+ of connected home device data. Implemented storage optimisation and database tuning strategies, improving system reliability and query performance.

Big Data Engineer at Honeywell, Jan 2018 - Oct 2018 · 10 mos, Karnataka, India, • Architected a high-volume data archiving solution to improve application performance and scalability. • Administered and supported a 20-node Cassandra DB cluster (DSE 5.1.11) managing 60+ TB of data. • Optimized database performance by reducing storage footprint from 100TB → 60TB through data purging and compaction.

System Analyst at Verint, Apr 2016 - Jan 2018 · 1 yr 10 mos, Karnataka, India, • Developed and supported an in-house product for Account Managers in Contact Centers, enabling them to fetch and display agent-related information (e.g., Extension IDs, daily/weekly call volumes) via a dashboard reporting tool. • Collaborated with the Data Analytics team to translate business requirements into functional specifications and implement analytical features that improved reporting accuracy and usability.

Education

Master of Computer Applications - MCA, Computer Science, Visvesvaraya Technological University

Associate's degree, Computer Programming, Specific Applications, SSMRV College