Headline

Senior Data Engineer | Azure Databricks | PySpark | Apache Spark | Delta Lake | ADF | Spark Streaming

About

Senior Data Engineer with 5+ years of experience designing large-scale data platforms using Azure Databricks, PySpark, and Delta Lake to process enterprise datasets and power analytics platforms. Specialized in Metadata-Driven Frameworks and GenAI-integrated analytics (NL2SQL) using the Databricks Lakehouse and Azure ecosystem. Proven track record of reducing engineering overhead by 80% through automation and optimizing distributed Spark workloads to handle multi-terabyte datasets. Expert in Unity Catalog governance, Medallion Architecture, and end-to-end CI/CD for mission-critical data products. My skills: Designing and implementing end-to-end ETL/ELT pipelines across the Bronze, Silver, and Gold layers in Medallion architecture Developing metadata-driven frameworks to dynamically control ingestion, transformation, SCD Type 1/2 logic, and schema evolution Working with structured streaming using Spark and Delta Live Tables, including checkpointing and fault tolerance Managing credentials securely using Azure Key Vault, access connectors, and parameterized configurations across environments Leveraging Azure Data Factory for orchestration, retries, dependency handling, and scheduled workflows Implementing CI/CD pipelines for workspace migration, job deployment, and environment-specific code Debugging and supporting production workloads with robust logging, alerting, and audit controls.

Location

Ballari

AI stack

claude code

gemini

github copilot

hugging face

Work experience

Senior Data Engineer at EY, Jan 2025 - Present · 1 yr 10 mos, Bengaluru, Karnataka, India

Data Engineer at Infosys, May 2021 - Jan 2025 · 3 yrs 9 mos, Bengaluru, Karnataka, India

Education

Bachelor of Engineering - BE, Electrical and Electronics Engineering, Rao Bahadur Y Mahabaleshwarappa Engineering College Bellary

Bachelor of Engineering - BE, Electrical and Electronics Engineering, Visvesvaraya Technological University