Headline

Senior Azure Data Engineer | Azure Databricks | Azure Data Factory | PySpark | Delta Lake | ADLS Gen2 | ETL/ELT | Data Warehousing | Medallion Architecture | Spark Performance Optimization

About

Open to connecting 📩 Email: banikhil567@gmail.com I’m a Data Engineer with 5+ years of experience building and optimizing scalable data pipelines and cloud-based data platforms, primarily using Microsoft Azure and Databricks. My expertise includes: • Azure Data Factory (ADF) and ETL/ELT Pipelines • Azure Databricks, PySpark, Apache Spark, Python, SQL • Azure Data Lake Storage Gen2 (ADLS Gen2) and Azure Synapse Analytics • Delta Lake and Medallion Architecture, Unity Catalog • Incremental Loading and Metadata-Driven Pipelines • Spark Performance Optimization and Large-Scale Data Processing • Data Quality, Validation, Monitoring, and Production Support • AWS Cloud Services and Data Integration I focus on building reliable data pipelines, developing reusable ingestion frameworks, processing structured and semi-structured data, and improving performance and data quality in production environments. I’m interested in modern data engineering, Lakehouse architecture, cloud data platforms, and scalable data solutions.

Location

Greater Bengaluru Area

AI stack

claude code

claude

cursor

deepseek

AssemblyAI

llama.cpp

Work experience

Senior Data Engineer at Reveal HealthTech, Oct 2025 - Present · 1 yr 1 mo, Bengaluru, • Designed and developed healthcare data pipelines using Azure Databricks and Medallion Architecture (Bronze, Silver, Gold). • Processed FHIR healthcare datasets using PySpark to build analytics-ready Silver and Gold data layers. • Implemented incremental data ingestion using watermark-based file tracking for efficient and reliable processing. • Optimized Spark workloads using partitioning, caching, broadcast joins, and performance tuning techniques. • Developed scalable data transformation pipelines supporting healthcare analytics and reporting. • Performed data quality validation, production support, and issue resolution to ensure highly reliable data pipelines. • Collaborated with business stakeholders, analysts, and BI teams to deliver client-specific reporting solutions.

Data Engineer at ITTStar Consulting LLC, Aug 2022 - Sep 2025 · 3 yrs 2 mos, Bengaluru, • Designed and implemented end-to-end Azure data platforms using Azure Data Factory, Azure Databricks, ADLS Gen2, and Delta Lake. • Built scalable metadata-driven ETL pipelines using Lookup, ForEach, dynamic datasets, and parameterized workflows. • Developed high-performance PySpark transformation pipelines for large-scale analytical datasets. • Implemented Medallion Architecture and Delta Lake features including MERGE, UPSERT, schema evolution, and ACID transactions. • Migrated SQL Server data to Azure Data Lake using Self-Hosted Integration Runtime (SHIR). • Processed structured and semi-structured data including Delta, Parquet, CSV, and JSON. • Integrated Azure Key Vault with Azure Data Factory and Databricks for secure credential management. • Optimized Spark jobs using partitioning, caching, and query optimization techniques. • Implemented data validation, audit logging, monitoring, and exception handling to improve pipeline reliability. • Collaborated within Agile teams comprising architects, business analysts, QA engineers, and reporting teams to deliver enterprise data solutions.

Engineer at BAeHAL Software Ltd, Oct 2020 - Nov 2021 · 1 yr 2 mos, Bengaluru, Karnataka, India

Education

Bachelor of Engineering - BE, Aeronautical/Aerospace Engineering Technology/Technician, MVJ College of Engineering, Bangalore, India