Headline

Senior Azure Data Engineer | ADF · Databricks · PySpark · Delta Lake · Snowflake | Lakehouse Architecture | Open to Senior DE Roles

About

Reduced ETL pipeline runtime by 58% and eliminated recurring data inconsistency incidents by designing a HIPAA-compliant Bronze-Silver-Gold Lakehouse architecture using Azure Data Factory, Databricks, PySpark, and Delta Lake. I specialize in modernizing legacy ETL systems into scalable cloud-native data platforms — migrating SSIS workflows to ADF + Databricks, building metadata-driven orchestration frameworks, optimizing large-scale PySpark transformations, and implementing enterprise-grade governance controls. Currently working as a Migration Lead at HP Inc., designing target architecture standards for enterprise ETL modernization initiatives involving ADF, Databricks, RBAC governance, audit logging, and reusable orchestration patterns. Previously delivered healthcare data platforms processing 500 GB+ monthly data loads with: • 58% runtime reduction • 95% automated data quality detection • Pipeline failure reduction from 18% → <5% • Zero HIPAA audit violations Core Stack: Azure Data Factory · Databricks · PySpark · Delta Lake · Snowflake · ADLS Gen2 · Azure SQL · CI/CD · Power BI · T-SQL Open to: Senior Data Engineer · Databricks Engineer · Lakehouse Engineer · Azure Data Engineer roles in Bengaluru / Remote 📩 kspranav509@gmail.com

Location

Greater Bengaluru Area

AI stack

antigravity

claude

github copilot

hugging face

ollama

ChatGPT

Work experience

Data Engineer at HP, Mar 2026 - Present · 8 mos, Bengaluru, Karnataka, India, • Leading enterprise ETL modernization initiatives involving migration of legacy Power Automate and Python-based workflows into cloud-native Azure Data Factory + Databricks architecture. • Conducting end-to-end assessment of 15+ enterprise ETL pipelines — documenting data lineage, identifying transformation bottlenecks, evaluating orchestration dependencies, and defining migration strategy. • Designing standardized ADF + Databricks orchestration frameworks including reusable parameterized components, metadata-driven pipeline patterns, RBAC governance standards, and enterprise naming conventions. • Defining governance and compliance standards for the target Lakehouse platform including audit logging, role-based access control (RBAC), data classification, and operational monitoring. • Improved observability and monitoring workflows by implementing structured alert categorization and severity-based routing, reducing mean-time-to-resolution for production failures.

Software Engineer at Infinite Computer Solutions, Mar 2023 - Mar 2026 · 3 yrs 1 mo, Hybrid, • Architected and deployed Azure Data Factory pipelines migrating 500 GB+ monthly healthcare and claims data from on-prem SQL Server to Azure SQL Database — reducing pipeline runtime from 6 hours to 2.5 hours (58% improvement). • Led migration of 8 legacy SSIS workflows into Azure Data Factory + Databricks Lakehouse architecture using PySpark and Delta Lake, eliminating ~40% operational overhead and enabling ACID-compliant data processing. • Designed Medallion Architecture (Bronze-Silver-Gold) Lakehouse pipelines processing 100K+ records per batch using Delta Lake optimization techniques including OPTIMIZE, ZORDER, partition pruning, and broadcast joins. • Built reusable metadata-driven ADF orchestration frameworks with dynamic expressions, parameterized pipelines, trigger-based execution, and Integration Runtime configurations — reducing new pipeline development effort by 35%. • Engineered automated PySpark + T-SQL validation framework implementing row-count reconciliation, schema drift detection, null-check validation, and referential integrity checks — detecting 95% of data quality issues pre-reporting. • Optimized high-volume T-SQL workloads including stored procedures, clustered/non-clustered indexes, and execution plan tuning across OLTP and OLAP systems — improving query performance by 25%. • Integrated Snowflake as enterprise analytical data store through Azure Data Factory connectors — optimizing warehouse sizing, clustering keys, and micro-partition strategies, reducing dashboard query latency by 71%. • Implemented CI/CD pipelines using Azure DevOps for ADF ARM template deployments and Databricks notebook releases — reducing deployment cycles from 2 days to 2 hours with rollback support. • Reduced production pipeline failure rate from 18% to below 5% through structured error handling, automated retry mechanisms, and Azure Monitor-based observability pipelines.

Associate Software Engineer at Infinite Computer Solutions, Nov 2021 - Mar 2023 · 1 yr 5 mos, Bengaluru, Karnataka, India, - Developed and maintained SSIS-based ETL packages for US healthcare claims data — implementing control flow, data flow, and T-SQL stored procedures to automate data ingestion across 3 reporting streams. - Streamlined EDI 820 & 837 healthcare transaction file ingestion into SQL Server — standardising load processes and reducing integration errors by ~40%. - Supported QA and UAT validation cycles across 4 deployment phases — maintaining a 90%+ defect-free rate with zero production rollbacks.

Education

Bachelor of Engineering (BE), Electronics and Communications Engineering, AMC Engineering College - India

pre university, Computer Science, Narayana