Headline

Data Engineer | Azure Databricks | ADF | Spark (Python, Scala)

About

Data Engineer with 5+ years of experience in designing, developing, and optimizing scalable ETL/ELT pipelines on Azure. Strong expertise in Azure Databricks, PySpark, and SQL for building data processing workflows and transforming large-scale datasets. Hands-on experience in developing efficient data pipelines, improving Spark performance, and ensuring data quality. Proven ability to collaborate with cross-functional teams to deliver high quality, scalable, and production-grade data solutions.

Location

Greater Delhi Area

Work experience

Senior Associate Data Engineer at Affine, Mar 2023 - Present · 3 yrs 3 mos, Mumbai, Maharashtra, India, • Designed and developed scalable ETL/ELT pipelines using Azure Data Factory and Azure Databricks to ingest/process structured and unstructured data from multiple sources. • Built distributed data processing workflows using PySpark and Spark SQL within Databricks notebooks. • Implemented data transformation and aggregation logic to support downstream analytics and reporting. • Worked with Azure Data Lake Storage (ADLS) for efficient storage, partitioning and retrieval of data. • Collaborated with cross-functional teams to gather requirements and deliver data-driven solutions. • Implemented data quality rules and validation checks across pipelines to ensure accuracy and consistency. • Troubleshoot pipeline failures, performance bottlenecks, and cluster-related issues in Databricks environments.

Associate Software Engineer at Mphasis, Jan 2022 - Mar 2023 · 1 yr 3 mos

Associate Software Engineer Trainee at Mphasis, Nov 2020 - Dec 2021 · 1 yr 2 mos

Education

Bachelor of Technology - BTech, Computer Science Engineering, KRISHNA INSTITUTE OF ENGINEERING AND TECHNOLOGY, GHAZIABAD