Dipak Kasar profile photo 2 on Tal
Dipak Kasar profile photo 1 on Tal
Dipak Kasar profile photo 2 on Tal
Dipak Kasar profile photo 1 on Tal

Dipak Kasar

Azure Data Engineer | ADF | Azure Databricks | PySpark | Azure Synapse | ADLS Gen2 | SQL | Delta Lake | ETL/ELT | Data Pipelines

Sinnar

3+ years of work experience

Self-Employed
Apr 20265 moPresent

Pune

Building end-to-end Azure Data Engineering solutions focused on scalable ingestion, transformation, orchestration, lakehouse architecture, security, and analytics. Key areas of hands-on work: • Designed metadata-driven Azure Data Factory pipelines for reusable multi-source ingestion and incremental loading using watermark/LastLoad patterns. • Built Bronze, Silver, and Gold Medallion Architecture on ADLS Gen2 with PySpark, Delta Lake, and Azure Databricks. • Implemented Databricks Auto Loader and Structured Streaming for incremental data processing with checkpointing. • Developed Delta Live Tables (DLT) pipelines with data-quality validation, CDC, and SCD Type 1 & Type 2 processing. • Built Azure Synapse Serverless SQL external tables and dimensional/star-schema models for analytical consumption. • Implemented secure data access and secret management using Azure Key Vault and Managed Identity. • Integrated Azure Logic Apps for pipeline monitoring, status, and error notifications. • Used Git and GitHub for version control, project documentation, and portfolio implementations. Tech Stack: Azure Data Factory | Azure Databricks | PySpark | Apache Spark | ADLS Gen2 | Azure Synapse Analytics | SQL | Python | Delta Lake | DLT | Auto Loader | Structured Streaming | Azure SQL | Azure Key Vault | Logic Apps
Career Break
Mar 20241 yrFeb 2025

Sinnar

Took a career break to support an immediate family member through an extended period of medical treatment and recovery.
Infosys
May 20221 yr 10 mosFeb 2024

India

Client: Westpac Banking Corporation • Designed and developed metadata-driven ETL pipelines using Azure Data Factory and Azure Databricks to ingest data from multiple relational databases into ADLS Gen2 and Delta Lake. • Developed a PySpark-based schema synchronization framework to dynamically create and maintain target Delta tables based on source schemas. • Built reusable data reconciliation and row-count validation frameworks using PySpark, including hash-based data-quality checks between source and target systems. • Implemented pipeline-delay and SLA-monitoring mechanisms using ADF monitoring, Databricks job status, and automated failure and delay notifications. • Developed SQL and PySpark transformation logic and implemented Delta Lake MERGE operations according to business requirements. • Developed a reusable PySpark ingestion framework in Azure Databricks to support scalable batch and incremental data processing. Technologies: Azure Data Factory, Azure Databricks, PySpark, Python, SQL, ADLS Gen2, Delta Lake, Azure SQL Database, Azure Key Vault and Unity Catalog.
GitHub
@CodeWithDipak

Dipak’s AI Stack

claude
claude
codex
codex
openai
openai
ChatGPT
ChatGPT

Education

K.K.Wagh institute of Engineering Education & Research

Bachelor of Engineering - BE, Electrical

K.K.Wagh institute of Engineering Education & Research

20155 yrs2020

Join Dipak and 10k+ other professionals on

Tal Boss

Tal is the app where bosses meet engineers & builders directly, without middlemen

Download Tal Boss

Dipak Kasar on Tal

Headline

Azure Data Engineer | ADF | Azure Databricks | PySpark | Azure Synapse | ADLS Gen2 | SQL | Delta Lake | ETL/ELT | Data Pipelines

About

Azure Data Engineer with 2 years of professional experience at Infosys, working on a data engineering project for Westpac. I have experience designing and developing Azure-based solutions for data ingestion, transformation, archival, pipeline orchestration, and monitoring using Azure Data Factory, Azure Databricks, PySpark, SQL, ADLS Gen2, Delta Lake, Azure Key Vault, and Azure Logic Apps. My experience includes developing metadata-driven ETL pipelines, schema synchronization frameworks, data-reconciliation checks, incremental data processing, Delta Lake MERGE operations, and pipeline monitoring mechanisms. I have also developed an end-to-end Spotify Azure Lakehouse project implementing metadata-driven ingestion, watermark-based incremental loading, BackDate historical reprocessing, Medallion Architecture, Databricks Auto Loader, Structured Streaming, Delta Live Tables, data-quality expectations, CDC, and SCD Type 1 and Type 2 processing. Core skills: Azure Data Factory | Azure Databricks | PySpark | Apache Spark | SQL | Python | ADLS Gen2 | Delta Lake | Azure Synapse Analytics | Azure SQL Database | Azure Key Vault | Unity Catalog | ETL/ELT | Medallion Architecture GitHub: https://github.com/CodeWithDipak

Location

Sinnar

AI stack

claude

codex

openai

ChatGPT

Work experience

Other at Self-Employed, Apr 2026 - Present · 5 mo, Pune, Building end-to-end Azure Data Engineering solutions focused on scalable ingestion, transformation, orchestration, lakehouse architecture, security, and analytics. Key areas of hands-on work: • Designed metadata-driven Azure Data Factory pipelines for reusable multi-source ingestion and incremental loading using watermark/LastLoad patterns. • Built Bronze, Silver, and Gold Medallion Architecture on ADLS Gen2 with PySpark, Delta Lake, and Azure Databricks. • Implemented Databricks Auto Loader and Structured Streaming for incremental data processing with checkpointing. • Developed Delta Live Tables (DLT) pipelines with data-quality validation, CDC, and SCD Type 1 & Type 2 processing. • Built Azure Synapse Serverless SQL external tables and dimensional/star-schema models for analytical consumption. • Implemented secure data access and secret management using Azure Key Vault and Managed Identity. • Integrated Azure Logic Apps for pipeline monitoring, status, and error notifications. • Used Git and GitHub for version control, project documentation, and portfolio implementations. Tech Stack: Azure Data Factory | Azure Databricks | PySpark | Apache Spark | ADLS Gen2 | Azure Synapse Analytics | SQL | Python | Delta Lake | DLT | Auto Loader | Structured Streaming | Azure SQL | Azure Key Vault | Logic Apps

Caregiving at Career Break, Mar 2024 - Feb 2025 · 1 yr, Sinnar, Took a career break to support an immediate family member through an extended period of medical treatment and recovery.

System Engineer | Data Engineer at Infosys, May 2022 - Feb 2024 · 1 yr 10 mos, India, Client: Westpac Banking Corporation • Designed and developed metadata-driven ETL pipelines using Azure Data Factory and Azure Databricks to ingest data from multiple relational databases into ADLS Gen2 and Delta Lake. • Developed a PySpark-based schema synchronization framework to dynamically create and maintain target Delta tables based on source schemas. • Built reusable data reconciliation and row-count validation frameworks using PySpark, including hash-based data-quality checks between source and target systems. • Implemented pipeline-delay and SLA-monitoring mechanisms using ADF monitoring, Databricks job status, and automated failure and delay notifications. • Developed SQL and PySpark transformation logic and implemented Delta Lake MERGE operations according to business requirements. • Developed a reusable PySpark ingestion framework in Azure Databricks to support scalable batch and incremental data processing. Technologies: Azure Data Factory, Azure Databricks, PySpark, Python, SQL, ADLS Gen2, Delta Lake, Azure SQL Database, Azure Key Vault and Unity Catalog.

Education

Bachelor of Engineering - BE, Electrical, K.K.Wagh institute of Engineering Education & Research

Public links

github: https://github.com/CodeWithDipak