Abhishek Paul profile photo 1 on Tal

Abhishek Paul

Azure Data Engineer @ PwC India | Azure Databricks | Azure Data Factory | PySpark | Azure SQL | Unity Catalog | Delta Table | Azure DevOps | Power BI

Kolkata

5+ years of work experience

PwC India
Jun 20251 yr 1 moPresent

Kolkata, West Bengal, India

• Designed and implemented an end-to-end Azure Data Engineering platform using Azure Data Factory, Azure Databricks, and ADLS Gen2 following Medallion Architecture (Bronze, Silver and Gold layers), enabling centralized storage, processing, and governance of enterprise data for analytical reporting. • Developed parameterized Azure Data Factory pipelines to ingest data from Azure SQL Database into ADLS Gen2, creating reusable and metadata-driven ingestion frameworks that reduced onboarding effort for new datasets by approximately 80%. • Implemented watermark-based incremental loading to capture only newly inserted and updated records, reducing data processing volume and pipeline execution time by nearly 60% compared to full-load processing. • Built and maintained 20+ PySpark notebooks for data cleansing, transformation, validation, and business rule implementation, improving data quality and reducing manual intervention by 50%. • Designed and implemented Star Schema data models consisting of Fact and Dimension tables, including SCD Type 1 processing using Delta Lake MERGE operations, enabling optimized analytical reporting and improving query performance, by approximately 30%. • Developed Delta Lake-based transformation pipelines and curated Gold-layer datasets to support KPI reporting, trend analysis, and Power BI dashboards, providing analytics-ready data for business users and management stakeholders. • Configured Unity Catalog for centralized metadata management, lineage tracking, schema governance, and role- based access controls, improving data discoverability, governance compliance, and secure access management across analytical assets. • Optimized Spark workloads using partitioning, caching, Delta Lake optimization and efficient transformation strategies, improving query performance by approximately 35% while reducing overall compute resource utilization.
Tata Consultancy Services
Dec 20213 yrs 7 mosJun 2025

- Design, build, and maintain ELT pipelines to collect, store, and process data from various sources using Azure Databricks. - Design and optimize spark notebooks and jobs for handling large datasets, reducing runtime by 30%. -Implemented data validation scripts, reducing data discrepancies and ensuring data integrity. - Utilize Databricks workflow and task orchestration to streamline full load and incremental load processing from raw to structured data layers. - Develop and enforce best practices for data governance, security, and privacy, ensuring compliance with industry standards and regulations using Unity Catalog. - Work on integrating the Data validation framework for robust data quality checks.
Tata Consultancy Services
Aug 20215 mosDec 2021

India

Learnt about different Cloud Platforms - GCP, Azure, and AWS
Internshala
Mar 20203 mosMay 2020

Kolkata, West Bengal, India

As an ISP, my role involved spreading awareness about Internshala, facilitating internships, organizing events, and providing feedback to Internshala. I actively promoted Internshala among my peers and helped interested candidates apply for relevant internships. I organized and conducted events such as workshops, webinars, and networking sessions to help students gain valuable skills and knowledge.
The SnapStore App
Feb 20182 mosMar 2018

Kolkata

Generated 100+ new customer accounts and assisted in formulating effective marketing strategies and campaigns. Handled tie-ups for social media posts and worked to increase the user base.

Education

RCC Institute of Information Technology

Bachelor of Technology - BTech, Computer Science

RCC Institute of Information Technology

20174 yrs2021
Delhi Public School - India

Senior Secondary, Science

Delhi Public School - India

20152 yrs2017

Join Abhishek and 10k+ other professionals on

Tal Boss

Tal is the app where bosses meet engineers & builders directly, without middlemen

Download Tal Boss

Abhishek Paul on Tal

Headline

Azure Data Engineer @ PwC India | Azure Databricks | Azure Data Factory | PySpark | Azure SQL | Unity Catalog | Delta Table | Azure DevOps | Power BI

About

Azure Data Engineer with around 5 years of experience designing enterprise data integration and analytics solutions using Azure Data Factory, Azure Databricks, PySpark, Delta Lake, and ADLS Gen2. Experienced in building scalable ETL/ELT pipelines, implementing incremental loading frameworks, optimizing Spark workloads, and delivering analytics-ready datasets for reporting and business intelligence. TECHNICAL SKILLS Programming: Python, SQL, PySpark Data Engineering: Azure Databricks, Azure Data Factory, Mapping Data flows, Delta Lake, Unity Catalog, Databricks Workflow, Azure Logic Apps Storage and Databases: ADLS Gen2, Azure SQL Database, Delta Tables Data Modelling: Star Schema, Data Warehousing, Slowly Changing Dimension (SCD), Medallion Architecture DevOps & Monitoring: Azure DevOps, Git, Azure Monitor Visualization: Power BI Cloud Platform: Microsoft Azure

Location

Kolkata

Work experience

Senior Associate at PwC India, Jun 2025 - Present · 1 yr 1 mo, Kolkata, West Bengal, India, • Designed and implemented an end-to-end Azure Data Engineering platform using Azure Data Factory, Azure Databricks, and ADLS Gen2 following Medallion Architecture (Bronze, Silver and Gold layers), enabling centralized storage, processing, and governance of enterprise data for analytical reporting. • Developed parameterized Azure Data Factory pipelines to ingest data from Azure SQL Database into ADLS Gen2, creating reusable and metadata-driven ingestion frameworks that reduced onboarding effort for new datasets by approximately 80%. • Implemented watermark-based incremental loading to capture only newly inserted and updated records, reducing data processing volume and pipeline execution time by nearly 60% compared to full-load processing. • Built and maintained 20+ PySpark notebooks for data cleansing, transformation, validation, and business rule implementation, improving data quality and reducing manual intervention by 50%. • Designed and implemented Star Schema data models consisting of Fact and Dimension tables, including SCD Type 1 processing using Delta Lake MERGE operations, enabling optimized analytical reporting and improving query performance, by approximately 30%. • Developed Delta Lake-based transformation pipelines and curated Gold-layer datasets to support KPI reporting, trend analysis, and Power BI dashboards, providing analytics-ready data for business users and management stakeholders. • Configured Unity Catalog for centralized metadata management, lineage tracking, schema governance, and role- based access controls, improving data discoverability, governance compliance, and secure access management across analytical assets. • Optimized Spark workloads using partitioning, caching, Delta Lake optimization and efficient transformation strategies, improving query performance by approximately 35% while reducing overall compute resource utilization.

System Engineer at Tata Consultancy Services, Dec 2021 - Jun 2025 · 3 yrs 7 mos, - Design, build, and maintain ELT pipelines to collect, store, and process data from various sources using Azure Databricks. - Design and optimize spark notebooks and jobs for handling large datasets, reducing runtime by 30%. -Implemented data validation scripts, reducing data discrepancies and ensuring data integrity. - Utilize Databricks workflow and task orchestration to streamline full load and incremental load processing from raw to structured data layers. - Develop and enforce best practices for data governance, security, and privacy, ensuring compliance with industry standards and regulations using Unity Catalog. - Work on integrating the Data validation framework for robust data quality checks.

Assistant System Engineer-Trainee at Tata Consultancy Services, Aug 2021 - Dec 2021 · 5 mos, India, Learnt about different Cloud Platforms - GCP, Azure, and AWS

Internshala Student Partner 17 at Internshala, Mar 2020 - May 2020 · 3 mos, Kolkata, West Bengal, India, As an ISP, my role involved spreading awareness about Internshala, facilitating internships, organizing events, and providing feedback to Internshala. I actively promoted Internshala among my peers and helped interested candidates apply for relevant internships. I organized and conducted events such as workshops, webinars, and networking sessions to help students gain valuable skills and knowledge.

Campus Ambassador at The SnapStore App, Feb 2018 - Mar 2018 · 2 mos, Kolkata, Generated 100+ new customer accounts and assisted in formulating effective marketing strategies and campaigns. Handled tie-ups for social media posts and worked to increase the user base.

Education

Bachelor of Technology - BTech, Computer Science, RCC Institute of Information Technology

Senior Secondary, Science, Delhi Public School - India