Abhinav Gupta profile photo 1 on Tal

Abhinav Gupta

Data Engineer | Databricks & GCP Certified | PySpark · Delta Lake · Snowflake · BigQuery · SQL · ETL/ELT | Building Scalable Data Lakehouses at Enterprise Scale

Delhi

3+ years of work experience

InfoCepts
Apr 20251 yr 4 mosPresent

On-site

• Optimized multi-TB PySpark and Spark SQL pipelines using AQE, Broadcast Joins, Partition Pruning, and Skew Handling, reducing processing time by 40% and improving SLA compliance. • Designed and developed end-to-end ETL/ELT pipelines on Databricks (GCP), processing multi-terabyte structured and semi-structured data from Salesforce, SAP, BigQuery, TIBCO, and REST APIs. • Built a unified 360° Customer Data Platform integrating sales, campaign, and loyalty data to enable enterprise-wide analytics and reporting. • Implemented Medallion Architecture (Bronze, Silver, Gold) with Delta Lake, improving data quality, lineage tracking, and analytics readiness. • Developed complex PySpark and BigQuery SQL transformations for KPI calculations, schema standardization, data enrichment, and validation. • Established automated data quality checks and monitoring frameworks to ensure accuracy, consistency, and reliability of business-critical reporting datasets. • Applied enterprise data governance, security, and privacy controls across GCP data platforms. • Authored solution design documents, architecture diagrams, and operational runbooks while collaborating with business stakeholders, analysts, and BI teams to deliver scalable data solutions. Tech Stack: Databricks, PySpark, Spark SQL, Delta Lake, BigQuery, GCP, SQL, ETL/ELT, Data Modeling, Data Quality, Data Governance, REST APIs, Salesforce, SAP.
InfoCepts
Sep 20231 yr 8 mosApr 2025

Hybrid

• Led migration from legacy Netezza to a modern Data Lakehouse and Snowflake Data Warehouse, ensuring zero data loss and zero production downtime throughout the transition. • Analyzed and documented 2,000+ Informatica PowerCenter mappings, enabling legacy system modernization, data governance initiatives, and large-scale migration planning. • Developed and maintained enterprise ETL/ELT pipelines using Informatica PowerCenter, supporting data integration, transformation, and migration across multiple business-critical systems. • Built and orchestrated automated Databricks Workflows for batch data processing, ensuring reliable and SLA-compliant data delivery to downstream applications and reporting platforms. • Optimized complex SQL, Spark SQL, and Snowflake transformations, improving query performance and processing efficiency for high-volume datasets. • Supported REST API-based data ingestion frameworks and resolved integration challenges across enterprise platforms including Oracle, Snowflake, Informatica, and Databricks. • Collaborated with cross-functional teams including business analysts, architects, and data engineers to deliver scalable data solutions and support enterprise modernization programs. Tech Stack: Snowflake, Informatica PowerCenter, Databricks, Spark SQL, SQL, Oracle, Netezza, REST APIs, ETL/ELT, Data Migration, Data Warehousing, Data Lakehouse.
InfoCepts
Feb 20238 mosSep 2023

On-site

• Developed and maintained ETL pipelines using Python, PySpark, and SQL across cloud platforms including GCP (BigQuery, Cloud Storage) and AWS (S3, Glue, Athena) to support data ingestion, transformation, and analytics workloads. • Built Python-based automation solutions for data preprocessing, validation, and scheduled batch processing, improving operational efficiency and reducing manual effort. • Assisted in designing and implementing scalable data processing workflows for structured datasets, gaining hands-on experience with distributed computing and cloud-native data engineering practices. • Collaborated with cross-functional teams to gather business requirements, troubleshoot pipeline issues, and deliver reliable data solutions for downstream reporting and analytics. • Contributed to data lake and data warehouse initiatives by supporting ingestion, transformation, and consumption layers within production-grade data platforms. • Worked with Databricks, PySpark, SQL, and cloud services to develop foundational expertise in large-scale data processing, ETL development, and modern data architecture. Tech Stack: Python, PySpark, SQL, GCP, BigQuery, Cloud Storage, AWS S3, AWS Glue, Athena, Databricks, Linux, Shell Scripting, ETL.
Pepcoding Education Private Limited
May 20223 mosJul 2022

Noida, Uttar Pradesh, India

I have experience working on web development projects such as a To-Do List and a Camera Gallery Web Application. Additionally, I have developed DSA quizzes, actively participated in them, and served as a doubt solver for DSA-related queries.

Education

Greater Noida Institute of Technology(GNIOT)

Bachelor of Technology - BTech, Computer Science

Greater Noida Institute of Technology(GNIOT)

20194 yrs2023

XIIth

Convent Of Gagan Bharti,New Delhi

20171 yr2018

Join Abhinav and 10k+ other professionals on

Tal Boss

Tal is the app where bosses meet engineers & builders directly, without middlemen

Download Tal Boss

Abhinav Gupta on Tal

Headline

Data Engineer | Databricks & GCP Certified | PySpark · Delta Lake · Snowflake · BigQuery · SQL · ETL/ELT | Building Scalable Data Lakehouses at Enterprise Scale

About

I build the data infrastructure that powers business decisions at scale. As a Data Engineer with 3 years of hands-on experience, I specialize in designing and optimizing large-scale data pipelines, Data Lakehouse architectures, and cloud-native ETL/ELT systems — primarily on Databricks, GCP, AWS, and Snowflake. My engineering work spans multi-TB distributed pipelines, Medallion Architecture (Bronze/Silver/Gold), real-time data integration, and complex data migrations across enterprise-grade platforms. 🔧 Core Stack: PySpark · Spark SQL · Databricks · Delta Lake · Snowflake · BigQuery · GCP · AWS · Informatica · Oracle SQL ⚡ What I've delivered: • Engineered multi-TB PySpark pipelines with ~40% performance uplift through Adaptive Query Execution (AQE), Broadcast Joins, Partition Pruning, and Skew Handling • Implemented end-to-end Medallion Architecture (Bronze → Silver → Gold) on Databricks for a Retail & E-commerce client, significantly improving data quality and downstream reliability • Led Netezza-to-Snowflake/Data Lakehouse migration, modernizing legacy infrastructure without disrupting production workloads • Built and maintained data quality frameworks and CI/CD workflows for automated pipeline governance 🌐 Integrations I've worked with: Salesforce · SAP · TIBCO · REST APIs · BigQuery · Oracle · Snowflake 🏆 Certifications: • Databricks Certified Data Engineer Associate • GCP Professional Data Engineer I thrive at the intersection of distributed computing, cloud engineering, and business impact — translating raw data chaos into reliable, performant data products. 📩 Open to Data Engineer and Senior Data Engineer roles across Gurgaon, Noida, Pune, Bangalore, Hyderabad and Remote. Let's connect: abhinavgupta10101@gmail.com

Location

Delhi

Work experience

Data Engineer at InfoCepts, Apr 2025 - Present · 1 yr 4 mos, On-site, • Optimized multi-TB PySpark and Spark SQL pipelines using AQE, Broadcast Joins, Partition Pruning, and Skew Handling, reducing processing time by 40% and improving SLA compliance. • Designed and developed end-to-end ETL/ELT pipelines on Databricks (GCP), processing multi-terabyte structured and semi-structured data from Salesforce, SAP, BigQuery, TIBCO, and REST APIs. • Built a unified 360° Customer Data Platform integrating sales, campaign, and loyalty data to enable enterprise-wide analytics and reporting. • Implemented Medallion Architecture (Bronze, Silver, Gold) with Delta Lake, improving data quality, lineage tracking, and analytics readiness. • Developed complex PySpark and BigQuery SQL transformations for KPI calculations, schema standardization, data enrichment, and validation. • Established automated data quality checks and monitoring frameworks to ensure accuracy, consistency, and reliability of business-critical reporting datasets. • Applied enterprise data governance, security, and privacy controls across GCP data platforms. • Authored solution design documents, architecture diagrams, and operational runbooks while collaborating with business stakeholders, analysts, and BI teams to deliver scalable data solutions. Tech Stack: Databricks, PySpark, Spark SQL, Delta Lake, BigQuery, GCP, SQL, ETL/ELT, Data Modeling, Data Quality, Data Governance, REST APIs, Salesforce, SAP.

Data engineer at InfoCepts, Sep 2023 - Apr 2025 · 1 yr 8 mos, Hybrid, • Led migration from legacy Netezza to a modern Data Lakehouse and Snowflake Data Warehouse, ensuring zero data loss and zero production downtime throughout the transition. • Analyzed and documented 2,000+ Informatica PowerCenter mappings, enabling legacy system modernization, data governance initiatives, and large-scale migration planning. • Developed and maintained enterprise ETL/ELT pipelines using Informatica PowerCenter, supporting data integration, transformation, and migration across multiple business-critical systems. • Built and orchestrated automated Databricks Workflows for batch data processing, ensuring reliable and SLA-compliant data delivery to downstream applications and reporting platforms. • Optimized complex SQL, Spark SQL, and Snowflake transformations, improving query performance and processing efficiency for high-volume datasets. • Supported REST API-based data ingestion frameworks and resolved integration challenges across enterprise platforms including Oracle, Snowflake, Informatica, and Databricks. • Collaborated with cross-functional teams including business analysts, architects, and data engineers to deliver scalable data solutions and support enterprise modernization programs. Tech Stack: Snowflake, Informatica PowerCenter, Databricks, Spark SQL, SQL, Oracle, Netezza, REST APIs, ETL/ELT, Data Migration, Data Warehousing, Data Lakehouse.

Data Engineer - Intern at InfoCepts, Feb 2023 - Sep 2023 · 8 mos, On-site, • Developed and maintained ETL pipelines using Python, PySpark, and SQL across cloud platforms including GCP (BigQuery, Cloud Storage) and AWS (S3, Glue, Athena) to support data ingestion, transformation, and analytics workloads. • Built Python-based automation solutions for data preprocessing, validation, and scheduled batch processing, improving operational efficiency and reducing manual effort. • Assisted in designing and implementing scalable data processing workflows for structured datasets, gaining hands-on experience with distributed computing and cloud-native data engineering practices. • Collaborated with cross-functional teams to gather business requirements, troubleshoot pipeline issues, and deliver reliable data solutions for downstream reporting and analytics. • Contributed to data lake and data warehouse initiatives by supporting ingestion, transformation, and consumption layers within production-grade data platforms. • Worked with Databricks, PySpark, SQL, and cloud services to develop foundational expertise in large-scale data processing, ETL development, and modern data architecture. Tech Stack: Python, PySpark, SQL, GCP, BigQuery, Cloud Storage, AWS S3, AWS Glue, Athena, Databricks, Linux, Shell Scripting, ETL.

Web Developer at Pepcoding Education Private Limited, May 2022 - Jul 2022 · 3 mos, Noida, Uttar Pradesh, India, I have experience working on web development projects such as a To-Do List and a Camera Gallery Web Application. Additionally, I have developed DSA quizzes, actively participated in them, and served as a doubt solver for DSA-related queries.

Education

Bachelor of Technology - BTech, Computer Science, Greater Noida Institute of Technology(GNIOT)

XIIth, Convent Of Gagan Bharti,New Delhi