Sanchit Goel profile photo 1 on Tal

Sanchit Goel

Data Engineer III at BharatPe | Spark | GCP | Python | Bigdata | Airflow

West Delhi

6+ years of work experience

BharatPe
Apr 20251 yr 6 mosPresent

Gurugram

• Led end-to-end data platform migration, rebuilding pipelines and datasets from scratch with zero downtime while working closely with business and tech teams. • Designed and maintained a large-scale recommendation pipeline for 4Cr merchants (70L active), computing daily ranked recommendations and delivering location-based merchant suggestions to FSEs across regions • Designed and implemented a fault-tolerant event ingestion pipeline consuming multi-source callbacks (call centers, marketing tools, banks) via Kafka into BigQuery, ensuring zero data loss and high reliability at scale • Built and scaled data pipelines using BigQuery, Dataflow, and Airflow, supporting high-volume, near real-time analytics across multiple business verticals. • Developed modular data models using dbt, enabling faster analytics, reuse, and consistent business metrics across teams. • Drove performance and reliability improvements, optimizing queries, automating pipelines, and upgrading observability (41 APIs) to support rapid iteration in a fast-paced environment. • Built a declarative ETL platform on GCP integrating multiple data sources, with Jinja-based DAG auto-generation and validation layers, enabling self-serve pipeline onboarding and significantly reducing engineering effort for new data integrations.
BharatPe
Mar 20241 yr 2 mosApr 2025

On-site

Testbook
Jul 20211 yr 9 mosMar 2023

Mumbai, Maharashtra, India

Testbook
Feb 20213 yrs 2 mosMar 2024

Mumbai

• Aided the business team by developing batch and real-time ETL pipelines using Apache-Airflow, GCP cloud functions, and Pub/Sub for generating leads on CRM, which is responsible for 30% of the revenue. • Designed a complex framework for data extraction from multiple data sources into a centralized data lake using EL and ETL pipelines using Airflow, pyspark, and Kafka. • Streamlined manual workflows by leveraging expertise in Airflow, Python, and shell scripting; automated processes, resulting in a 50% reduction in time spent on repetitive tasks and enabling the team to focus on strategic initiatives. • Developed a robust REST API using Django REST framework and Redis to serve personalized banners on app and web, enhancing the user experience and driving engagement. • Built a class scheduling system, helping the operations team to efficiently use the availability of faculty and recording studios using Django REST framework. • Developed real-time reporting Dashboards on  Redash, empowering the business team to leverage data-driven insights for effective decision-making. • Leveraged Mongo oplogs to construct a unified database, consolidating data from multiple MongoDbs. This streamlined data retrieval through a single query, resulting in significant time and effort savings. • Enhanced engagement levels by utilizing historical data analysis to identify students' areas of weakness, enabling targeted subject preparation.
Testbook
Feb 20216 mosJul 2021

Sapio Analytics
Jul 20206 mosDec 2020

New Delhi, Delhi, India

Sapio Analytics
Jul 20206 mosDec 2020

Implemented a robust CI/CD pipeline utilizing Jenkins, automating the software development lifecycle and promoting efficient and reliable deployment processes. Achieved a 15% cost reduction for AWS EC2 instances by implementing automated downtime schedules for testing and developer environments during non-working hours, optimizing resource allocation, and maximizing cost efficiency. Utilized serverless technologies such as AWS Lambda to deploy microservices, resulting in a remarkable 30% cost reduction and 100% uptime.
GlobalLogic
May 20193 mosJul 2019

Noida, Uttar Pradesh, India

Internity Foundation
Dec 20182 mosJan 2019

New Delhi, Delhi, India

Sanchit’s AI Stack

claude code
claude code
claude
claude
cursor
cursor
gemini
gemini
github copilot
github copilot

Education

International Institute of Information Technology Bangalore

Postgraduate Degree, Data Engineering

International Institute of Information Technology Bangalore

20212 yrs2023
jaypee institute of information technology

Bachelor of Technology, Computer Science

jaypee institute of information technology

20174 yrs2021

Join Sanchit and 10k+ other professionals on

Tal Boss

Tal is the app where bosses meet engineers & builders directly, without middlemen

Download Tal Boss

Sanchit Goel on Tal

Headline

Data Engineer III at BharatPe | Spark | GCP | Python | Bigdata | Airflow

About

Experienced Data Engineer with over 5+ years of expertise in designing and building robust data pipelines, scalable APIs, and automation solutions. Proficient in leveraging cutting-edge technologies and frameworks to extract, transform, and load data from various sources, ensuring its accuracy, availability, and reliability for analytics and business intelligence purposes.

Location

West Delhi

AI stack

claude code

claude

cursor

gemini

github copilot

Work experience

Data Engineer III at BharatPe, Apr 2025 - Present · 1 yr 6 mos, Gurugram, • Led end-to-end data platform migration, rebuilding pipelines and datasets from scratch with zero downtime while working closely with business and tech teams. • Designed and maintained a large-scale recommendation pipeline for 4Cr merchants (70L active), computing daily ranked recommendations and delivering location-based merchant suggestions to FSEs across regions • Designed and implemented a fault-tolerant event ingestion pipeline consuming multi-source callbacks (call centers, marketing tools, banks) via Kafka into BigQuery, ensuring zero data loss and high reliability at scale • Built and scaled data pipelines using BigQuery, Dataflow, and Airflow, supporting high-volume, near real-time analytics across multiple business verticals. • Developed modular data models using dbt, enabling faster analytics, reuse, and consistent business metrics across teams. • Drove performance and reliability improvements, optimizing queries, automating pipelines, and upgrading observability (41 APIs) to support rapid iteration in a fast-paced environment. • Built a declarative ETL platform on GCP integrating multiple data sources, with Jinja-based DAG auto-generation and validation layers, enabling self-serve pipeline onboarding and significantly reducing engineering effort for new data integrations.

Data Engineer II at BharatPe, Mar 2024 - Apr 2025 · 1 yr 2 mos, On-site

Data Engineer at Testbook, Jul 2021 - Mar 2023 · 1 yr 9 mos, Mumbai, Maharashtra, India

Data Engineer II at Testbook, Feb 2021 - Mar 2024 · 3 yrs 2 mos, Mumbai, • Aided the business team by developing batch and real-time ETL pipelines using Apache-Airflow, GCP cloud functions, and Pub/Sub for generating leads on CRM, which is responsible for 30% of the revenue. • Designed a complex framework for data extraction from multiple data sources into a centralized data lake using EL and ETL pipelines using Airflow, pyspark, and Kafka. • Streamlined manual workflows by leveraging expertise in Airflow, Python, and shell scripting; automated processes, resulting in a 50% reduction in time spent on repetitive tasks and enabling the team to focus on strategic initiatives. • Developed a robust REST API using Django REST framework and Redis to serve personalized banners on app and web, enhancing the user experience and driving engagement. • Built a class scheduling system, helping the operations team to efficiently use the availability of faculty and recording studios using Django REST framework. • Developed real-time reporting Dashboards on Redash, empowering the business team to leverage data-driven insights for effective decision-making. • Leveraged Mongo oplogs to construct a unified database, consolidating data from multiple MongoDbs. This streamlined data retrieval through a single query, resulting in significant time and effort savings. • Enhanced engagement levels by utilizing historical data analysis to identify students' areas of weakness, enabling targeted subject preparation.

Software Development Intern at Testbook, Feb 2021 - Jul 2021 · 6 mos

Cloud Intern at Sapio Analytics, Jul 2020 - Dec 2020 · 6 mos, New Delhi, Delhi, India

Cloud Engineer at Sapio Analytics, Jul 2020 - Dec 2020 · 6 mos, Implemented a robust CI/CD pipeline utilizing Jenkins, automating the software development lifecycle and promoting efficient and reliable deployment processes. Achieved a 15% cost reduction for AWS EC2 instances by implementing automated downtime schedules for testing and developer environments during non-working hours, optimizing resource allocation, and maximizing cost efficiency. Utilized serverless technologies such as AWS Lambda to deploy microservices, resulting in a remarkable 30% cost reduction and 100% uptime.

Internship Trainee at GlobalLogic, May 2019 - Jul 2019 · 3 mos, Noida, Uttar Pradesh, India

Winter Intern at Internity Foundation, Dec 2018 - Jan 2019 · 2 mos, New Delhi, Delhi, India

Education

Postgraduate Degree, Data Engineering, International Institute of Information Technology Bangalore

Bachelor of Technology, Computer Science, jaypee institute of information technology