---
title: "Prerna Gupta - Associate System Engineer in Bengaluru | Tal"
description: "Meet Prerna Gupta, Data Engineer | Python · PySpark · SparkSQL | Airflow · dbt · Delta Lake | AWS S3 & Athena | ETL/ELT Pipelines | Bash · SQL based in Bengaluru. AI stack includes claude code, claude, and github copilot."
image: "https://tal.club/meet/prerna-8/share-image"
---

# Prerna Gupta on Tal

## Headline

Data Engineer \| Python · PySpark · SparkSQL \| Airflow · dbt · Delta Lake \| AWS S3 & Athena \| ETL/ELT Pipelines \| Bash · SQL

## About

I build production data pipelines — not prototypes\. At IBM, I operate 30\+ ETL/ELT pipelines that process 50 GB\+ of structured and semi\-structured data daily across AWS S3, Athena, and Delta Lake, maintaining 99%\+ SLA compliance\. I've cut pipeline MTTR by \~40% through systematic Airflow orchestration debugging, and eliminated \~8 hrs/week of manual work by building SQL validation frameworks and Python/Bash shell scripting for schema drift detection\. When pipelines break at scale, I find the root cause — not just the workaround\. Over two quarters, I reduced recurring failures by 30% by tracing failures across 100\+ interdependent batch jobs and fixing systemic issues in dependency chains\. What I work with: Python · PySpark · SparkSQL · Apache Airflow · dbt · Delta Lake · AWS \(S3, Athena\) · SQL · Bash/Shell · Git · Parquet What I focus on: → Batch ETL/ELT pipeline design and reliability at scale → SQL\-based data validation and quality frameworks → Root\-cause analysis across complex Airflow orchestration graphs → Cloud data lake architecture on AWS → Schema design and physical data modeling for semi\-structured datasets Background: B\.Tech in Electronics and communication Engineering with minors in Computer Science from BIT Mesra — transitioned fully into data engineering, with more than 2 years building and owning production\-grade systems at IBM\. Currently open to Data Engineer roles focused on scalable pipeline design, modern data stack \(dbt · Airflow · Spark · Delta Lake\), and cloud\-native architectures on AWS\.

## Location

Bengaluru

## AI stack

claude code

claude

github copilot

openai

claudecode

githubcopilot

## Work experience

Associate System Engineer at IBM, Sep 2023 \- Present · 2 yrs 10 mos, Bengaluru, Karnataka, India, · Designed, built, and maintained 30\+ production ETL/ELT pipelines processing 50 GB\+ of structured and semi\-structured data daily on enterprise data platforms, achieving 99%\+ SLA compliance across batch workloads\. · Resolved failures across 100\+ interdependent batch jobs by analysing Apache Airflow orchestration logs and dependency chains — reducing Mean Time to Recovery \(MTTR\) by \~40% through systematic root\-cause analysis\. · Built SQL\-based data validation frameworks and Python/Bash shell scripts for schema drift detection and data quality checks, saving \~8 hrs/week of manual troubleshooting and improving pipeline observability\. · Designed physical data models and managed schema evolution for semi\-structured Parquet and Delta Lake datasets on AWS S3, ensuring compatibility across downstream Athena query layers\. · Reduced recurring pipeline failures by 30% over two quarters by identifying and resolving systemic issues in orchestration dependency chains across 500\+ daily scheduled workflow runs\. · Developed SparkSQL and PySpark transformations for large\-scale batch data processing on distributed compute, integrating heterogeneous structured and semi\-structured data sources into unified data lake layers on AWS\. · Monitored and proactively resolved bottlenecks across 500\+ daily Airflow DAG runs, ensuring on\-time data delivery to downstream analytics and reporting consumers\. Stack: Python · PySpark · SparkSQL · Apache Airflow · AWS \(S3, Athena\) · SQL · Bash/Shell · Parquet · Delta Lake · dbt · Git

## Education

Bachelor of Technology, Electrical, Electronics and Communications Engineering, Birla Institute of Technology, Mesra

[View the original profile](https://tal.club/meet/prerna-8)

```json
{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "Person",
      "@id": "https://tal.club/meet/prerna-8#person",
      "name": "Prerna Gupta",
      "url": "https://tal.club/meet/prerna-8",
      "image": "https://tal.club/meet/prerna-8/profile-photo.jpg",
      "description": "Meet Prerna Gupta, Data Engineer | Python · PySpark · SparkSQL | Airflow · dbt · Delta Lake | AWS S3 & Athena | ETL/ELT Pipelines | Bash · SQL based in Bengaluru. AI stack includes claude code, claude, and github copilot.",
      "jobTitle": "Associate System Engineer",
      "worksFor": {
        "@type": "Organization",
        "name": "IBM"
      },
      "hasOccupation": {
        "@type": "Occupation",
        "name": "Associate System Engineer"
      },
      "homeLocation": {
        "@type": "Place",
        "name": "Bengaluru"
      },
      "alumniOf": [
        {
          "@type": "EducationalOrganization",
          "name": "Birla Institute of Technology, Mesra"
        }
      ],
      "knowsAbout": [
        "claude code",
        "claude",
        "github copilot",
        "openai",
        "claudecode",
        "githubcopilot"
      ],
      "mainEntityOfPage": {
        "@id": "https://tal.club/meet/prerna-8#webpage"
      }
    },
    {
      "@type": "ProfilePage",
      "@id": "https://tal.club/meet/prerna-8#webpage",
      "url": "https://tal.club/meet/prerna-8",
      "name": "Prerna Gupta on Tal",
      "description": "Meet Prerna Gupta, Data Engineer | Python · PySpark · SparkSQL | Airflow · dbt · Delta Lake | AWS S3 & Athena | ETL/ELT Pipelines | Bash · SQL based in Bengaluru. AI stack includes claude code, claude, and github copilot.",
      "dateModified": "2026-10-08T16:18:05.473Z",
      "isPartOf": {
        "@id": "https://tal.club/#website"
      },
      "mainEntity": {
        "@id": "https://tal.club/meet/prerna-8#person"
      },
      "primaryImageOfPage": {
        "@type": "ImageObject",
        "url": "https://tal.club/meet/prerna-8/profile-photo.jpg",
        "contentUrl": "https://tal.club/meet/prerna-8/profile-photo.jpg",
        "caption": "Prerna Gupta profile photo on Tal"
      },
      "inLanguage": "en-IN"
    }
  ]
}
```