Headline

Data Engineer | Python · PySpark · SparkSQL | Airflow · dbt · Delta Lake | AWS S3 & Athena | ETL/ELT Pipelines | Bash · SQL

About

I build production data pipelines — not prototypes. At IBM, I operate 30+ ETL/ELT pipelines that process 50 GB+ of structured and semi-structured data daily across AWS S3, Athena, and Delta Lake, maintaining 99%+ SLA compliance. I've cut pipeline MTTR by ~40% through systematic Airflow orchestration debugging, and eliminated ~8 hrs/week of manual work by building SQL validation frameworks and Python/Bash shell scripting for schema drift detection. When pipelines break at scale, I find the root cause — not just the workaround. Over two quarters, I reduced recurring failures by 30% by tracing failures across 100+ interdependent batch jobs and fixing systemic issues in dependency chains. What I work with: Python · PySpark · SparkSQL · Apache Airflow · dbt · Delta Lake · AWS (S3, Athena) · SQL · Bash/Shell · Git · Parquet What I focus on: → Batch ETL/ELT pipeline design and reliability at scale → SQL-based data validation and quality frameworks → Root-cause analysis across complex Airflow orchestration graphs → Cloud data lake architecture on AWS → Schema design and physical data modeling for semi-structured datasets Background: B.Tech in Electronics and communication Engineering with minors in Computer Science from BIT Mesra — transitioned fully into data engineering, with more than 2 years building and owning production-grade systems at IBM. Currently open to Data Engineer roles focused on scalable pipeline design, modern data stack (dbt · Airflow · Spark · Delta Lake), and cloud-native architectures on AWS.

Location

Bengaluru

AI stack

claude code

claude

github copilot

openai

claudecode

githubcopilot

Work experience

Associate System Engineer at IBM, Sep 2023 - Present · 2 yrs 10 mos, Bengaluru, Karnataka, India, · Designed, built, and maintained 30+ production ETL/ELT pipelines processing 50 GB+ of structured and semi-structured data daily on enterprise data platforms, achieving 99%+ SLA compliance across batch workloads. · Resolved failures across 100+ interdependent batch jobs by analysing Apache Airflow orchestration logs and dependency chains — reducing Mean Time to Recovery (MTTR) by ~40% through systematic root-cause analysis. · Built SQL-based data validation frameworks and Python/Bash shell scripts for schema drift detection and data quality checks, saving ~8 hrs/week of manual troubleshooting and improving pipeline observability. · Designed physical data models and managed schema evolution for semi-structured Parquet and Delta Lake datasets on AWS S3, ensuring compatibility across downstream Athena query layers. · Reduced recurring pipeline failures by 30% over two quarters by identifying and resolving systemic issues in orchestration dependency chains across 500+ daily scheduled workflow runs. · Developed SparkSQL and PySpark transformations for large-scale batch data processing on distributed compute, integrating heterogeneous structured and semi-structured data sources into unified data lake layers on AWS. · Monitored and proactively resolved bottlenecks across 500+ daily Airflow DAG runs, ensuring on-time data delivery to downstream analytics and reporting consumers. Stack: Python · PySpark · SparkSQL · Apache Airflow · AWS (S3, Athena) · SQL · Bash/Shell · Parquet · Delta Lake · dbt · Git

Education

Bachelor of Technology, Electrical, Electronics and Communications Engineering, Birla Institute of Technology, Mesra