Headline

Data Engineer | Apache Spark, PySpark, Python & SQL | AWS, Databricks, Snowflake | Scalable ETL/ELT Data Pipelines

About

I build production-grade, scalable data pipelines that power critical business decisions in capital markets and automotive. At TCS, I work with Nomura's capital-markets technology team designing Spark ETL systems for trade, risk, and valuation data — processing millions of records daily across Apache Iceberg and Dremio-backed lakehouse architecture. I've reduced Spark batch runtimes by 30% through Catalyst execution-plan optimization and led the migration of Spark workloads from YARN/HDFS to Kubernetes, reducing compute costs by 35% through targeted infrastructure optimization and orchestrating daily/monthly batch schedules to feed downstream ML anomaly detection models. Previously, I developed Python-based AWS Lambda functions for Nissan to parse, clean, and validate diagnostic logs in multiple formats (CSV, JSON, Excel) uploaded to S3, automating ingestion via Step Functions into Snowflake — reducing manual data preparation by 60% and QA verification cycles by 30% by achieving 90%+ test coverage through automated testing. My core stack: Apache Spark, PySpark, Spark SQL, Spark Structured Streaming, Python, SQL, Scala, AWS, Snowflake, Databricks, Delta Lake, Apache Iceberg, Airflow, MLflow, Kubernetes, Docker. I'm particularly interested in distributed systems, data modeling, lakehouse architecture, performance optimization, and building tools that make data engineering teams more productive. Open to connecting on data engineering, distributed systems, and scalable pipeline design. Let's talk.

Location

Hyderabad

AI stack

antigravity

codex

github copilot

hugging face

kiro

langchain

Work experience

Data Engineer at Tata Consultancy Services, Jan 2024 - Present · 2 yrs 9 mos, Hyderabad, • Nomura Capital: Engineered and maintained large-scale Spark ETL pipelines using Spark SQL and complex analytical SQL queries for capital markets datasets (trades, reference, risk, and valuation feeds), involving multi-way joins and window functions to support downstream reporting and risk analytics. • Analyzed Spark execution metrics to improve execution time by 30% through broadcast joins, partition pruning, predicate pushdown, and optimized star schema joins between fact and dimension tables. • Orchestrated Spark workflows using AutoSys, managing dependency chains, reruns, and recovery logic, improving batch completion reliability and reducing manual intervention by 25%. • Executed migration of Spark workloads from YARN to Kubernetes, transitioning storage from HDFS to MinIO (S3-compatible), resolving Spark connector and storage compatibility issues. • Nissan: Architected daily batch pipelines using AWS Lambda and Step Functions, while developing a Streamlit interface for ad-hoc file ingestion and idempotent re-runs for business users. • Implemented schema validation, data quality checks, and incremental batch processing with idempotent re-runs, preparing curated datasets for Snowflake Data Warehouse ingestion and reducing manual effort by 60%. • Improved pipeline reliability via retries, implementing comprehensive monitoring and alerting using CloudWatch with automated notifications for pipeline failures, data freshness, and quality issues, reducing MTTR by 30%.

Systems Engineer at Tata Consultancy Services, Nov 2023 - Dec 2023 · 2 mos, Trivandrum, Kerala, India, • Successfully completed TCS ILP training as Digital Trainee, specializing in Java Full Stack development. • Applied acquired skills effectively in a project, demonstrating proficiency in Java Full Stack technologies. • Contributed actively to the project's development, gaining practical experience and enhancing proficiency in software engineering practices.

Education

Bachelor of Technology - BTech, Computer Science, Guru Nanak Institutions(GNI)

Public links

github: https://github.com/pavan-badempet

leetcode: https://leetcode.com/pavanbadempet

pavanbadempet.gitlab.io: https://pavanbadempet.gitlab.io/