Headline

Software Engineer III @ Walmart | IIT Kanpur | Distributed Systems | Data Engineering | Python | Scala | Apache Spark | Airflow | GCP | Backend Engineering | CI/CD

About

I am a Software Engineer III at Walmart Global Tech with over 4 years of experience building scalable data platforms, cloud data architecture, and ETL/ELT pipelines. I specialize in distributed systems and data-intensive applications, leveraging modern data stack technologies like Scala, Apache Spark, Airflow, BigQuery, and Google Cloud Platform (GCP). My expertise lies in designing robust data modeling solutions, optimizing large-scale data processing workflows, and ensuring high data quality across enterprise data warehousing environments. I am passionate about solving complex challenges related to scalability, performance tuning, and platform reliability. Career Highlights: • Processed over 1.5 TB of daily clickstream data by building distributed ETL systems using Scala, Spark, Airflow, and BigQuery. • Reduced infrastructure costs by 53% and improved scalability by leading the migration of Spark workloads to Dataproc Serverless. • Boosted Spark application performance by 56% through executor tuning, adaptive resource allocation, and broadcast joins. • Architected historical data migration pipelines to backfill 2 years of production data for international market onboarding. In addition to cloud data engineering, I have strong backend engineering experience developing REST APIs and microservices using Java, Python, and Spring Boot. I am always eager to tackle new challenges in data architecture and distributed computing.

Location

Bengaluru

AI stack

claude

cursor

windsurf

githubcopilot

Work experience

Software Engineer III at Walmart Global Tech India, Sep 2024 - Present · 2 yrs 1 mo, Bengaluru · Hybrid, • Led migration of large-scale Spark workloads from Dataproc clusters to Dataproc Serverless across production environments, reducing infrastructure costs by 53% while improving scalability and resource utilization. • Designed and deployed scalable Spark-based data processing systems supporting customer analytics across multiple international markets, enabling production rollout for 24 large-scale datasets. • Architected historical data migration and backfill pipelines for 2 years of production data, enabling seamless onboarding of new international markets. • Improved Spark application performance by 56% through executor tuning, adaptive resource allocation, partition optimization, caching, and broadcast joins. • Consolidated 8 fragmented pipeline modules into a unified modular architecture, reducing maintenance overhead by approximately 40% and enabling independent deployments. • Designed scalable hierarchical data models to expand analytics coverage and simplify downstream data consumption. • Developed Java Spring Boot REST APIs following OpenAPI standards to support internal distributed data platforms. • Led 8–10 zero-rollback production deployments and resolved 15+ high-priority production incidents annually across multiple international deployments. • Partnered with product managers and cross-functional engineering teams to translate business requirements into scalable production solutions.

Software Engineer II at Walmart Global Tech India, Jun 2022 - Aug 2024 · 2 yrs 3 mos, Bengaluru · Hybrid, • Built distributed ETL systems processing 1.5+ TB of clickstream data daily using Scala, Apache Spark, Airflow, Hive, BigQuery, Dataproc, and Presto. • Designed scalable batch data pipelines powering customer analytics and reporting across multiple business domains. • Improved production observability by integrating Spotlight alerting and xMatters incident notifications, enabling earlier failure detection and faster incident response. • Improved engineering productivity through CI/CD automation, production validation, and end-to-end testing while maintaining 80%+ SonarQube coverage. • Built automated data validation and quality frameworks to improve reliability of large-scale production pipelines. • Migrated production pipelines and Airflow DAGs across GCP projects with zero data loss and no business disruption. • Led platform upgrades involving Spark, Dataproc, Airflow, and BigQuery connectors with zero production downtime. • Developed Java Spring Boot REST APIs and implemented controller-first API development using OpenAPI specifications.

Teaching Assistant at Indian Institute of Technology, Kanpur, Aug 2021 - May 2022 · 10 mos

Department Placement Coordinator at Students' Placement Office, IIT Kanpur, May 2021 - May 2022 · 1 yr 1 mo

Graduate Rotational Internship Program at The Sparks Foundation, Apr 2019 - Jun 2019 · 3 mos

Education

Masters of Technology-M.Tech, Geoinformatics, INDIAN INSTITUTE OF TECHNOLOGY KANPUR

Bachelor of Technology - BTech, Computer science and engineering, INSTITUTE OF ENGG. & RURAL TECHNOLOGY,ALLAHABAD