Headline

Data Engineer III @ Walmart Global Tech India | Building Scalable Data Solutions

About

As a dedicated Software Engineer with over 4+ years of experience in Big Data, I have a proven track record of leveraging cutting-edge technologies to design and build scalable data platforms, distributed data pipelines, and real-time streaming applications with a strong focus on cloud cost efficiency. My journey in the tech industry has been fueled by a passion for transforming raw data into actionable insights, enabling organizations to make data-driven decisions through scalable, efficient, and cost-effective data processing systems. 𝗕𝗶𝗴 𝗗𝗮𝘁𝗮 𝗧𝗲𝗰𝗵𝗻𝗼𝗹𝗼𝗴𝗶𝗲𝘀: Proficient in Spark, Kafka, Apache Hudi, and Spark Streaming. 𝗣𝗿𝗼𝗴𝗿𝗮𝗺𝗺𝗶𝗻𝗴 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲𝘀: Experienced in Scala, Python, and Java. 𝗗𝗮𝘁𝗮 𝗠𝗮𝗻𝗮𝗴𝗲𝗺𝗲𝗻𝘁: Skilled in designing and implementing ETL pipelines, data warehousing solutions, and real-time data processing. 𝗖𝗹𝗼𝘂𝗱 𝗣𝗹𝗮𝘁𝗳𝗼𝗿𝗺𝘀: Adept at using AWS and GCP for deploying and managing big data solutions. 𝗗𝗮𝘁𝗮𝗯𝗮𝘀𝗲𝘀: Extensive experience with SQL and NoSQL databases, including MySQL, PostgreSQL, and MongoDB.

Location

Bengaluru

AI stack

claude

github copilot

Work experience

Data Engineer III at Walmart Global Tech India, Dec 2024 - Present · 1 yr 10 mos, Bengaluru, Karnataka, India, • Designed high-throughput data pipelines on GCP based data lakehouse using Apache Spark, Scala, and Apache Hudi, processing 6+ TB of data daily with improved scalability, reliability, and cost efficiency. • Migrated and enhanced critical batch workflows to Spark Structured Streaming integrated with Apache Kafka, improving data delivery latency by 60%. • Engineered a raw data compression utility, resulting in 25% reduction in storage costs across the organization. • Transitioned legacy 50+ workflows from Automic to Apache Airflow, improving DAG scheduling reliability and system scalability

Senior Data Engineer at Paytm, Jul 2024 - Nov 2024 · 5 mos, As a member of the Central Data Platform team, developed batch data pipelines in Spark Scala to efficiently process 25-30 TB of data daily, utilizing AWS EMR and storing outputs in AWS S3.

Software Engineer at Paytm, Jul 2022 - Jun 2024 · 2 yrs, • Working in the Central Data Platform team, developing batch data pipelines in Spark Scala that efficiently process 25-30 TB of data daily, prioritizing SLA-critical tasks. • Engineered scalable, real-time data streaming pipelines using frameworks like Apache Kafka and Spark Structured Streaming, reducing the response times from 15 minutes to under 1 minute compared to previous framework. • Enhanced and refined the architecture and code for the Merchant Settlement Report (MSR), implementing robust data quality checks to ensure the report's integrity. • Collaborated closely with cross functional teams and business users in analysing business requirements, assessing operational impacts and ensuring delivery of quality solutions. • Achieved estimated AWS cost savings ranging between $70k-$120k USD by upgrading Spark versions and optimizing daily jobs, leading to a 30%- 40% reduction in job compute time.

Research Intern at Defence Research and Development Organisation (DRDO), Jan 2021 - Jun 2021 · 6 mos, Chandigarh, India, • Worked on a drone that is used for detection of trapped people in Avalanche. • Implemented and trained Machine Learning Algorithm for Human Detection. Used YOLO object detection algorithm to implement the model. Accuracy of the model is 97.8 %.

Education

Bachelor of Technology, Electrical, Electronics and Communications Engineering, Punjab Engineering College