Headline

Immediate Joiner | AWS Data Engineer | PySpark | Apache Spark | AWS EMR | S3 | Step Functions | Snowflake | Hive | SQL | Data Lakehouse | ETL Pipelines

About

Immediate Joiner AWS Data Engineer with nearly 3 years of experience building scalable data pipelines and distributed data processing systems on AWS. Experienced in designing and optimizing large-scale ETL pipelines processing 2–3 TB of daily data across Amazon S3, Snowflake, and REST APIs using PySpark and Apache Spark on AWS EMR clusters. Strong experience in building Data Lake architectures using Medallion (Bronze → Silver → Gold) architecture, enabling reliable data ingestion, transformation, and analytics for downstream consumers using Amazon Athena. Key expertise includes: • Distributed data processing using PySpark and Apache Spark • Building scalable ETL pipelines on AWS EMR clusters • Designing S3-based Data Lake architectures • Data ingestion from APIs, Snowflake, and cloud storage • Spark performance optimization (broadcast joins, partition pruning, executor tuning) • Handling production challenges such as data skew, schema evolution, and pipeline failures • Workflow orchestration using AWS Step Functions In my recent role, I designed and maintained distributed PySpark pipelines on a 30-node AWS EMR cluster processing up to 2–3 TB of daily telecom and analytics data. I optimized Spark workloads which reduced pipeline runtime by more than 2 hours while maintaining SLA-based batch processing reliability. I am currently open to new opportunities and available to join immediately for Data Engineer roles focused on AWS, Spark, and large-scale data platforms.

Location

Pune/Pimpri-Chinchwad Area

AI stack

antigravity

claude

opencode

Work experience

Data Engineer at Microline informatiion systems, Mar 2023 - Dec 2025 · 2 yrs 10 mos, Hyderabad, • Designed and maintained distributed PySpark ETL pipelines processing ~2–3 TB of daily data across multiple sources including Amazon S3, Snowflake tables, and 7 REST APIs. • Built scalable data processing workloads on a 30-node AWS EMR cluster to handle high-volume telecom and analytics datasets. • Implemented Medallion-style Data Lake architecture (Bronze → Silver → Gold) on Amazon S3 enabling reliable downstream analytics through AWS Athena. • Developed source-specific ingestion pipelines and a master aggregation pipeline to transform and consolidate intermediate datasets into curated business-ready outputs. • Optimized Spark jobs using broadcast joins, anti-joins, partition pruning, and executor memory tuning, reducing pipeline runtime by more than 2 hours. • Resolved production data skew issues during heavy shuffle stages using salting techniques and optimized Spark partition strategies. • Automated end-to-end workflow orchestration using AWS Step Functions including EMR cluster provisioning, job execution, monitoring, and failure recovery. • Implemented secure Snowflake integration using AWS Secrets Manager ensuring credential security and pipeline reliability. • Ensured SLA adherence for daily batch pipelines with a 4–5 hour execution window and ~99% production reliability. Key Projects: Airtel – Telecom Billing & CDR Data Platform Built large-scale PySpark pipelines on AWS EMR processing high-volume Call Detail Records (CDR) and telecom billing datasets. Designed S3-based Bronze → Silver → Gold data lake architecture and optimized heavy Spark transformations. OYO Workspaces – Customer & Booking Analytics Pipeline Developed PySpark ETL pipelines integrating data from S3, Snowflake, and REST APIs to support customer behavior analytics and pricing insights. Optimized Spark workloads and stored curated datasets in Parquet for Athena-based analytics.