Headline

Data Engineer | PySpark, Apache Airflow, AWS, BigQuery | Batch ETL, Incremental Loads, Backfills

About

Data Engineer with 4 years of experience building and operating scalable batch ETL pipelines using PySpark, Apache Airflow, AWS, and BigQuery. Experienced in incremental loads, backfills, partitioned data models, and data quality checks. Worked on cross-cloud data integration and privacy-aware data processing to deliver reliable datasets for analytics and data science teams.

Location

Greater Bengaluru Area

AI stack

hugging face

langchain

langgraph

ollama

Work experience

Data Engineer at HiveMinds, Sep 2024 - Present · 1 yr 8 mos, Bengaluru, - Building and operating scalable batch ETL pipelines using PySpark, Apache Airflow, AWS Glue, and BigQuery. - Ingesting multi-source data from Ads platforms, CRM systems, and third-party sources, implementing incremental loads, backfills, and data quality checks. - Optimizing large analytical datasets using partitioned data models, significantly improving query performance and reducing compute costs. - Implementing privacy-aware data processing, including hashing of first-party data and automated retention policies to support secure analytics and cleanroom use cases.

Data Analyst at Enphase Energy, May 2024 - Aug 2024 · 4 mos, Bengaluru, - Worked on data pipelines and analytics for micro-inverter fleet operations using PySpark and Spark SQL. - Developed transformations on Redshift datasets using Incorta and automated batch processes in Python, reducing processing time by 75%.

Senior Data Analyst at HiveMinds, Jun 2023 - May 2024 · 1 yr, Bengaluru, Karnataka, India, - Led migration of analytics data workflows to AWS, building automated Python-based ingestion and transformation scripts to improve scalability and reduce manual effort. - Developed automation for Ads and e-commerce data processing, significantly reducing processing time and enabling reliable downstream reporting. - Collaborated with cross-functional teams to streamline data ingestion pipelines and reduce manual data collection.

Data Analyst at HiveMinds, Jun 2022 - May 2023 · 1 yr, Bangalore, - Built and automated data ingestion pipelines from Ads platforms and third-party APIs using Python and SQL. - Integrated data from Google Ads, Facebook Ads, and analytics tools into centralized data stores for reporting and analysis. - Developed Python scripts to support Ads optimization workflows and structured data preparation.

Education

Bachelor of Technology - BTech, Electrical and Electronics Engineering, National Institute of Technology warangal