Headline

Data Engineer @ News Corp | Python, SQL, PySpark,Airflow,GCP | Google cloud certified professional data engineer

About

A Data Engineer with 4+ years of experience building scalable data pipelines, cloud data platforms, ETL/ELT workflows, and data solutions across GCP and AWS. Experienced with Python, SQL, PySpark, Spark, Airflow, BigQuery, Snowflake, Dataform, GCP, AWS, and Data Modeling, with strong expertise in data quality, data reconciliation, data governance, anomaly detection, pipeline optimization, and data migration. I have worked on large-scale data engineering initiatives including 5TB Snowflake-to-BigQuery migration with zero data loss, resolving a 95% KPI variance across enterprise reporting, improving automated data classification across 45K+ records, and building BigQuery ML-based anomaly detection solutions. I also have hands-on experience applying GenAI to data engineering, including automating SQL-to-Dataform migration using Vertex AI and Gemini, reducing migration effort by 70%. Passionate about building reliable, scalable, and production-grade data platforms and solving complex data problems through automation and technology. Core Skills: Python | SQL | PySpark | Spark | Airflow | BigQuery | Snowflake | GCP | AWS | Dataform | Data Modeling | ETL/ELT | Data Quality | Data Governance | Data Migration | Anomaly Detection | Vertex AI | Gemini

Location

Bengaluru

AI stack

claude code

github copilot

langchain

langgraph

obsidian

githubcopilot

Work experience

Data Engineer at Insight Global, Jun 2025 - Present · 1 yr 5 mos, Worked closely with business stakeholders to troubleshoot metric deviations and ensure consistent definitions across dashboards and data products. Designed and optimized end-to-end ETL pipelines to process high-volume retail and services data, improving data freshness and reliability across enterprise dashboards. Designed and implemented an anomaly detection system leveraging BigQuery ML and Cloud Functions, enabling proactive identification of irregular KPI shifts and data-quality issues.

Data Engineer at News Corp, Nov 2023 - Jun 2025 · 1 yr 8 mos, Bengaluru, Karnataka, India, ➡️ GenAI-Assisted SQL & Dataform Migration ✅ Tech Stack – Vertex AI, Gemini 1.5 Flash, Python, SQL, Git, Dataform - Automated migration of SQL scripts from Git repositories into Dataform using Vertex AI Prompt Management and Gemini 1.5 Flash. - Designed an AI-assisted workflow to accelerate SQL transformation and migration while maintaining engineering validation. - Reduced migration effort by 70% and improved overall process efficiency by 95%. ➡️ Airflow Platform Engineering & Automation ✅ Tech Stack – Python, Apache Airflow, GCP, Dataform - Developed reusable Python modules and optimized existing codebases across multiple Airflow DAGs, reducing debugging effort by 60%. - Built reusable workflow components to standardize troubleshooting and improve scalability across data pipelines. - Designed a custom Airflow operator to automate Dataform workflow execution, reducing DAG complexity by 50% and improving operational efficiency by 25%. ➡️ GCS File Processing Optimization ✅ Tech Stack – Python, Apache Airflow, GCS, GCP - Designed a custom Airflow operator to overcome the 32-file GCS composition limit using an iterative merging strategy. - Reduced file-processing time by 40% while improving pipeline robustness and reliability. - Implemented reusable logic that could be incorporated into multiple data-processing workflows. ➡️ Cloud Data Ingestion ✅ Tech Stack – GCP Data Fusion, Salesforce, Python, Airflow, SQL - Developed and deployed production-grade Data Fusion pipelines for Salesforce and database file ingestion. - Improved data processing efficiency by 30% through optimized ingestion and transformation workflows. - Built reliable cloud-based ingestion pipelines to support downstream analytics and reporting requirements.

Data Engineer at Searce Inc, Jun 2022 - Nov 2023 · 1 yr 6 mos, Bengaluru, Karnataka, India, ➡️ Snowflake → BigQuery Data Warehouse Migration ✅ Tech Stack – Snowflake, BigQuery, GCP, Python, SQL - Led the end-to-end development of an automated Snowflake-to-BigQuery migration pipeline, transferring approximately 5TB of data within two weeks. - Designed migration workflows focused on data validation, consistency, and reliability, achieving zero data loss during the platform transition. - Automated migration processes to enable a seamless transition to BigQuery while reducing manual intervention. ➡️ SQL Server → BigQuery Migration ✅ Tech Stack – SQL Server, BigQuery, GCP, Python, SQL - Designed and implemented an end-to-end solution for migrating SQL Server databases to BigQuery. - Optimized migration and data-processing workflows, reducing migration costs by 30% and improving performance by 25%. - Developed scalable cloud-based data workflows to support the transition from traditional database infrastructure to BigQuery. ➡️ Data Platform Monitoring & Alerting ✅ Tech Stack – BigQuery, Python, SQL, SMTP - Designed and implemented an automated email alerting mechanism using BigQuery and SMTP to improve data pipeline and operational monitoring. - Automated detection and notification of data and pipeline issues, improving incident response time by 50%. - Reduced dependency on manual monitoring and improved operational visibility for data engineering workflows. ➡️ Data Mesh Architecture Workshop ✅ Tech Stack – GCP, Data Mesh, Cloud Data Architecture - Ideated and prepared a Data Mesh implementation workshop focused on cloud-based data platform architecture. - Presented the proposed approach to Google Cloud leadership and received commendation for the architectural insights shared.

Intern at The Sparks Foundation, Jun 2021 - Jul 2021 · 2 mos

Education

Bachelor of Technology - BTech, Production engineering, National institute of technology Calicut

Intermediate, Sri Chaitanya College of Education