Headline

Data engineer @ Havells India Ltd || ASE23

About

Data Engineer with 3+ years of experience in PySpark, Databricks, and Azure, building scalable batch and real-time data pipelines for high-volume IoT telemetry. Experienced in designing end-to-end data systems using Apache Spark, Kafka, Azure Event Hub, and IoT Hub, with strong focus on fault-tolerant ETL pipelines, performance optimization, and cost-efficient architecture. Proven track record of improving compute and storage efficiency, implementing Delta Lake-based recovery mechanisms, and orchestrating workflows using Airflow in distributed environments. Skilled in Docker, Kubernetes, and cloud-native data platforms, with experience collaborating across engineering, platform, and product teams to deliver reliable, production-grade data solutions. Actively shares technical insights through blogs on PySpark, Airflow, and real-world data engineering challenges.

Location

Greater Hyderabad Area

AI stack

antigravity

claude

gemini

openai

Work experience

Associate Research Engineer at Havells India Ltd, Jul 2023 - Present · 3 yrs 3 mos, Designed and implemented end-to-end batch and streaming data pipelines using PySpark, Databricks, Kafka, and Azure Event Hub to process large-scale IoT telemetry data Built fault-tolerant ETL pipelines using Delta Lake, enabling recovery from missed and late-arriving data and ensuring data completeness across processing cycles Developed and orchestrated data pipelines using Apache Airflow, including data-driven workflow triggers that reduced unnecessary executions by 70% Optimized Databricks infrastructure by migrating workloads from all-purpose clusters to job clusters, significantly reducing compute costs (₹10K+/month) Designed and deployed a containerized Airflow platform (Docker + VMSS) across environments with centralized access and integrations across Azure services Migrated real-time consumers from App Service to Azure Kubernetes Service (AKS), improving scalability, reliability, and fault tolerance Built a real-time consumer lag monitoring system for Azure Event Hub, reducing debugging and downtime analysis time by 50% Reduced telemetry data volume by 80–90% using Parquet optimization and resampling techniques, achieving ₹10K/month storage cost savings Led large-scale data migrations across Cosmos DB → MongoDB, Redis, and MySQL, ensuring data integrity, minimal downtime, and achieving ₹12K/month cost savings Collaborated closely with backend, platform, and product teams to design reliable, production-grade data systems and improve overall system performance

Data Engineering Intern at Havells India Ltd, Jan 2023 - Jul 2023 · 6 mo, Bengaluru, Karnataka, India, Built and tested real-time data pipelines using Kafka and Azure services (Event Hub, IoT Hub, Blob Storage, Service Bus, Key Vault) Led Proof-of-Concept (PoC) initiatives to evaluate new architectures and technologies, reducing implementation risk and accelerating decision-making Identified data quality issues and contributed to debugging, system design documentation, and sprint planning in Agile environments

Education

Bachelor of Technology, Computer Science, Amrita Vishwa Vidyapeetham

Intermediate, MPC, VelocIITy Junior College

Public links

medium: https://medium.com/@postvishal14