Headline

Data Engineer | Azure Data Factory | Databricks | PySpark | Spark SQL | Delta Lake | Batch Data Pipelines | OTT Streaming Analytics

About

Data Engineer with experience building and maintaining scalable batch data pipelines on Azure for large-scale OTT streaming platforms. Currently working at GlobalLogic, where I design and maintain data pipelines using Azure Data Factory, Azure Databricks, PySpark, and Spark SQL to process millions of user interaction and playback events. My work focuses on transforming high-volume event data into reliable, analytics-ready datasets using Bronze–Silver–Gold Delta Lake architecture. I have hands-on experience in building incremental data pipelines, implementing data quality checks, and optimizing Spark jobs for performance and scalability. Key skills include: • Azure Data Factory (ADF) • Azure Databricks • PySpark & Spark SQL • Delta Lake • Batch Data Pipelines • Data Quality & Validation • OTT Event Data Processing I enjoy solving complex data engineering problems and building reliable data platforms that enable analytics and data-driven decision making.

Location

Bengaluru

AI stack

gemini

hugging face

langchain

openai

Work experience

Data Engineer at GlobalLogic, Jun 2022 - Present · 4 yrs, Bengaluru, Karnataka, India, Working on data engineering solutions for OTT platforms. • Designed and maintained scalable batch data pipelines using Azure Data Factory and Databricks to process millions of OTT event records daily. • Developed PySpark and Spark SQL transformations to clean, standardize, enrich, and aggregate datasets including player events, app actions, account data, and recommendation rails. • Implemented Bronze–Silver–Gold Delta Lake architecture for raw ingestion, transformation, and analytics-ready datasets. • Built incremental data processing pipelines using partitioning, deduplication, and Delta MERGE strategies. • Implemented data quality checks including schema validation, null checks, range validation, and record reconciliation. • Optimized Spark jobs by tuning joins, partitions, and file sizes to improve pipeline performance and scalability. • Supported production pipelines by monitoring batch jobs, troubleshooting failures, and implementing safe rerun strategies. • Delivered Gold-layer datasets supporting reporting, engagement analysis, and recommendation performance tracking.

Education

Bachelor of Technology - BTech, Computer Science, REVA UNIVERSITY, BANGALORE