Balaji K G
Senior Software Engineer - Data @Roku | Architecting Data Lakehouse Platform | Building scalable data systems
Bengaluru
10+ years of work experience
Jan 20251 yr 8 mosPresent
Bengaluru, Karnataka, India
Feb 20223 yrsJan 2025
Chennai, Tamil Nadu, India
- Led a team in developing and implementing a scalable self service data lakehouse platform, leveraging cutting-edge technologies with spark, kafka, S3 using medallion architecture to integrate data from diverse sources, handling petabytes of scale, managing 350+ tables, orchestrated with Airflow and deployed via Github CI/CD.
- Conceptualized & architected data as a service (DaaS) solution by leveraging lakehouse platform, to enable merchants to seamlessly transfer data from CB to their lakehouse using Chargbee own Connectors and providing interface to query data using in-house Query Editor.
- Architected & implemented Event Driven Data Pipeline to process the core billing CB events including logs into lakehouse for identifying the lineage for an entity, auditing, providing historical & benchmarking analytics.
- Built an automated schema evolution solution for lakehouse table schemas using OpenAPI spec and Schema Registry, automated quality pipelines to resolve data inconsistency issues in event tables through an API-based approach.
- Involved Cost Cutting initiatives and identified efficient approach for CB Data Application Log system to have centralized approach, data archival policy using S3 Glacier which results in saving $20k per month.
Feb 20184 yrs 1 moFeb 2022
Chennai, Tamil Nadu, India
- Designed & developed high performance Recommendation Engine which enables real time Route Predictions, Weather Forecasting, Traffic Predictions, Redzone Alerts with help of Data Science team with their advanced models for enhancing truck operations resulting in 80% delivery time improvement.
- Redesigned reporting system to empower shippers to create template reports and implemented real time ad-hoc report generation, facilitated creation of >50k custom templates and >20k+ ad-hoc reports.
- Led the optimization initiative of data processing & storage layer by understanding internals of streaming applications and identifying gaps, analyzing the Query Planner for better data retrieval latency, improving >80% of overall SLA.
- Led the migration team to replicate the existing data pipelines in AWS to Azure to have a multi cloud data platform for customer needs including infrastructure provision, setting up pipelines, and monitoring & alerting systems.
- Engineered self service ingestion & processing framework with hybrid pull/push model to gather a variety of source data including 3rd party data into lakehouse within a short span of time.
Jan 20171 yr 1 moJan 2018
Chennai, Tamil Nadu, India
- Led administrative tasks for HDP clusters, optimizing performance, managing resources, and enhancing operational efficiency with 60% performance improvement & 25% reduction in security incidents.
- Established robust monitoring infrastructure by integrating Prometheus with Grafana, Pagerduty and effectively monitored streaming applications with various custom solutions, resulting in improved system stability.
- Conducted the PoC of custom data pipelines with CDC model for real time scalable processing compared with existing batch pipeline for better SLA and reducing latency for analytics.
Nov 20151 yr 3 mosJan 2017
Chennai
- Led a PoC to evaluate viability of adopting open source Pentaho ETL tool and it yielded positive outcomes, which resulted in migrating legacy in-house ETL pipeline to Pentaho.
- Successfully initiated and executed development of Enterprise Data Warehouse from inception, leveraging custom Python-based pipeline.
Balaji’s AI Stack
Education
Bachelor's degree, Electrical and Electronics Engineering
Anna University Chennai
20114 yrs2015
Join Balaji and 10k+ other professionals on
Tal is the app where bosses meet engineers & builders directly, without middlemen
Download Tal Boss