Headline

Data Engineering Leader | Spark, Databricks, Kafka, AWS | Alumni : Thoughtworks, TCS | Databricks Certified x 4

About

I lead data engineering teams that build reliable platforms, accelerate delivery, and create measurable business impact. Over 16+ years, I’ve worked across distributed systems, data engineering, analytics platforms, and engineering leadership. My background spans hands-on platform architecture and people leadership, with experience scaling data systems, leading cross-functional programs, mentoring engineers, and building stronger execution systems for teams. Most recently at Hinge Health, I am leading a data engineering team with ownership across roadmap planning, sprint execution, hiring, and people development. Before that, as a Staff Data Engineer, I led platform modernization and optimization initiatives including a Databricks Unity Catalog migration across 1.2 PB of data, 400+ pipelines, and 10K+ objects; a Kafka migration that reduced costs by 50%; and optimization work projected to save $1M+ annually. Earlier in my career, I led teams building self-service data platforms, MLOps capabilities, real-time systems, and large-scale analytics products on AWS. Across roles, I’ve consistently enjoyed helping engineers grow, improving delivery systems, and creating the technical and organizational foundations that let teams move faster.

Location

Greater Bengaluru Area

AI stack

anthropic apis

claude code

claude

codex

cursor

gemini

Work experience

Staff Data Engineer/ Engineering Manager at Hinge Health, Feb 2024 - Present · 2 yrs 8 mos, Bengaluru, Lead a 5-person data engineering team with ownership across roadmap planning, sprint execution, hiring, people management, and delivery outcomes Established team-wide AI-assisted engineering practices and internal developer workflows to improve engineering effectiveness and encourage platform-level improvements Oversaw development of an AI-powered Spark configuration tuning workflow that reduced manual tuning effort and improved engineering leverage Led a PostgreSQL 13 to 16 major version upgrade across 38 production Aurora clusters, coordinating execution across reliability and delivery constraints Developed engineers who independently identified and drove upstream platform improvements alongside roadmap commitments Led workload right-sizing and optimization initiatives projected to save $1M annually across the data platform estate Spearheaded a zero-incident Databricks Unity Catalog migration across 1.2 PB of data, 400+ pipelines, and 10K+ objects while maintaining SLAs for critical data systems Led Kafka migration from MSK to Aiven for CDC workloads, reducing platform cost by 50% while preserving service continuity Architected and implemented a near real-time data pipeline that reduced data availability latency from days to seconds Evaluated AI-assisted approaches for data profiling and pipeline debugging to improve developer productivity and platform operations

Technical Lead (Data) at Thoughtworks, Nov 2022 - Feb 2024 · 1 yr 4 mos, Chennai, Led a distributed 5-member data platform team in a **Data Mesh paradigm**, catering to multiple lines of business Led the delivery of a developer portal for data producers and consumers, reducing infrastructure provisioning from 1 week to 3 hours Established the CI/CD process for a robust data integration framework Mentored team on PySpark optimization techniques, significantly decreasing critical job runtimes

Data Scientist at Thoughtworks, Oct 2021 - Nov 2022 · 1 yr 2 mos, Chennai, Built a predictive model for lead identification using Random Forest classifier, unlocking new revenue opportunities Designed an MLOps platform featuring CD4ML (Continuous Delivery for Machine Learning) and feature store capabilities Performed web traffic analysis using clustering to create cohorts, supporting data-led discovery for product teams Utilized data storytelling to guide product teams towards strategic high-impact bets

Data Architect at Thoughtworks, Oct 2020 - Oct 2021 · 1 yr 1 mo, Chennai, Designed the enterprise data platform from the ground up to derive strategic value from data assets Architected a commercial data product on AWS (Spark, SQL, S3), creating scalable data pipelines that generated a $20M revenue stream Led iterative development of an analytical data platform on AWS, designing datasets and data marts for GTM teams

Technical Manager at Tata Consultancy Services, Dec 2016 - Jan 2018 · 1 yr 2 mos, Greater Chennai Area, Managed delivery for a large-scale ETL application interacting across multiple sources and consumers Led development of a next-gen promotion generation and forecast application Oversaw an application ecosystem providing centralized access to all customer-facing applications

Technical Lead at Tata Consultancy Services, Aug 2014 - Dec 2016 · 2 yrs 5 mos, Mexico City, Led development for an inventory consolidation initiative using AngularJS, achieving ~500 hours of productivity savings across the retail chain Received Customer Appreciation Award for developing an instant order notification solution using Node-Webkit, which boosted CSAT scores by 700 points Redesigned pricing architecture for parallel processing, reducing nightly batch processing time by 8 hours Established a rich Single Page Application (SPA) architecture for a complex costing and pricing application

Senior Developer at Tata Consultancy Services, Feb 2009 - Aug 2014 · 5 yrs 7 mos, Chennai, Led high-impact Mainframe to Distributed platform transformations, reducing time-to-market for key services Executed a critical ADABAS Migration and handled system consolidation efforts Developed a real-time Java/Spring ETL pipeline for merchandising, providing critical updates to downstream systems Implemented Dijkstra's algorithm for automated network path provisioning, improving reliability and reducing processing time by 2 hours per order

Education

Master's degree, Computer Science, Illinois Institute of Technology

B.Tech, Information Technology, Anna University Chennai

Public links

github: https://github.com/rparthas