Headline

Data Engineer @ Forma.ai | PySpark, Databricks, SQL Architecture, AWS | Building Scalable Distributed Data Pipelines & Cloud Infrastructure

About

High-performing Data & Analytics Engineer with hands-on experience building high-throughput batch transformations, distributed Big Data pipelines, and cloud lakehouse architectures. Selected as the sole candidate across the entire university graduating batch for the core Analytics Engineering track at Forma.ai. I specialize in tackling complex, scale-driven data platform challenges—from refactoring legacy processing tasks into distributed PySpark/Databricks execution plans to designing pre-aggregated SQL cache architectures that cut dashboard queries from minutes to milliseconds. Core Technical Expertise: - *Languages & Data Engineering:* Python (PySpark, Pandas), SQL (PostgreSQL, Spark SQL, ANSI), TypeScript, C/C++ - *Distributed Big Data Platforms:* Apache Spark (Execution & Memory Tuning, Partitioning), Databricks Lakehouse - *ETL/ELT Pipelines:* Blueprint v3 Architecture, PythonNodes, Nested Map/Array Transformations (explode, flatten, transform), Automated Validation - *SQL & Data Modeling:* Multi-Tier CTEs, Window Functions, Dimensional Modeling, Star/Snowflake Schemas - *Backend & Cloud Infrastructure:* Django (Multi-Tenant Systems), Redis, Celery, AWS S3, AWS ECR, Docker, Git/GitHub Always open to connecting with engineers, tech leaders, and recruiters in the Big Data, Data Platform, and Analytics Engineering space! *Email:* spreehagarg@gmail.com

Location

Pune City

AI stack

claude code

claude

codex

notion

openai

ChatGPT

Work experience

Data Engineer at Forma.ai, Jul 2026 - Present · 4 mos, - Distributed Pipeline Optimization: Refactored single-threaded data transformation scripts into distributed PySpark jobs, reducing execution runtimes by over *45%* across multi-GB enterprise datasets. - Databricks Lakehouse Engineering: Built production PySpark notebooks and Spark SQL managed tables to power automated enterprise incentive compensation engines. - Nested Schema Transformations: Architected PySpark workflows using map_values, map_filter, explode, and array_distinct to flatten deeply nested JSON data structures. - Pipeline Engine Extensions: Engineered custom PythonNodes leveraging RunContext within Forma’s proprietary Blueprint v3 execution engine to handle dynamic DataFrame state propagation. - Multi-Tier SQL Analytics Modeling: Constructed modular SQL pipeline architectures using multi-level CTEs, window functions, and time-based rollups to generate YTD attainment and quota metrics. - Performance & Cache Acceleration: Designed pre-aggregated SQL cache tables, optimizing query execution paths and reducing platform dashboard response latencies from minutes to milliseconds. - Data Reliability Frameworks: Implemented automated schema validation layers across ETL pipeline stages, eliminating recurring runtime failures and data drift issues. - Production Debugging & Hotfixes: Conducted root-cause stack trace analysis across PySpark, SQL, and Python layers to issue rapid commit-level hotfixes and perform opsval-to-prod environment restorations. - Cloud & Platform Infrastructure: Configured AWS S3 bucket lifecycles and containerized microservices using Docker, Redis task brokers, and Celery asynchronous workers.

Engineer Intern at Forma.ai, Jan 2026 - Jun 2026 · 6 mos

Summer Intern at Honda Cars India Pvt. Ltd., May 2024 - Jul 2024 · 3 mos, Gautam Buddha Nagar, Uttar Pradesh, India, - Executed functional and performance testing on the beta release of Honda Connect telematics software, validating edge-to-cloud telemetry transmission. - Conducted in-depth architectural research on Telematics Control Units (TCU) and evaluated enterprise frameworks for vehicular data ingestion and monetization. - Applied QCDM (Quality, Cost, Delivery, Management) methodologies to automate corporate workflows and streamline system testing procedures.

Education

Bachelor of Technology - BTech, Electrical and Electronics Engineering, Manipal institute of technology

Senior Secondary, Non- medical, Queen's Valley School