Headline

Senior Data Engineer · Distributed Systems · AWS · Databricks · PySpark · Kafka · Agentic AI

About

Data engineer with 3+ years of owning data systems end-to-end — from architecture decisions to production reliability. I've worked on problems at genuine scale: migrating 3,000+ legacy pipelines to AWS Cloud by designing an AI-agent-based automation system that eliminated manual code translation entirely, building real-time streaming platforms processing 2M+ events daily with 100% data accuracy, and engineering a Graph Theory-based deduplication system that made customer data trustworthy enough to power business-critical reporting and ML models. I care about building things that don't break — pipelines with embedded data quality checks, CI/CD workflows that cut deployment failures, and architectures that other engineers can actually maintain and extend. Always up for conversations about data architecture, system reliability, or what's actually moving the needle in the modern data stack.

Location

Dubai

AI stack

anthropic apis

claude code

cursor

gemini

perplexity

Work experience

Data Engineer - II at OneByZero, Apr 2026 - Present · 6 mos, Bengaluru, Migrating data platform from on-prem to cloud using Agentic AI

Data Engineer at OneByZero, Mar 2024 - Apr 2026 · 2 yrs 2 mos, Bengaluru, - Solving Complex Data Problems using modern data stack tools. - Consulting Clients across multiple domains and providing them the best solution. Spearheaded cloud migration initiative, transitioning legacy on-premise Talend pipelines and UNIX jobs to Databricks, slashing operational overhead by 30% and boosting billing system performance by 50% - Architected and deployed a containerized data anonymization pipeline using Docker, AWS Lambda, and Python with SHA-256 hashing, securing 1M+ sensitive customer records while maintaining compliance standards - Engineered high-performance ETL pipelines on Databricks and Apache Spark, reducing analytics latency by 40% through optimized batch processing and Delta Live Tables implementation - Optimized Spark jobs for petabyte-scale datasets, cutting runtime by 35% through strategic partitioning, caching, and performance tuning techniques - Pioneered GenAI-powered SQL dialect converter, accelerating migration speed by 80% and eliminating manual conversion bottlenecks across heterogeneous database systems - Implemented CI/CD automation using GitHub Actions and Infrastructure as Code, reducing deployment errors by 25% and streamlining release cycles - Designed real-time data pipeline leveraging Apache Kafka for ingestion with downstream Spark processing, enabling live analytical reports with minimal latency - Built comprehensive data quality framework with automated validation checks, improving pipeline reliability by 35% and enhancing data governance - Delivered stakeholder-driven data solutions aligned with business KPIs, improving delivery efficiency by 20% through proactive collaboration and requirements gathering

Big Data Engineer at Celebal Technologies, Feb 2023 - Feb 2024 · 1 yr 1 mo, Noida, Architected end-to-end IoT data pipeline connecting 500+ devices via Azure Event Hub to Databricks, processing 2M+ real-time events daily through Delta Live Tables with 99.8% data accuracy.

Education

Bachelor of Technology, Information Technology, Poornima college of engineering

12, St. Francis School, Anpara

Public links

github: https://github.com/Shaswat19

medium: https://medium.com/@shaswatp19