Headline

Data Engineer | Databricks, DP-600, DP-700, GCP, SnowPro Associate: Platform Certified | Python, PySpark, SQL, SSIS, SSRS, Fabric, PowerBI, Data Modelling, Snowflake, DataBricks, AWS, dbt

About

​Most data pipelines break under production scale I build the resilient, zero-data-loss infrastructure that keeps enterprise systems running. ​As a Data Engineer at Endava with 5 industry certifications, I specialize in engineering high-throughput ETL/ELT pipelines, multi-cloud lakehouses, and real-time streaming architectures across AWS, Databricks, Snowflake, and Microsoft Fabric. My focus centers on building fault-tolerant systems handling multi-terabyte workloads with 99.9%+ pipeline uptime, complete data quality guarantees, and sub-second query performance. ​Core Capabilities & Ecosystem Expertise ​Lakehouse & Data Warehousing: Architecting multi-layer Medallion platforms (Bronze/Silver/Gold) using Delta Lake, Apache Iceberg, Snowflake, and Microsoft Fabric to process 10M+ daily records. ​High-Performance Pipeline Engineering: Building metadata-driven batch and incremental pipelines across 5 formats (CSV, JSON, JSONL, Parquet, Excel) using PySpark, SQL, AWS Glue, Snowpipe, Streams/Tasks, and dbt. ​Data Governance & Observability: Implementing 10+ operational reliability controls, including automated CDC, SCD Types 1/2, source-to-target reconciliation, quarantine handling, and 100% idempotent rerun capabilities. ​Key Technical Engineering Milestones ​Enterprise AWS Lakehouse Accelerator: Integrated 15+ AWS and open-source engine components (Spark, Iceberg, Polaris, Trino, dbt) into a 3-layer lakehouse architecture; eliminated 100% of metadata loss across EC2 container restarts via JDBC-backed PostgreSQL state persistence. ​Retail360 Analytics Platform: Engineered a 6-layer RAW-to-MART architecture automating 23 Snowpipe streaming channels, Streams/Tasks, and Dynamic Tables reducing data freshness latency from hours to under 5 minutes. ​Enterprise AI Observability Lakehouse: Designed metadata-driven PySpark pipelines on Databricks with schema enforcement, automated duplicate purge, and 5 curated analytics domains powering real-time model evaluation and cost tracking. ​Certifications: Databricks Certified Data Engineer Associate | Databricks Certified Data Analyst Associate | Microsoft Certified: Fabric Data Engineer Associate (DP-700) | Microsoft Certified: Fabric Analytics Engineer Associate (DP-600) | Snowflake SnowPro Associate Platform ​Interested in scaling modern data infrastructure or discussing cloud data architectures? Feel free to reach out directly or send a connection request!

Location

Bengaluru

AI stack

codex

gemini

geminicli

github copilot

ollama

openai

Work experience

Data Engineer at Endava India, Aug 2026 - Sep 2026 · 2 mos, Bengaluru, Karnataka, India, • Engineered scalable data solutions utilizing Databricks, PySpark, AWS, Python, and SQL to support diverse business domains, improving pipeline processing efficiency by 30%. • Executed Delta Lake architectures and Medallion data concepts, employing Delta MERGE, CDC, and SCD Type 1/2 to process over 100 GB of incremental data daily. • Built reusable batch and ETL/ELT pipelines involving schema validation, cleansing, deduplication, and metadata enrichment for 50+ structured and semi-structured datasets. • Investigated and resolved 200+ data quality anomalies using SQL-based source-to-target reconciliation, exception handling, and root-cause analysis, reducing reporting discrepancies by 35% across cloud platforms. • Modeled fact and conformed dimension tables using Star Schema principles to produce optimized semantic models, accelerating query performance for downstream analytics by 40%. • Utilized Git for collaborative development, version control, and CI/CD best practices across 30+ database DDLs and Python transformations

Junior Data Engineer at Endava India, May 2026 - Aug 2026 · 4 mos, Bengaluru, Karnataka, India, Designed and developed scalable ETL/ELT pipelines supporting analytics, reporting, and warehouse workloads across insurance, retail, customer analytics, and data platform use cases. Engineered ingestion and transformation workflows using SQL, PySpark, SSIS, Microsoft Fabric, and Snowflake, supporting structured and semi-structured datasets. Built reusable pipeline components with historical + incremental loading, improving maintainability and reducing repeated transformation effort. Implemented data quality validation, schema checks, audit logging, duplicate detection, reject handling, and edge-case processing to improve trust and operational reliability. Applied CDC and SCD Type 2 techniques for historical tracking and business entity versioning across warehouse layers. Designed warehouse-ready datasets using dimensional modeling (Fact & Dimension), semantic layer practices, and reporting optimization. Contributed to pipeline orchestration, observability, cost-aware execution, and performance tuning through optimized joins, partitioning, and transformation strategies. Delivered analytics-ready datasets and reporting layers consumed through Power BI dashboards, KPIs, and business reporting workflows.

Associate Developer at Endava India, Aug 2025 - May 2026 · 10 mos, Bengaluru, Karnataka, India

Data Engineering Intern at GalaxE an Endava Company, Jan 2025 - Apr 2025 · 4 mos, Bangalore Urban, Karnataka, India, • Developed data ingestion workflows and SQL transformations to process 50+ weekly file-based datasets (CSV, Excel) into centralized relational databases, eliminating 15 hours of manual data entry per week. • Collaborated with senior engineers to troubleshoot 30+ failed pipeline loads, validate business rules, and enforce error redirection and logging protocols. • Documented repea

Student Intern at Voltix Solution, Oct 2023 - Nov 2023 · 2 mos, Build A Secure Portal with React.Js And Express.Js, Allowing Admin To Manage Projects, Services, Testimonials, And Social Media Feeds.

Student Intern at All India Council for Technical Education (AICTE), Jul 2022 - Sep 2022 · 3 mos, Karnataka, India, Excited to have completed my internship with the IKS Division, Ministry of Education, conducted by AICTE. Grateful for the opportunity to contribute to meaningful projects and learn from esteemed mentors. Looking forward to applying the skills and knowledge gained to future endeavors !@AICTE #InternshipExperience #AICTE #MinistryofEducation #IKSDivision

Education

Bachelor of Engineering - BE, Information science, Sahyadri College of Engineering & Management

Science, Premier S&C PU College

Public links

github: https://github.com/techsoniya

leetcode: https://leetcode.com/techsoniya

youtube.com: https://youtube.com/@soniyavinaykumarkambli