Headline

Data Engineer | Python | SQL | PySpark | Azure Data Factory | Databricks | ADLS Gen2 | ETL/ELT | Data Pipelines

About

Data Engineer with 2+ years of experience building ETL/ELT pipelines and working with data from APIs, CSV/Excel files, and business applications. I work mainly with Python, SQL, PySpark, Apache Spark, Azure Data Factory, Azure Databricks, ADLS Gen2, Delta Lake, and Snowflake. In my current role, I work on end-to-end data engineering solutions including data ingestion, transformation, data validation, reconciliation, incremental processing, and pipeline monitoring. I have also worked with Medallion Architecture (Bronze/Silver/Gold) and configuration-driven data processing for different client data structures. Before moving into Data Engineering, I worked as a Backend Engineer where I developed Python/FastAPI services, REST API integrations, and data-processing workflows using PostgreSQL and MongoDB. I enjoy solving data integration and data-quality problems and building reliable pipelines that can support analytics and downstream applications. Core skills: Python | SQL | PySpark | Apache Spark | Azure Data Factory | Azure Databricks | ADLS Gen2 | Delta Lake | Snowflake | ETL/ELT | Data Pipelines | Data Quality | REST APIs

Location

India

AI stack

antigravity

claude code

cursor

deepseek

gemini

github copilot

Work experience

Data Engineer at Excellence Integrity Management, May 2025 - Present · 1 yr 5 mos, Bengaluru, Karnataka, India, Architected and deployed an end-to-end cloud data platform for an international OTT streaming client ("Wonder"), centralizing fragmented user, content, and payment data. Key Contributions: - Built automated data ingestion pipelines using AWS Glue (Python), extracting streaming logs, viewing telemetry, and customer activities from REST APIs into Amazon S3 on a 4-hour schedule. - Automated continuous file ingestion from S3 landing buckets into Snowflake RAW staging tables leveraging Snowpipe notifications. - Implemented a 3-layer Medallion Architecture (Raw → Silver → Gold) via Snowflake Tasks and SQL to clean corrupted data types, resolve missing entity IDs, and standardize regional schemas. - Modeled an enterprise Star Schema (Fact and Dimension tables) in Snowflake, structurally resolving cross-system reconciliation mismatches between Evergent billing feeds and video activity streams. - Spearheaded a storage and performance POC evaluating Parquet columnar formats and Snappy compression against raw CSV in S3, benchmarking warehouse scanning costs and analytical query performance. - Engineered automated source-to-target data validation mechanisms (schema verification, null handling, and duplicate deduplication) to guarantee data integrity across business dashboards. Skills: Snowflake, AWS Glue, Amazon S3, Snowpipe, Snowflake Tasks, Medallion Architecture, Dimensional Modeling, Python, SQL

Back End Developer at triggerAll, Oct 2023 - Feb 2025 · 1 yr 5 mos, India, Architected and scaled an omnichannel backend engine unifying inventory, manufacturing tracking, and order fulfilment across Amazon (SP-API), Shopify, Etsy, and direct web channels. Key Contributions: - Engineered an event-driven webhook processing layer with FastAPI and PostgreSQL, processing real-time order streams with sub-50ms gateway latency. - Eliminated multi-channel race conditions and overselling by implementing pessimistic row-level locks (SELECT FOR UPDATE) in PostgreSQL for concurrent checkouts. - Designed asynchronous outbound sync workers using Celery and Redis to navigate external API rate limits (Amazon, Shopify) with exponential backoff and jitter, reducing stock propagation latency under 2 seconds. - Formulated a Bill of Materials (BOM) relational schema to automate raw material deduction (MDF sheets, acrylic, paints) and forecast production restock cycles. - Integrated pre-signed AWS S3 upload pipelines for custom engraving/photo assets, decoupling heavy payloads from transactional checkout flows. Skills Used: Python, FastAPI, PostgreSQL, Redis, Celery, AWS S3, Docker, REST APIs, System Design

Education

Bachelor of Technology - BTech, Computer Science, Biju Patnaik University of Technology, Odisha