Headline

Data Engineer @ Accenture | Databricks Certified | PySpark, Spark SQL, Azure & GCP | Building Lakehouse Architectures & Scalable ETL Pipelines

About

Data is only valuable when it's reliable, scalable, and trusted. I'm a Data Engineer at Accenture specializing in building modern data platforms using Databricks, PySpark, Azure, and GCP. My focus is designing scalable ETL/ELT pipelines, optimizing large-scale data processing, and creating lakehouse architectures that enable organizations to make better decisions from their data. Over the last two years, I've worked on enterprise-scale data engineering solutions for the banking and financial services domain. I've built production-grade pipelines processing 100M+ records daily, improved performance by 30–40% through Spark optimization techniques, implemented data governance frameworks with Unity Catalog, and developed reusable transformation libraries that reduced development effort by 40%. My experience spans Databricks, Delta Lake, Azure Data Factory, BigQuery, Cloud Composer, data quality frameworks, CDC, SCD Type 2, and Medallion Architecture. Beyond traditional data engineering, I'm actively exploring how modern data platforms power AI applications, RAG systems, and agentic workflows. My goal is to contribute to high-impact teams where data infrastructure is treated as a strategic asset and where I can continue building scalable, reliable, and intelligent data ecosystems. mailId- ipsitasarkar0317@gmail.com

Location

Gurugram

AI stack

cursor

hugging face

langchain

openai

Work experience

Data Engineer at Accenture in India, Oct 2024 - Present · 1 yr 9 mos, Gurugram, • Architected ETL/ELT pipelines using PySpark and Spark SQL, improving performance by 30–40% through adaptive query execution, broadcast joins, and partition-level tuning. • Designed and implemented a multi-layer Lakehouse Architecture (Landing → Bronze → Silver → Gold), ensuring data governance, lineage, and quality for banking and financial datasets. • Implemented Unity Catalog with Row-Level Security (RLS) and Column-Level Security (CLS), strengthening data access controls and supporting compliance requirements. • Built data quality gates and reconciliation frameworks across Bronze-to-Gold layers, reducing production defects by 30%. • Developed reusable PySpark transformation modules, reducing code duplication by 40% and accelerating development cycles by 2–3 weeks per sprint. • Orchestrated end-to-end data workflows using Azure Data Factory and GCP Cloud Composer, enabling seamless data movement across ADLS, BigQuery, and Cloud Storage. • Optimized SQL workloads using materialized views, Z-ordering, and clustered indexes, reducing query latency from 45 seconds to 18 seconds. • Engineered ELT workflows in BigQuery and Azure Synapse, reducing Spark compute costs by 25% and processing load by 24%. • Leveraged Delta Lake features including ACID transactions, schema evolution, and time travel to improve data reliability, auditability, and regulatory compliance. • Implemented CDC and SCD Type 2 patterns, eliminating full-table processing and reducing data pipeline overhead by 70%.

Education

Master of Computer Applications - MCA, Bharati Vidyapeeth's Institute Of Computer Applications and Management (GGSIPU)

bca, Computer Programming, Asansol Engineering College (AEC)