Headline

Data Engineer @ Tata Consultancy Services | Data Migration, Streaming Solutions

About

As a versatile Data Engineer with over a year of experience, I possess expertise in building cloud-native data platforms, real-time streaming pipelines, and big data systems. My proficiency in Python, PySpark, Spark SQL, and PL/SQL has enabled me to process 700 TB of enterprise data and architect complex migrations from legacy Oracle systems to modern Lakehouse architectures. With a strong foundation in Apache Spark, Apache Kafka, and Delta Lake, I excel in designing robust ETL/ELT pipelines and scalable ingestion workflows that meet rigorous enterprise requirements. ​I am highly skilled in both the Azure and AWS ecosystems, specializing in technologies like S3, Lambda, and Databricks. Beyond core data engineering, I am passionate about innovation in Generative AI, having developed production-grade RAG-based chatbots utilizing Ollama and vector databases. I am always seeking new opportunities to leverage my certifications in Oracle OCI GenAI and PL/SQL to contribute to cutting-edge AWS data platform projects.

Location

New Delhi

Work experience

Data System Engineer at Tata Consultancy Services, Apr 2025 - Present · 1 yr 2 mos, • Orchestrated migration of 700 TB of structured data across 16 legacy Oracle databases to a cloud-native IBM DB2 environment for a national public-sector platform. • Engineered and deployed 8,000+ DB2 tables and 2,000+ stored procedures, optimising partitions, triggers, and indexes for high-performance query execution and schema integrity. • Built end-to-end data pipelines using Azure Data Factory (ADF) to extract data from Oracle and SAP, stage in Azure Data Lake Storage (ADLS Gen2), and load curated datasets into DB2. • Developed and maintained Databricks notebooks using PySpark and Spark SQL for data transformation, enrichment, standardisation, and business rule application during migration. • Implemented Apache Kafka for high-throughput real-time data streaming, enabling seamless synchronisation between legacy Oracle systems and the target cloud architecture. • Leveraged Apache Spark (PySpark) and Hadoop/HDFS to process and store 700 TB of structured data, optimising cluster data distribution for high-volume ingestion. • Ensured data quality through rigorous validation, reconciliation, and variance analysis between source and target systems using SQL and Spark-based checks. • Led data governance activities including PII discovery, data masking, synthetic data generation, deduplication, and backfilling to meet security and compliance standards. • Utilised SQL window functions (ROW_NUMBER, LAG, LEAD) to detect duplicates, data drift, and validate incremental ETL changes during reconciliation workflows.

Data Engineer at Accenture, Jul 2024 - Oct 2024 · 4 mos, • Completed structured Data Engineering enablement program; gained hands-on experience in PySpark, Spark SQL, and Databricks through guided labs and case studies. • Developed proficiency in PL/SQL — stored procedures, functions, cursors, data validation and reconciliation on relational datasets. • Acquired foundational AWS cloud experience covering data storage concepts, cloud-based processing workflows, and security fundamentals. • Explored ETL/ELT pipeline design: source-to-target mapping, ingestion patterns, transformation logic, and data quality checks. • Gained exposure to Generative AI fundamentals and applications in data engineering contexts (metadata automation, intelligent data insights).

Education

Army Public School (APS)

Class XII, PCM, army public school