Soumik Roy profile photo 1 on Tal

Soumik Roy

Azure Data Engineer | 4+ Yrs | ADF | Databricks | PySpark | Reduced Spark Runtime 85% on 1M+ Records | Medallion Architecture | 8x Microsoft & Databricks Certified | Immediate Joiner

Kolkata

4+ years of work experience

Tata Consultancy Services
Dec 20214 yrs 6 mosPresent

Kolkata

• Architected and delivered a production-grade, end-to-end Azure data pipeline integrating data from Hive, Teradata, and Oracle into PostgreSQL, leveraging Azure Data Factory (ADF) and ADLS Gen2. • Designed and implemented Medallion Architecture (Bronze → Silver → Gold), enabling structured data processing: raw ingestion (Bronze), data cleansing and validation using PySpark in Azure Databricks (Silver), and business transformations with aggregated outputs for analytics (Gold). • Developed complex PySpark and Spark SQL notebooks in Databricks to standardize multi-source data, including null handling, deduplication, schema enforcement, and data type transformations, ensuring high data quality and consistency. • Built and optimized ADF pipelines using Copy Activity, loading curated Gold-layer datasets into PostgreSQL, supporting reliable and scalable downstream reporting and analytics. • Automated end-to-end data workflows (100%) using ADF event-based and scheduled triggers, enabling fully hands-free pipeline execution with zero manual intervention. • Monitored pipeline execution, performance, and failures using ADF monitoring tools, proactively resolving data quality and latency issues to ensure smooth data delivery. • Authored High-Level Design (HLD) and Low-Level Design (LLD) documentation, and led technical walkthroughs with stakeholders to ensure alignment across cross-functional teams.
Tata Consultancy Services
Jan 20221 yr 3 mosMar 2023

Bhubaneswar

• Designed and developed a modular Python-based ETL framework, leveraging a main.py orchestrator with separate modules for extraction (query.py) and loading (load.py), ensuring scalability and clear separation of concerns. • Built robust data ingestion pipelines using PySpark with JDBC connectors to extract large-scale customer and transactional data from Hive and Teradata into a unified processing layer. • Implemented advanced PySpark transformations, including window functions to identify latest customer records and complex business logic for revenue and usage metrics across multiple business segments. • Optimized large-scale joins on 1M+ customer records with 500+ attributes, utilizing broadcast joins and data skew handling techniques, significantly improving processing efficiency. • Developed a dedicated aggregation pipeline in PySpark to apply business rules on staged Hive data and load curated datasets into PostgreSQL (PgAdmin), enabling reliable reporting and analytics. • Fine-tuned Spark configurations (executor memory, cores, dynamic allocation), reducing job runtime from 6 hours to ~45–60 minutes, enabling near real-time data availability. • Delivered production-ready data pipelines and collaborated with operations teams for deployment; orchestrated end-to-end workflows using Apache NiFi, automating data flow from ingestion to final load.

Education

Academy of Technology

Bachelor of Technology - BTech, Mechanical Engineering

Academy of Technology

20174 yrs2021

Join Soumik and 10k+ other professionals on

Tal Boss

Tal is the app where bosses meet engineers & builders directly, without middlemen

Download Tal Boss

Soumik Roy on Tal

Headline

Azure Data Engineer | 4+ Yrs | ADF | Databricks | PySpark | Reduced Spark Runtime 85% on 1M+ Records | Medallion Architecture | 8x Microsoft & Databricks Certified | Immediate Joiner

About

I'm Soumik Roy, an Azure Data Engineer with 4+ years at Tata Consultancy Services, building production grade ETL pipelines, cloud-native data platforms, and large-scale transformation solutions on Microsoft Azure. I specialize in designing end-to-end data pipelines that are fast, reliable, and built to scale including optimizing a Spark job from 6 hours down to under 1 hour (~85% reduction) through advanced tuning, broadcast joins, and dynamic resource allocation. My work spans the full data lifecycle from ingesting raw data across heterogeneous sources like Hive, Teradata, and Oracle, to delivering clean, analytics-ready datasets using Medallion Architecture (Bronze → Silver → Gold) on Azure Databricks. ☁️ Cloud & Platforms: Azure Data Factory · ADLS Gen2 · Azure Databricks 🔄 Orchestration: ADF Pipelines · Apache NiFi · Event-based & Scheduled Triggers 💻 Languages: PySpark · Spark SQL · Python · SQL · Shell Scripting 🗄️ Databases: PostgreSQL · Teradata · Hive Metastore · Oracle 📊 Architecture: Medallion Architecture · ETL/ELT · Big Data · Distributed Systems Always looking to collaborate on challenging cloud data engineering problems specially around real-time processing and distributed systems. Let's connect! Feel free to DM me here.

Location

Kolkata

Work experience

System Engineer at Tata Consultancy Services, Dec 2021 - Present · 4 yrs 6 mos, Kolkata, • Architected and delivered a production-grade, end-to-end Azure data pipeline integrating data from Hive, Teradata, and Oracle into PostgreSQL, leveraging Azure Data Factory (ADF) and ADLS Gen2. • Designed and implemented Medallion Architecture (Bronze → Silver → Gold), enabling structured data processing: raw ingestion (Bronze), data cleansing and validation using PySpark in Azure Databricks (Silver), and business transformations with aggregated outputs for analytics (Gold). • Developed complex PySpark and Spark SQL notebooks in Databricks to standardize multi-source data, including null handling, deduplication, schema enforcement, and data type transformations, ensuring high data quality and consistency. • Built and optimized ADF pipelines using Copy Activity, loading curated Gold-layer datasets into PostgreSQL, supporting reliable and scalable downstream reporting and analytics. • Automated end-to-end data workflows (100%) using ADF event-based and scheduled triggers, enabling fully hands-free pipeline execution with zero manual intervention. • Monitored pipeline execution, performance, and failures using ADF monitoring tools, proactively resolving data quality and latency issues to ensure smooth data delivery. • Authored High-Level Design (HLD) and Low-Level Design (LLD) documentation, and led technical walkthroughs with stakeholders to ensure alignment across cross-functional teams.

Assistant System Engineer at Tata Consultancy Services, Jan 2022 - Mar 2023 · 1 yr 3 mos, Bhubaneswar, • Designed and developed a modular Python-based ETL framework, leveraging a main.py orchestrator with separate modules for extraction (query.py) and loading (load.py), ensuring scalability and clear separation of concerns. • Built robust data ingestion pipelines using PySpark with JDBC connectors to extract large-scale customer and transactional data from Hive and Teradata into a unified processing layer. • Implemented advanced PySpark transformations, including window functions to identify latest customer records and complex business logic for revenue and usage metrics across multiple business segments. • Optimized large-scale joins on 1M+ customer records with 500+ attributes, utilizing broadcast joins and data skew handling techniques, significantly improving processing efficiency. • Developed a dedicated aggregation pipeline in PySpark to apply business rules on staged Hive data and load curated datasets into PostgreSQL (PgAdmin), enabling reliable reporting and analytics. • Fine-tuned Spark configurations (executor memory, cores, dynamic allocation), reducing job runtime from 6 hours to ~45–60 minutes, enabling near real-time data availability. • Delivered production-ready data pipelines and collaborated with operations teams for deployment; orchestrated end-to-end workflows using Apache NiFi, automating data flow from ingestion to final load.

Education

Bachelor of Technology - BTech, Mechanical Engineering, Academy of Technology