---
title: "Soumik Roy - System Engineer in Kolkata | Tal"
description: "Meet Soumik Roy, Azure Data Engineer | 4+ Yrs | ADF | Databricks | PySpark | Reduced Spark Runtime 85% on 1M+ Records | Medallion Architecture | 8x Microsoft & Databricks Certified | Immediate Joiner based in Kolkata."
image: "https://tal.club/meet/soumik-roy/share-image"
---

# Soumik Roy on Tal

## Headline

Azure Data Engineer \| 4\+ Yrs \| ADF \| Databricks \| PySpark \| Reduced Spark Runtime 85% on 1M\+ Records \| Medallion Architecture \| 8x Microsoft & Databricks Certified \| Immediate Joiner

## About

I'm Soumik Roy, an Azure Data Engineer with 4\+ years at Tata Consultancy Services, building production grade ETL pipelines, cloud\-native data platforms, and large\-scale transformation solutions on Microsoft Azure\. I specialize in designing end\-to\-end data pipelines that are fast, reliable, and built to scale including optimizing a Spark job from 6 hours down to under 1 hour \(\~85% reduction\) through advanced tuning, broadcast joins, and dynamic resource allocation\. My work spans the full data lifecycle from ingesting raw data across heterogeneous sources like Hive, Teradata, and Oracle, to delivering clean, analytics\-ready datasets using Medallion Architecture \(Bronze → Silver → Gold\) on Azure Databricks\. ☁️ Cloud & Platforms: Azure Data Factory · ADLS Gen2 · Azure Databricks 🔄 Orchestration: ADF Pipelines · Apache NiFi · Event\-based & Scheduled Triggers 💻 Languages: PySpark · Spark SQL · Python · SQL · Shell Scripting 🗄️ Databases: PostgreSQL · Teradata · Hive Metastore · Oracle 📊 Architecture: Medallion Architecture · ETL/ELT · Big Data · Distributed Systems Always looking to collaborate on challenging cloud data engineering problems specially around real\-time processing and distributed systems\. Let's connect\! Feel free to DM me here\.

## Location

Kolkata

## Work experience

System Engineer at Tata Consultancy Services, Dec 2021 \- Present · 4 yrs 6 mos, Kolkata, • Architected and delivered a production\-grade, end\-to\-end Azure data pipeline integrating data from Hive, Teradata, and Oracle into PostgreSQL, leveraging Azure Data Factory \(ADF\) and ADLS Gen2\. • Designed and implemented Medallion Architecture \(Bronze → Silver → Gold\), enabling structured data processing: raw ingestion \(Bronze\), data cleansing and validation using PySpark in Azure Databricks \(Silver\), and business transformations with aggregated outputs for analytics \(Gold\)\. • Developed complex PySpark and Spark SQL notebooks in Databricks to standardize multi\-source data, including null handling, deduplication, schema enforcement, and data type transformations, ensuring high data quality and consistency\. • Built and optimized ADF pipelines using Copy Activity, loading curated Gold\-layer datasets into PostgreSQL, supporting reliable and scalable downstream reporting and analytics\. • Automated end\-to\-end data workflows \(100%\) using ADF event\-based and scheduled triggers, enabling fully hands\-free pipeline execution with zero manual intervention\. • Monitored pipeline execution, performance, and failures using ADF monitoring tools, proactively resolving data quality and latency issues to ensure smooth data delivery\. • Authored High\-Level Design \(HLD\) and Low\-Level Design \(LLD\) documentation, and led technical walkthroughs with stakeholders to ensure alignment across cross\-functional teams\.

Assistant System Engineer at Tata Consultancy Services, Jan 2022 \- Mar 2023 · 1 yr 3 mos, Bhubaneswar, • Designed and developed a modular Python\-based ETL framework, leveraging a main\.py orchestrator with separate modules for extraction \(query\.py\) and loading \(load\.py\), ensuring scalability and clear separation of concerns\. • Built robust data ingestion pipelines using PySpark with JDBC connectors to extract large\-scale customer and transactional data from Hive and Teradata into a unified processing layer\. • Implemented advanced PySpark transformations, including window functions to identify latest customer records and complex business logic for revenue and usage metrics across multiple business segments\. • Optimized large\-scale joins on 1M\+ customer records with 500\+ attributes, utilizing broadcast joins and data skew handling techniques, significantly improving processing efficiency\. • Developed a dedicated aggregation pipeline in PySpark to apply business rules on staged Hive data and load curated datasets into PostgreSQL \(PgAdmin\), enabling reliable reporting and analytics\. • Fine\-tuned Spark configurations \(executor memory, cores, dynamic allocation\), reducing job runtime from 6 hours to \~45–60 minutes, enabling near real\-time data availability\. • Delivered production\-ready data pipelines and collaborated with operations teams for deployment; orchestrated end\-to\-end workflows using Apache NiFi, automating data flow from ingestion to final load\.

## Education

Bachelor of Technology \- BTech, Mechanical Engineering, Academy of Technology

[View the original profile](https://tal.club/meet/soumik-roy)

```json
{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "Person",
      "@id": "https://tal.club/meet/soumik-roy#person",
      "name": "Soumik Roy",
      "url": "https://tal.club/meet/soumik-roy",
      "image": "https://tal.club/meet/soumik-roy/profile-photo.jpg",
      "description": "Meet Soumik Roy, Azure Data Engineer | 4+ Yrs | ADF | Databricks | PySpark | Reduced Spark Runtime 85% on 1M+ Records | Medallion Architecture | 8x Microsoft & Databricks Certified | Immediate Joiner based in Kolkata.",
      "jobTitle": "System Engineer",
      "worksFor": {
        "@type": "Organization",
        "name": "Tata Consultancy Services"
      },
      "hasOccupation": {
        "@type": "Occupation",
        "name": "System Engineer"
      },
      "homeLocation": {
        "@type": "Place",
        "name": "Kolkata"
      },
      "alumniOf": [
        {
          "@type": "EducationalOrganization",
          "name": "Academy of Technology"
        }
      ],
      "mainEntityOfPage": {
        "@id": "https://tal.club/meet/soumik-roy#webpage"
      }
    },
    {
      "@type": "ProfilePage",
      "@id": "https://tal.club/meet/soumik-roy#webpage",
      "url": "https://tal.club/meet/soumik-roy",
      "name": "Soumik Roy on Tal",
      "description": "Meet Soumik Roy, Azure Data Engineer | 4+ Yrs | ADF | Databricks | PySpark | Reduced Spark Runtime 85% on 1M+ Records | Medallion Architecture | 8x Microsoft & Databricks Certified | Immediate Joiner based in Kolkata.",
      "dateModified": "2026-06-10T13:55:54.713Z",
      "isPartOf": {
        "@id": "https://tal.club/#website"
      },
      "mainEntity": {
        "@id": "https://tal.club/meet/soumik-roy#person"
      },
      "primaryImageOfPage": {
        "@type": "ImageObject",
        "url": "https://tal.club/meet/soumik-roy/profile-photo.jpg",
        "contentUrl": "https://tal.club/meet/soumik-roy/profile-photo.jpg",
        "caption": "Soumik Roy profile photo on Tal"
      },
      "inLanguage": "en-IN"
    }
  ]
}
```