Komal Singh profile photo 1 on Tal

Komal Singh

Senior consultant at Xebia

Ghaziabad

7+ years of work experience

Xebia
Sep 202510 mosPresent

Gurugram, Haryana, India

I am currently working on the data lake enhancement project for SkyShowtime, a leading OTT streaming platform similar to Amazon Prime Video. My focus is on building and optimizing the Silver layer of the data architecture, which integrates and transforms finance and marketing data from multiple upstream systems to support advanced analytics and business reporting. Key Responsibilities & Contributions: Designing and developing data transformation pipelines within the Silver layer to convert raw ingestion data (Bronze) into standardized, analytics-ready datasets. Using PySpark and SQL on Databricks / Spark clusters to perform large-scale data cleansing, enrichment, and validation across transactional and streaming datasets. Integrating finance and marketing datasets to enable metrics such as revenue analysis, subscription trends, customer segmentation, and campaign ROI tracking. Implementing incremental data loading, Delta Lake optimization, and partitioning strategies to ensure efficient querying, scalability, and reduced processing cost. Collaborating closely with data modeling, BI, and analytics teams to deliver curated Silver data that powers Gold-layer dashboards and business KPIs. Maintaining data quality, schema evolution, and error-handling frameworks to ensure accuracy and reliability across finance and marketing data sources. Supporting compliance and governance by aligning with data lineage, cataloging, and auditing best practices.
Tata Consultancy Services
Nov 20222 yrs 11 mosSep 2025

Noida, Uttar Pradesh, India

Designed and implemented large-scale data ingestion, integration, and AML analytics pipelines across hybrid environments using Hadoop, Oozie, and the Google Cloud Platform (GCP). Automated end-to-end ETL workflows using Python, Shell, Hive, and Airflow to process and transform large volumes of structured and semi-structured data from Oracle, HDFS, and Cloud Storage into analytical datasets supporting compliance and fraud detection. Developed modular Python scripts for parsing, validation, enrichment, and dynamic HiveQL generation, improving reusability and maintainability across pipelines. Integrated these components within Oozie workflows, orchestrating dependencies, error handling, and recovery mechanisms across distributed clusters. Optimized Hive transformations and aggregations to support transaction monitoring, regulatory reporting, and risk analytics. On GCP, built an end-to-end AML detection and reporting system handling 500+ TB of financial transactions. Leveraged Cloud Composer (Airflow) for orchestration, Dataproc (PySpark) for large-scale transformation and enrichment, and BigQuery for high-performance analytics. Implemented rule-based detection logic to identify anomalies such as smurfing, rapid fund transfers, and high-risk behavior. Improved query efficiency through advanced partitioning, clustering, and schema evolution strategies. Executed EAP-based ingestion pipelines for multi-layered banking data, implementing reconciliation via manifest files and row-level validation to ensure full data traceability. Automated test data simulation and regression validation using Shell scripts across global regions, enhancing data integrity and scalability. PySpark-based AML search modules to identify suspicious user accounts and transactions, integrating multiple financial data sources for real-time compliance monitoring. Collaborated with risk and compliance teams to refine detection thresholds, reduce false positives, and accelerate AML reporting.
Teradata
Nov 20211 yr 1 moNov 2022

Pune, Maharashtra, India

Led end-to-end ETL and enterprise data warehousing initiatives using Teradata, delivering scalable, high-performance analytics on telecom datasets such as CDRs, subscriber usage, and network performance. Designed and optimized complex SQL-based data transformations, integrating semi-structured data (e.g., JSON logs, network event feeds) into unified telecom data models to support business KPIs. Automated data ingestion pipelines using PySpark, Teradata TPT/BTEQ, and custom Python scripts, enabling seamless data integration from operational systems to the data warehouse. Implemented robust data governance practices, including metadata management, audit trails, and data lineage tracking to ensure regulatory compliance and traceability. Enhanced performance and reporting by applying indexing, partitioning, and semantic layer design, supporting advanced telecom analytics such as churn prediction, ARPU analysis, and network optimization.
NTT DATA Services
Feb 20201 yr 10 mosNov 2021

Bengaluru, Karnataka, India

Led ingestion of diverse healthcare data sources including EHR files, HL7/FHIR streaming data, and clinical data from Oracle and other relational databasesinto big data ecosystems using Apache Spark, Hive, SQL, and GCP services. Developed scalable, efficient data pipelines on GCP (Dataflow, Dataproc, Cloud Storage) for cleansing and transforming healthcare data using Spark, Hive, and SQL, enabling advanced analytics for clinical insights, quality reporting, and population health management. Designed and implemented high-throughput, multi-threaded event processing systems using GCP Pub/Sub and Dataflow, ensuring low latency, fault tolerance, and HIPAA-compliant scalability for real-time healthcare data. Built reusable Spark-based frameworks deployed on GCP Dataproc, optimized for processing complex healthcare datasets across distributed environments, improving data throughput and maintainability. Ensured platform reliability through GCP Operations Suite (Stackdriver) for monitoring, along with secure logging, exception handling, and SQL-based data quality checks aligned with regulatory standards. Architected and deployed end-to-end big data solutions on GCP, capable of handling massive and rapidly growing volumes of healthcare data while maintaining regulatory compliance, data integrity, and system performance.
NTT DATA Services
Sep 20196 mosFeb 2020

Hyderabad, Telangana, India

I Joined NTT as a fresher and got trained on SQL and Oracle.

Education

Dr. A.P.J. Abdul Kalam Technical University

Bachelor of Technology - BTech, Computer Science

Dr. A.P.J. Abdul Kalam Technical University

20154 yrs2019
ABES Engineering College

Bachelor of Technology - BTech, Computer Science

ABES Engineering College

20154 yrs2019

Join Komal and 10k+ other professionals on

Tal Boss

Tal is the app where bosses meet engineers & builders directly, without middlemen

Download Tal Boss

Komal Singh on Tal

Headline

Senior consultant at Xebia

About

Ingenious and determined Database Engineer with over 5+ years of experience. Seeks to support L3 level on different Database language like SQL, Pyspark SQL, Bigdata, Hadoop Ecosystems, GCP-Terraforms.

Location

Ghaziabad

Work experience

Senior Consultant at Xebia, Sep 2025 - Present · 10 mos, Gurugram, Haryana, India, I am currently working on the data lake enhancement project for SkyShowtime, a leading OTT streaming platform similar to Amazon Prime Video. My focus is on building and optimizing the Silver layer of the data architecture, which integrates and transforms finance and marketing data from multiple upstream systems to support advanced analytics and business reporting. Key Responsibilities & Contributions: Designing and developing data transformation pipelines within the Silver layer to convert raw ingestion data (Bronze) into standardized, analytics-ready datasets. Using PySpark and SQL on Databricks / Spark clusters to perform large-scale data cleansing, enrichment, and validation across transactional and streaming datasets. Integrating finance and marketing datasets to enable metrics such as revenue analysis, subscription trends, customer segmentation, and campaign ROI tracking. Implementing incremental data loading, Delta Lake optimization, and partitioning strategies to ensure efficient querying, scalability, and reduced processing cost. Collaborating closely with data modeling, BI, and analytics teams to deliver curated Silver data that powers Gold-layer dashboards and business KPIs. Maintaining data quality, schema evolution, and error-handling frameworks to ensure accuracy and reliability across finance and marketing data sources. Supporting compliance and governance by aligning with data lineage, cataloging, and auditing best practices.

Big Data Engineer at Tata Consultancy Services, Nov 2022 - Sep 2025 · 2 yrs 11 mos, Noida, Uttar Pradesh, India, Designed and implemented large-scale data ingestion, integration, and AML analytics pipelines across hybrid environments using Hadoop, Oozie, and the Google Cloud Platform (GCP). Automated end-to-end ETL workflows using Python, Shell, Hive, and Airflow to process and transform large volumes of structured and semi-structured data from Oracle, HDFS, and Cloud Storage into analytical datasets supporting compliance and fraud detection. Developed modular Python scripts for parsing, validation, enrichment, and dynamic HiveQL generation, improving reusability and maintainability across pipelines. Integrated these components within Oozie workflows, orchestrating dependencies, error handling, and recovery mechanisms across distributed clusters. Optimized Hive transformations and aggregations to support transaction monitoring, regulatory reporting, and risk analytics. On GCP, built an end-to-end AML detection and reporting system handling 500+ TB of financial transactions. Leveraged Cloud Composer (Airflow) for orchestration, Dataproc (PySpark) for large-scale transformation and enrichment, and BigQuery for high-performance analytics. Implemented rule-based detection logic to identify anomalies such as smurfing, rapid fund transfers, and high-risk behavior. Improved query efficiency through advanced partitioning, clustering, and schema evolution strategies. Executed EAP-based ingestion pipelines for multi-layered banking data, implementing reconciliation via manifest files and row-level validation to ensure full data traceability. Automated test data simulation and regression validation using Shell scripts across global regions, enhancing data integrity and scalability. PySpark-based AML search modules to identify suspicious user accounts and transactions, integrating multiple financial data sources for real-time compliance monitoring. Collaborated with risk and compliance teams to refine detection thresholds, reduce false positives, and accelerate AML reporting.

Data Engineer at Teradata, Nov 2021 - Nov 2022 · 1 yr 1 mo, Pune, Maharashtra, India, Led end-to-end ETL and enterprise data warehousing initiatives using Teradata, delivering scalable, high-performance analytics on telecom datasets such as CDRs, subscriber usage, and network performance. Designed and optimized complex SQL-based data transformations, integrating semi-structured data (e.g., JSON logs, network event feeds) into unified telecom data models to support business KPIs. Automated data ingestion pipelines using PySpark, Teradata TPT/BTEQ, and custom Python scripts, enabling seamless data integration from operational systems to the data warehouse. Implemented robust data governance practices, including metadata management, audit trails, and data lineage tracking to ensure regulatory compliance and traceability. Enhanced performance and reporting by applying indexing, partitioning, and semantic layer design, supporting advanced telecom analytics such as churn prediction, ARPU analysis, and network optimization.

Database Engineer at NTT DATA Services, Feb 2020 - Nov 2021 · 1 yr 10 mos, Bengaluru, Karnataka, India, Led ingestion of diverse healthcare data sources including EHR files, HL7/FHIR streaming data, and clinical data from Oracle and other relational databasesinto big data ecosystems using Apache Spark, Hive, SQL, and GCP services. Developed scalable, efficient data pipelines on GCP (Dataflow, Dataproc, Cloud Storage) for cleansing and transforming healthcare data using Spark, Hive, and SQL, enabling advanced analytics for clinical insights, quality reporting, and population health management. Designed and implemented high-throughput, multi-threaded event processing systems using GCP Pub/Sub and Dataflow, ensuring low latency, fault tolerance, and HIPAA-compliant scalability for real-time healthcare data. Built reusable Spark-based frameworks deployed on GCP Dataproc, optimized for processing complex healthcare datasets across distributed environments, improving data throughput and maintainability. Ensured platform reliability through GCP Operations Suite (Stackdriver) for monitoring, along with secure logging, exception handling, and SQL-based data quality checks aligned with regulatory standards. Architected and deployed end-to-end big data solutions on GCP, capable of handling massive and rapidly growing volumes of healthcare data while maintaining regulatory compliance, data integrity, and system performance.

Database Developer at NTT DATA Services, Sep 2019 - Feb 2020 · 6 mos, Hyderabad, Telangana, India, I Joined NTT as a fresher and got trained on SQL and Oracle.

Education

Bachelor of Technology - BTech, Computer Science, Dr. A.P.J. Abdul Kalam Technical University

Bachelor of Technology - BTech, Computer Science, ABES Engineering College