RAHUL KUMAR SINGH profile photo 1 on Tal

RAHUL KUMAR SINGH

Data Engineering Lead | Spark ,Databricks ,Azure,AWS ,Applied AI at scale | Data Storyteller | Data Modeling | Data Architecture

Bengaluru

12+ years of work experience

Commonwealth Bank
May 20242 y 4 moPresent

Greater Bengaluru Area

Commonwealth Bank
Mar 20242 mosApr 2024

Sydney, New South Wales, Australia

Commonwealth Bank
Jan 20222 y 1 moFeb 2024

Greater Bengaluru Area

ABB
Feb 20193 yrsJan 2022

Greater Bengaluru Area

• Cleaning existing data using outliers, rolling mean window and fill missing values. • Perform univariate analysis (paired t-test, kl divergence test etc.) on key variables to identify key difference in operation. • Automated data ingestion pipeline with ADF. • Generated a power BI report for hardware engineers. • Creating a Json schema validator for generic pipeline Predictive Maintenance & Anomaly detection. • Writing Generic functions for Data ingestion, Data cleansing, Descriptive statistics and Feature Engineering for machine learning pipelines using spark. • Build test cases for generic pipelines with unittest framework. • Adding streaming Data ingestion pipelines using EventHub and Structured Streaming. • Deploying Data bricks workspace in ABB Vnet. • Ingest Ability cold Storage data in Training platform using various methods. • Evaluated and documented Ability Data ingestion approaches in Data Lake. • Using kusto query language in log analytics workspace for collecting activity and event logs from databricks, storage and machine learning. • Created a feedback plugin which takes incoming data from devices and scoring result message from ABB Ability™ Platform which is consumed through service bus periodically and stored in delta table. • Converting existing python code to pyspark for better performance and parallelization. • Using Data & Information model Api to access data from ability cold storage (time-series database). • Configuring & establishing databricks with local IDE using databricks-connect. • Creating databricks widgets, scope & secrets whenever required to store password and keys. • Build and Run test cases for Data Quality Module with unittest framework. • Create a common library and implement it for existing ML model formats to Onnx.
Robert Bosch Engineering and Business Solutions Private Limited
Sep 20171 yr 6 mosFeb 2019

Bangalore

• Involved in end to end development of Hadoop & Spark project individually. • Built a Data Quality Framework using Spark SQL and Scala. • Developed Calculation logic for different station. • Programmed code to handle small file issues in hdfs. • Optimizing the spark code to process 15k files in 2.5 min. • Use of UNIX shell scripting for pre-processing and automation of repetitive work. • Involved in code review and bug fixing for improving the performance. • Developed schema of dataset programmatically using case class and reflection for csv files. • Analysis of change from J2T batch to J2T Streaming • Developed a poc for read stream and write stream data using structured streaming • Impact analysis of files due to change in structure of fields • Writing generic spark code to load different types of Json files • Dynamically creating staged, pickup, merged folder for holding and movement of files based on project flow. • Developed application configuration for spark job and using logging mechanism • Programmed shell script for running the spark job and archival. • Developed spark job to merge small file inside partitioned table using Hadoop API. • Built program to handle no of files and merge behavior of files for processing.
HCL Technologies
Feb 20152 yrs 8 mosSep 2017

Bengaluru, Karnataka, India

• Importing data into HDFS and Hive using Sqoop. • Involved in creating tables scripts for Hive. • Developed Hive queries for data analysis to meet the business requirements. • Developed the Pig Scripts as per design document. • Programmed UDF’s (User Defined Functions) in Pig when needed. • Worked in preparation and dispatch of Oozie jobs. • Executing the Test Cases and Documenting the Test Results. • Helping the team members to track and get the issues resolved. • Participating in project status tracking call to update the status • Worked very closely with users and prepared Functional Specification and Technical Design Document according to requirements. • Programmed on staging/target tables as per business requirement. • Developed the Apache PIG scripts to process the HDFS data. • Automation of UNIX shell scripts for creating the reports from hive view using Control-m. • Performed joins, group by and other operations in Mapreduce by using Java and PIG. • Involved in handling different format of data transformation. • Involved in integration of Hive and Hbase.
ITC Infotech
Aug 20146 mosJan 2015

Bengaluru, Karnataka, India

Worked on many Internal projets of Itc infotech.Trained in MS dyanamics AX in logistics and HR module .

RAHUL’s AI Stack

claude code
claude code
claude
claude
codex
codex
github copilot
github copilot
openai
openai
claudecode
claudecode

Education

Bachelor of Technology (BTech), Computer Engineering

NM INSTIUTE OF ENGINEERING & TECHNOLOGY

20094 yrs2013

Join RAHUL and 10k+ other professionals on

Tal Boss

Tal is the app where bosses meet engineers & builders directly, without middlemen

Download Tal Boss

RAHUL KUMAR SINGH on Tal

Headline

Data Engineering Lead | Spark ,Databricks ,Azure,AWS ,Applied AI at scale | Data Storyteller | Data Modeling | Data Architecture

About

Lead Data Engineer with 12+ years of experience in Banking and Financial Services, delivering enterprise data platforms and leading engineering teams. My work has focused on designing scalable data architectures, building cloud-based solutions on AWS and Azure, and delivering reliable data platforms that support business-critical applications. Over the years, I have worked across the complete software delivery lifecycle, from solution design and development to deployment and production support. My experience includes Big Data, Data Warehousing, ETL/ELT, real-time data processing, cloud migration, and platform modernization. I enjoy working closely with business stakeholders, architects, and engineering teams to solve complex data challenges and deliver practical solutions. As a Lead Data Engineer I lead and mentor engineers, encourage engineering best practices, support DevSecOps adoption, and help teams deliver secure, reliable, and maintainable solutions. I value collaboration, continuous improvement, and building teams that consistently deliver quality outcomes.

Location

Bengaluru

AI stack

claude code

claude

codex

github copilot

openai

claudecode

Work experience

Chapter Lead Data Engineering at Commonwealth Bank, May 2024 - Present · 2 y 4 mo, Greater Bengaluru Area

Senior Data Engineer at Commonwealth Bank, Mar 2024 - Apr 2024 · 2 mos, Sydney, New South Wales, Australia

Senior Data Engineer at Commonwealth Bank, Jan 2022 - Feb 2024 · 2 y 1 mo, Greater Bengaluru Area

R&D Engineer at ABB, Feb 2019 - Jan 2022 · 3 yrs, Greater Bengaluru Area, • Cleaning existing data using outliers, rolling mean window and fill missing values. • Perform univariate analysis (paired t-test, kl divergence test etc.) on key variables to identify key difference in operation. • Automated data ingestion pipeline with ADF. • Generated a power BI report for hardware engineers. • Creating a Json schema validator for generic pipeline Predictive Maintenance & Anomaly detection. • Writing Generic functions for Data ingestion, Data cleansing, Descriptive statistics and Feature Engineering for machine learning pipelines using spark. • Build test cases for generic pipelines with unittest framework. • Adding streaming Data ingestion pipelines using EventHub and Structured Streaming. • Deploying Data bricks workspace in ABB Vnet. • Ingest Ability cold Storage data in Training platform using various methods. • Evaluated and documented Ability Data ingestion approaches in Data Lake. • Using kusto query language in log analytics workspace for collecting activity and event logs from databricks, storage and machine learning. • Created a feedback plugin which takes incoming data from devices and scoring result message from ABB Ability™ Platform which is consumed through service bus periodically and stored in delta table. • Converting existing python code to pyspark for better performance and parallelization. • Using Data & Information model Api to access data from ability cold storage (time-series database). • Configuring & establishing databricks with local IDE using databricks-connect. • Creating databricks widgets, scope & secrets whenever required to store password and keys. • Build and Run test cases for Data Quality Module with unittest framework. • Create a common library and implement it for existing ML model formats to Onnx.

Senior Engineer at Robert Bosch Engineering and Business Solutions Private Limited, Sep 2017 - Feb 2019 · 1 yr 6 mos, Bangalore, • Involved in end to end development of Hadoop & Spark project individually. • Built a Data Quality Framework using Spark SQL and Scala. • Developed Calculation logic for different station. • Programmed code to handle small file issues in hdfs. • Optimizing the spark code to process 15k files in 2.5 min. • Use of UNIX shell scripting for pre-processing and automation of repetitive work. • Involved in code review and bug fixing for improving the performance. • Developed schema of dataset programmatically using case class and reflection for csv files. • Analysis of change from J2T batch to J2T Streaming • Developed a poc for read stream and write stream data using structured streaming • Impact analysis of files due to change in structure of fields • Writing generic spark code to load different types of Json files • Dynamically creating staged, pickup, merged folder for holding and movement of files based on project flow. • Developed application configuration for spark job and using logging mechanism • Programmed shell script for running the spark job and archival. • Developed spark job to merge small file inside partitioned table using Hadoop API. • Built program to handle no of files and merge behavior of files for processing.

Senior Software Engineer at HCL Technologies, Feb 2015 - Sep 2017 · 2 yrs 8 mos, Bengaluru, Karnataka, India, • Importing data into HDFS and Hive using Sqoop. • Involved in creating tables scripts for Hive. • Developed Hive queries for data analysis to meet the business requirements. • Developed the Pig Scripts as per design document. • Programmed UDF’s (User Defined Functions) in Pig when needed. • Worked in preparation and dispatch of Oozie jobs. • Executing the Test Cases and Documenting the Test Results. • Helping the team members to track and get the issues resolved. • Participating in project status tracking call to update the status • Worked very closely with users and prepared Functional Specification and Technical Design Document according to requirements. • Programmed on staging/target tables as per business requirement. • Developed the Apache PIG scripts to process the HDFS data. • Automation of UNIX shell scripts for creating the reports from hive view using Control-m. • Performed joins, group by and other operations in Mapreduce by using Java and PIG. • Involved in handling different format of data transformation. • Involved in integration of Hive and Hbase.

Project Trainee at ITC Infotech, Aug 2014 - Jan 2015 · 6 mos, Bengaluru, Karnataka, India, Worked on many Internal projets of Itc infotech.Trained in MS dyanamics AX in logistics and HR module .

Education

Bachelor of Technology (BTech), Computer Engineering, NM INSTIUTE OF ENGINEERING & TECHNOLOGY