Headline

Data Engineer | PySpark • Apache Spark • Kafka • Airflow | Big Data Pipelines & Real-Time Streaming | British Telecom | Ex-BIT Mesra | ETL • Data Lake • SQL • BigQuery

About

As a Data Engineer at British Telecom, I’ve found my passion in the intricate dance of big data and real-time streaming. Each day is an adventure where I get to design and build scalable ETL pipelines using PySpark and SQL. It’s thrilling to transform raw data into insights that empower decision-making across the organization. My journey started at the Birla Institute of Technology in Mesra, where I first discovered my love for technology and problem-solving. One of my proudest achievements has been building ETL pipelines that process over 10 million records daily. This effort significantly improved data availability for analytics across our teams. I also had the opportunity to develop real-time data processing tools during my internship at BT Group, where I created Python scripts that sped up processing times by 30%. These experiences have not only honed my technical skills but also deepened my understanding of how crucial timely data is for driving business success. I thrive on challenges—whether it’s optimizing existing systems or diving into new technologies like Kafka and Airflow. The world of big data is ever-evolving, and I’m excited to be part of it. Outside of work, I enjoy exploring new tech trends and participating in local meetups focused on data science and engineering. If you’re interested in discussing big data solutions or just want to share insights about the latest trends in technology, feel free to reach out! I’d love to connect over email. Skills: - PySpark - SQL - Kafka - Airflow - ETL Development - Real-Time Data Processing

Location

Bengaluru

AI stack

claude code

dalle

ChatGPT

Fathom

Unstructured

Work experience

Data Engineer | SQL, Python, Spark, ETL & Cloud Data Engineering at BT Group, Jul 2024 - Present · 2 yrs 3 mos, Bengaluru, Developed Python scripts to transform large datasets, reducing processing time by 30% and improving data-validation workflows. Implemented SQL validation scripts to identify and resolve data-quality issues, reducing error rates by 25% and improving the reliability of analytical outputs. Built Kafka-based data-ingestion pipelines to enable real-time data flow and support analytics across multiple projects. Implemented distributed data-processing and storage workflows using Apache Hive and HDFS to improve data accessibility and support big-data engineering initiatives. Used PySpark, Databricks, Apache Spark, Kafka, SQL, Python, Hive, and HDFS to build and validate data-processing solutions. Skills: Python, SQL, PySpark, Databricks, Apache Spark, Kafka, Hive, HDFS

Data Engineering Intern | Python Development, SQL Scripting, and Kafka Pipelines at BT Group, Jan 2024 - Jul 2024 · 7 mos, On-site, During the internship at BT Group as a Data Engineering Intern, significant strides were made in developing essential data processing tools. A focus was placed on creating Python scripts that transformed large datasets efficiently—this effort led to a remarkable 30% reduction in processing times. The work didn't stop there; implementing SQL scripting played a crucial role in validating this data as well. This initiative resulted in a 25% decrease in error rates, ensuring reliable outputs for analytical tasks across various projects. Collaboration was key while building Kafka-based ingestion pipelines that allowed real-time data flow into our systems. This enhancement greatly supported immediate analytics capabilities across multiple initiatives. Additionally, exposure to Hive and HDFS provided valuable insights into distributed data storage solutions that improved access times significantly. Overall, this experience laid a solid foundation for future endeavors in big data engineering. Competencies: Big Data Engineering, Real-Time Streaming Analytics, ETL Processes

Education

Bachelor of Technology - BTech, Electrical, Electronics and Communications Engineering, Birla Institute of Technology, Mesra

Resonance Eduventures Limited