Headline

Data Engineer @Accenture | PySpark, Kafka, AWS | Exploring Real-Time & Cloud-Native Architectures

About

Hi there! I'm Shweta, I am a Data Engineer with 5.4 years of experience working on large-scale data systems in the financial domain, currently at Accenture. My work focuses on building and optimizing distributed data pipelines using PySpark, Hive, Hadoop (HDFS), Kafka, and AWS (S3, Athena), handling high-volume data at TB scale with an emphasis on performance and reliability. In my current role, I have worked on data ingestion, transformation, and secure data movement across systems. I have optimized Spark jobs, achieving a ~40–45% reduction in runtime, and contributed to improving pipeline stability through timely resolution of production issues. I have also implemented reusable data processing components and automated workflow steps, leading to reduced manual intervention and improved operational efficiency. In addition, I am working on a real-time data streaming project using Kafka, PySpark, and AWS, designed to simulate production-grade systems with capabilities such as real-time insights, anomaly detection, and user behavior analysis, supported by a Streamlit-based dashboard. I am now looking to transition into high-impact Data Engineering roles where I can work on real-time streaming systems, modern data platforms (Lakehouse architectures), and cloud-native solutions, contributing to scalable, data-driven decision-making systems.

Location

Pune Division

Work experience

Data Engineering Senior Analyst at Accenture, Dec 2023 - Present · 2 yrs 6 mos, Pune, - Worked on processing delinquent account data within a financial data platform, building and enhancing PySpark and Hive-based pipelines on Hadoop (HDFS) to support downstream systems such as risk analysis, reporting, and collections workflows. - Automated manual validation and reconciliation processes by developing PySpark jobs and Python scripts for schema checks, data consistency, and file-level validations, improving pipeline efficiency and reducing operational overhead. Optimized transformation logic and job workflows for better performance and maintainability. - Extended data pipelines by integrating Kafka-based ingestion and leveraging AWS services including S3 for storage, Athena for querying, and Glue for managing ETL workflows and metadata. Technologies Used: PySpark, Apache Kafka, AWS (S3, Athena, Glue), Hive, Hadoop (HDFS), SQL, Python, OpenSearch (ELK Stack)

Technology Analyst at Infosys, Oct 2023 - Dec 2023 · 3 mos, Pune District, - Designed and developed scalable batch data pipelines using PySpark and Hive to process transaction and spend data, generating segment-wise datasets (Inactive, Prospect, Customer) for campaign analytics and downstream consumption. - Re-architected data ingestion from SFTP-based pipelines to a Kafka-driven streaming framework, enabling near real-time data flow and implementing automated validation mechanisms to ensure data consistency across multiple global markets. - Optimized and refactored Spark and Hive workloads by introducing structured logging, modular code design, and Python-based testing frameworks, improving maintainability, debuggability, and overall pipeline efficiency. - Contributed to the migration of monitoring and visualization systems from Grafana to Kibana using the ELK stack, enabling centralized logging, enhanced observability, and improved debugging capabilities for production pipelines. - Built and maintained curated data layers in Hive for analytics and dashboarding, integrating with Kibana through scheduled Event Engine jobs to support business reporting and insights. Managed end-to-end pipeline deployment and monitoring using Jenkins, ensuring reliable job execution, proactive issue resolution, and adherence to code quality standards through SonarQube analysis. Technologies Used: PySpark, Apache Kafka, Hive, Hadoop (HDFS), SQL, Python, Jenkins, ELK Stack (Elasticsearch, Logstash, Kibana)

Senior System Engineer at Infosys, Apr 2023 - Sep 2023 · 6 mos, Pune, Maharashtra, India

System Engineer at Infosys, Jul 2021 - Mar 2023 · 1 yr 9 mos, Pune, Maharashtra, India

Education

Bachelor of Engineering, Electronics and Telecommunication, Savitribai Phule Pune University

HSC, Pure Science, Maharashtra State Board of Secondary and Higher Secondary Education (MSBSHSE)