Headline

Data Engineer

About

Data Engineer with 4.5 years of experience in designing, building, and optimizing scalable data pipelines on AWS. Azure, Strong expertise in Python, PySpark, SQL, AWS Glue, Amazon S3, Athena, Redshift Spectrum, Azure Data Factory, ADLS, Azure Databricks and Apache Airflow. Proven track record of developing end-to-end ETL and Data Lakehouse solutions for enterprise analytics and reporting. Experienced in ingesting data from multiple sources, performing large-scale data transformations, implementing data quality checks, and optimizing Spark jobs for performance and cost efficiency. Skilled in workflow orchestration, dependency management, monitoring, and automation using Apache Airflow. Hands-on experience in building Raw, Curated, and Analytics data zones, schema design, Parquet and Delta Lake formats, and high-performance query optimization. A proactive problem-solver with strong communication skills, capable of translating business requirements into reliable, cloud-based data solutions.

Location

Bhopal

Work experience

Data Engineer at SOFTINTON SOLUTIONS PRIVATE LIMITED, Feb 2023 - Present · 3 yrs 5 mos, India, 1. Designed and implemented scalable ETL pipelines on Azure to meet enterprise data integration requirements. 2. Developed and optimized ETL workflows using Azure Data Factory and Apache Airflow for automated ingestion from diverse data sources. 3. Performed data transformation, cleansing, and validation using PySpark to ensure data accuracy and consistency. 4. Leveraged Azure Synapse Analytics and Azure Data Lake Storage to optimize data querying and improve reporting performance. 5. Ensured pipeline reliability and scalability by implementing monitoring, logging, optimization, and issue-resolution strategies. 6. Optimized Azure Data Factory pipelines, Spark configurations, and query performance to improve processing efficiency and cost-effectiveness. 7. Collaborated with business stakeholders to gather requirements and deliver end-to-end data solutions aligned with business needs.

Data Engineer at ASK IT Solutions, Dec 2020 - Dec 2022 · 2 yrs 1 mo, India, 1. Designed data lake zones (raw, curated, analytics) on AWS S3 using Delta Lake and Parquet format. 2. Built PySpark ETL jobs using AWS Glue with partitioning and caching strategies to optimize the performance. 3. Deployed and Schedule the data pipelines using AWS Glue integrated with triggers and Apache Airflow for workflows orchestration. 4. Queried and analyzed the transformed data in Athena using SQL and Redshift Spectrum for reporting and analytics. 5. Optimized schema and storage for performance and cost efficiency. 6. Implemented data quality checks, schema validation and monitoring to ensure reliable and cosistant data across the pipelines

Business Development Executive at IFFT Pvt. Ltd., Sep 2018 - May 2020 · 1 yr 9 mos, Dehradun, Uttarakhand, India

Education

Bachelor of Engineering - BE, Electrical and Electronics Engineering, LNCT Group of Colleges

Senior High School (12th), CBS Convent HS School