Manish Arora
Data Engineer | Helping Organizations Build Scalable Data Architectures | Skilled in PySpark • AWS EMR • Airflow • ETL Development • Spark Optimization
Delhi
7+ years of work experience
Jan 20242 yrs 6 mosPresent
Gurugram, Haryana, India
Designed and deployed scalable ETL pipelines using PySpark on AWS EMR, processing 500GB of data daily and increasing overall processing efficiency by 15%. Optimized Spark applications using partitioning, caching, broadcast joins, and data skew handling techniques, reducing job runtime, saving 2+ hours daily. Automated EMR cluster provisioning and developed Apache Airflow DAGs to orchestrate PySpark jobs, improving pipeline reliability and minimizing manual operational efforts. Identified and resolved common Spark DataFrame challenges, such as data processing errors, performance bottlenecks, and scalability limitations, to ensure smooth and efficient operations. Debugged and resolved complex Spark data transformation issues in PySpark applications on AWS EMR, ensuring data integrity and pipeline stability. Ingested and wrote final transformed datasets into Snowflake to support downstream analytics and reporting. Created and managed partitioned, external, and internal Hive tables. Collaborated with business stakeholders to translate requirements into scalable data solutions.
Mar 20221 yr 10 mosDec 2023
On-site
Built scalable batch data processing workflows using PySpark on AWS EMR with data storage in Amazon S3. Performed Spark DataFrame transformations including filtering, aggregation, joins, and window functions on large datasets. Applied Spark performance tuning techniques including caching, partition optimization, shuffle optimization, and query optimization. Created hive queries and optimized execution plans to improve resource utilization. Implemented incremental data loading strategies and partitioning techniques to enhance batch processing efficiency and reduce overall job execution time.
May 20191 yr 7 mosNov 2020
Noida, Uttar Pradesh, India
Responsible for Extracting Production Data, Warehouse Data from Excel Files, SQL and to create Daily, Weekly
and Monthly Basis Report.
Provide business insights to Managers of Different Departments teams and through data by creating
Dashboards on Tableau.
Responsible for New Pilot Project Data Collection, and Analysis of production requirements of Raw
Material for it & accordingly production planning was formulated.
To maintain and create well detailed Data Report of Usage of Equipment for checking their lifecycle, which
helped in Tracking Equipment that were underperforming.
May 20191 yr 7 mosNov 2020
Engineered SQL-based dashboards to provide visibility into production KPIs and operational metrics. Built automated ETL workflows to improve reporting accuracy by 25%. Performed data analysis using Python (Pandas, NumPy) to identify production output trends and downtime. Worked closely with business stakeholders to design and deliver data-driven reporting solutions aligned with operational needs.
Jun 20181 yrMay 2019
Noida , India
Currently Working as a Graduate Engineer Trainee at the position of Maintenance Engineer.
- With Basic knowledge of SMT machines used in production of PCBA.
- As Maintenance Engineer, Responsible for maintenance of all the machines, other units and equipment used in SMT production.
- Basic knowledge of how to operate and handle problems occuring in NXT II machines and FUJI FLEXA software.
- Knowledge of GPX printer.
- Coordinate with the whole Engineering Team to ensure proper maintenance of every equipment.
Manish’s AI Stack
Education
Bachelor of Technology, Electrical and Electronics Engineering
sharda university
20144 yrs2018
Join Manish and 10k+ other professionals on
Tal is the app where bosses meet engineers & builders directly, without middlemen
Download Tal Boss