Headline

immediate joiner , having 4+ years experinece as a data engineer...

Location

Karnataka

Work experience

Data Engineer at Asics Technologies, May 2024 - Present · 2 yrs 2 mos, Hyderabad, Telangana, India, Expertise in architecting and optimizing large-scale data solutions across diverse industries. Proficient in designing robust data pipelines using Hadoop, Apache Spark and Hive with a strong command of distributed computing principles. ❖ Understanding of data modelling, partitioning strategies, and performance tuning for high-throughput systems. ❖ Collaborated with cross-functional teams including Data Scientists, Analysts, and DevOps to deliver end-toend data solutions. ❖ Skilled in leveraging Hive partitioning, bucketing, indexing, and caching features to improve query performance and reduce data processing overhead. ❖ Managed Hadoop clusters using Cloudera distributions, including configuration, monitoring, and troubleshooting ❖ Developed Hive queries for data summarization, ad-hoc analysis, and reporting on large datasets. ❖ Proficient in using Apache Sqoop to import and export data between Hadoop Distributed File System (HDFS) and relational databases like MySQL, Oracle, and PostgreSQL. ❖ Optimized Sqoop performance using parallel imports, split-by columns, and compression techniques ❖ Integrated Hive with Apache Sqoop for seamless data ingestion from relational databases into Hadoop. ❖ Created DAG templates to standardize job orchestration across multiple Spark use cases. ❖ Used AWS EMR to run distributed Spark and Hadoop jobs, tuning cluster configurations for performance and cost optimization ❖ Developed Spark jobs on AWS EMR for large-scale data stored in AWS S3 buckets. ❖ Optimized PySpark applications on AWS EMR by tuning shuffling and partitioning strategies. ❖ Used AWS Step Functions to parallelize PySpark jobs for faster execution on AWS EMR clusters.

Data Engineer at Amdocs, Dec 2021 - Apr 2024 · 2 yrs 5 mos, Pune District, Maharashtra, India, Designed and developed DAGs in Apache Airflow for orchestrating complex Spark pipelines on AWS EMR. ❖ Integrated Airflow with AWS services including S3, EMR, Lambda, and SNS for event-driven workflows. ❖ Tuned Airflow DAG execution timings and concurrency for optimal resource usage and parallelism. ❖ Optimized Airflow worker pool configuration for scaling orchestration in large-scale environments. ❖ Proficient in handling hive partitions and buckets with respect to the business requirement. ❖ Experience in handling hive schema evolution with avro file format ❖ Skilled in handling semi structured/serialised data processing using hive (AVRO,PARQUET,ORC) ❖ Experienced in efficiently using Hive managed and external table with respect to the business requirement. ❖ Proficient in creating and managing Hive tables, including managed, external, and partitioned tables. ❖ Ability to troubleshoot common issues with Hive tables, such as data skew, table corruption, and query optimization.

Education

Bachelor of Engineering - BE, Mechanical Engineering, Visvesvaraya Technological University

Bachelor of Engineering - BE, Mechanical Engineering, Visvesvaraya Technological University