Headline

Data Engineering Manager

About

· 13+ years of experience in Data warehouse application Development, Testing and Production Support on BFSI, Healthcare, Insurance, Retail and E-commerce Domain. · 8+ years of experience in providing analytics-based business solutions using Hadoop Ecosystem (Hadoop, Cloudera, Impala, Hortonworks). · Leading a Team of Data Engineers, Test Engineers (14 people), acquiring/nurturing talent, helping them align with roadmap to deliver continued value to the business. · Expertise in Big Data Technologies Hadoop Ecosystem, HDFS, Sqoop, Scala, Python, Spark, Kafka, Spark Streaming, Informatica PowerCenter, Oracle, DB2, HBase, Unix, JIRA, Autosys, Airflow · Experience in importing and exporting data from RDBMS to HDFS, Hive tables and HBase by using Sqoop. · Deep knowledge in incremental imports, partitioning and bucketing concepts in Hive and Spark SQL needed for optimization. · Experience in processing large set of Structured, Semi-structured, Unstructured datasets and Streaming data using Spark. · Experience working on Spark in performing ETL using Spark Core, Spark-SQL and Real-time data processing using Spark Streaming. · Worked with different file formats like JSON, XML, Avro data files and text files. · Ability to troubleshoot and tune relevant programming languages like SQL, Python, Scala, PIG, Hive, RDDs, Data Frames. · Experience in NoSQL databases such as HBase and Cassandra. · Developed end to end ETL Pipeline using Spark, Scala and Kafka API’s · Good Understanding of Big query and Migration Agent in GCP · Knowledge on Cloud Computing with Amazon Web Services like EMR, EC2, S3,Glue,Athena Optimized Hive QL/ pig scripts by using execution engine like Tez, Spark. · Good Understanding of Different RDBMS databases like Oracle, Teradata, Netezza. · Experience integrating data from AWS S3 into Snowflake, including loading nested JSON data. · Experience in Writing Complex SQL queries and Optimization. · Experience using Snowflake Clone and Time Travel features. · Good understanding of Datawarehouse Concepts and RDBMS databases such as SQL Server, Oracle & Teradata · Good understanding of Software Development Life Cycle and development methodologies (Agile/Waterfall). · Experienced in handling production support which includes monitoring online, production batch runs, problem determination and Providing resolution. · Excellent verbal and written communication skills, with ability to lead a project team through entire project lifecycle.

Location

Bengaluru

AI stack

claude code

claude

codex

github copilot

llmapi

ChatGPT

Work experience

Assistant Vice President-Engineering Manager at Commonwealth Bank, Oct 2022 - Present · 3 y 11 mo, Bengaluru, Karnataka, India

Senior Data Engineer-II at Rakuten, Jan 2022 - Oct 2022 · 10 mos, Bengaluru, Karnataka, India

Senior Data Engineer-I at Rakuten, Jan 2020 - Feb 2022 · 2 yrs 2 mos, Bengaluru, Karnataka, India

Data Engineer at Rakuten, Apr 2018 - Jan 2020 · 1 yr 10 mos, Bengaluru, Karnataka, India

Data Engineer at Logitech, Oct 2017 - Apr 2018 · 7 mos, Chennai, Tamil Nadu, India

ETL Developer at AstraZeneca, Sep 2016 - Oct 2017 · 1 yr 2 mos, Greater Chennai Area, The Next Generation Warehouse will provide the foundation to enable the transformational business capabilities and support existing information needs. NGW enhance efficiency and effectiveness and business decisions and insight generation by providing a commercial information management foundation with the appropriate breadth and depth of data. The SIMON is the application system developed to support Headquarters Commercial Analytics and Field Sales Reporting tool in AstraZeneca North America. Simon utilizes Micro Strategy Business Intelligence Tool as the platform for delivering the needed data. The current reporting solution leverages Netezza and Informatica as the technical platform.

Software Developer at Anthem, Inc., Jan 2014 - Sep 2016 · 2 yrs 9 mos, Greater Bengaluru Area, • Regular meetings with Onsite coordinator on the Project Status. • Analyzing the technical specifications. • Understanding existing business model and Customer Requirements. • Involved in Performance Tuning of Informatica ETL mappings and database. • Creating Mapping Document based on BRD. • Creating RTM (Requirement Traceability Matrix) based on BRD. • Fixing invalid Mappings, testing of Stored Procedures and Functions, Unit and Integration Testing of Informatica Sessions, Batches and Target Data • Extensively used Transformations like Router, Aggregator, Source Qualifier, Joiner, Expression, Aggregator and Sequence generator. • Worked with different Sources such as Oracle and flat files. • Prepared documentation to describe program development, logic, coding, testing, changes and corrections.

Education

Postgraduate Level Advanced Certificate - Master's Level, Computational Data Science, Data Science,Machine Learning,Deep Learning,Neural networks,AI,GenAI, Indian Institute of Science (IISc)

Bachelor of Technology (B.Tech.), Electronics and Communications Engineering, Raj Kumar Goel Institute of Technology, Ghaziabad