Headline

Sr. Data Engineer |IIIT Bhagalpur-22 | Azure Databricks |Spark, AWS & Azure, Python,PySpark, ADF, Data Lake |Data Analytics |Lakehouse (Hudi, Iceberg, Delta) |Real-time Streaming | AI/Semantic Layer| 4.8 years

About

Welcome to my LinkedIn profile! I'm Shubh Agrawal, a passionate Sr. Data Engineer and BI Developer and based in Pune, India with 4.8 years of experience designing and scaling data platforms across AWS and Azure. I specialize in building real-time and batch data pipelines using Spark, PySpark, and Databricks, along with lakehouse architectures using Hudi, Iceberg, and Delta Live Tables. I have worked on large-scale systems processing 500M+ transactions, achieving 40% performance improvement and 35% cost reduction. Recently, I have been working on AI-driven semantic layers using RDF/OWL and Stardog, enabling natural language querying over data platforms. Technical Proficiencies: Core Competencies: Data Engineering, Big Data Processing, ETL Development, Data Modeling, Lakehouse Architecture, Real-time Data Processing, Semantic Layer Design, Data Governance Languages: Python, SQL, T-SQL, KQL, JavaScript, C# Big Data & Processing: Apache Spark, PySpark, Databricks, Spark Structured Streaming Lakehouse & Storage: Apache Hudi, Apache Iceberg, Delta Lake, Delta Live Tables (DLT) Query Engines & Analytics: Presto, Trino, Cube.js Cloud Platforms: AWS (EMR, S3, Glue), Azure (ADF, Databricks, Synapse, Fabric, ADLS) Visualization: Power BI, Tableau, QlikView, Looker, Metabase Tools & DevOps: Airflow, SSIS, Azure DevOps, Docker, Jenkins Others: OpenAI, Prompt Flow, Azure Logic Apps, Key Vault, GitHub Actions Project Management: JIRA, Bitbucket, Kanban Soft Skills: Project Management, Communication, Leadership, Stakeholder Collaboration, Cross-functional Coordination Committed to continuous learning, I hold certifications and digital badges showcasing my expertise, including recognitions from IBM, Microsoft, Snowflake and other industry leaders. Feel free to connect with me to explore synergies and opportunities to collaborate on impactful, data-driven initiatives. Let’s turn data into actionable insights together! Open to opportunities in Data Engineering and Data + AI domains.

Location

Noida

AI stack

codex

cursor

Work experience

Senior Data Engineer at R Systems, Mar 2025 - Present · 1 yr 6 mos, Pune, Maharashtra, India, ● Architected scalable cloud-native data platforms on Azure Databricks, Microsoft Fabric, and AWS, building enterprise ETL/ELT pipelines processing multi-terabyte datasets with Python, PySpark, SQL, Spark Structured Streaming, and Delta Live Tables (DLT). ● Built 50+ Delta Live Tables (DLTs) using Medallion Architecture and Unity Catalog, implementing automated data quality, governance, schema evolution, and lineage for enterprise IoT data platforms. ● Engineered near real-time streaming pipelines using Spark Structured Streaming with incremental processing, optimized checkpointing, and partition tuning, improving pipeline reliability and recovery time by 75%. ● Developed a metadata-driven data engineering framework with reusable PySpark modules and SQL templates, implementing Spark performance optimizations (salting, shuffle tuning, executor optimization) that reduced executor failures by 90%. ● Designed an AI-powered semantic query layer by converting Cube.js models into RDF/OWL ontologies using Python and integrating with Ontotext/Stardog, enabling natural language analytics through AI agents and improving query response accuracy to 80%. ● Modernized the enterprise semantic layer by migrating 100+ Looker Explores into Cube.js models, standardizing business logic and enabling scalable API-driven analytics with 340+ business metrics powered by Presto/Trino. ● Architected a scalable analytics platform on AWS EMR using Apache Hudi and Apache Iceberg, processing 500M+ transactions while reducing processing time by 40%, infrastructure costs by 35%, and maintaining sub-3-second analytical queries. ● Established an end-to-end data quality and observability framework with schema enforcement, automated validation, Azure Monitor, Log Analytics, and proactive alerting, reducing production data incidents by 85% and improving operational reliability.

Software Engineer 2 at MAQ Software, Aug 2024 - Mar 2025 · 8 mos, Noida, Uttar Pradesh, India, ● Developed an automated framework for backing up large Power BI datasets to ADLS Gen 2 via ADF, saving 80% in developer effort. ● Delivered business intelligence to 40K+ users by implementing ETL techniques and creating Power BI reports on Azure Databricks. ● Optimized performance of over 10 legacy Databricks SQL and Python notebooks by 60%, enhancing pipeline execution efficiency with PySpark optimizations. ● Improved user experience significantly by reducing page load times of 15 Power BI reports by 90% through DAX best practices. ● Architected intricate data models using SQL, leading to a 30% increase in data processing efficiency and a 20% reduction in time spent on manual data manipulation. ● Designed and implemented Row-Level Security (RLS) for a law firm, enabling country-specific data access and ensuring compliance through dynamic filtering in reports. ● Led cross-functional teams, driving successful execution of multiple projects while maintaining compliance with performance benchmarks (report SLAs < 10 seconds). ● Managing cross-functional teams and cross-vendor projects, adhering to agile methodologies, overseeing teams of 3-15 members. ● Trained and supported on data visualization tools and best practices to 60 business users. ● Collaborated with DBAs to optimize database performance and ensure efficient data storage and retrieval on 17 complex queries. ● Automated complex ETL processes using SSIS packages and SQL jobs, achieving a 62% improvement in data processing efficiency.

Software Engineer -1 at MAQ Software, Jul 2022 - Aug 2024 · 2 yrs 2 mos, Noida, Uttar Pradesh, India, ● Spearheaded the formulation of KPIs, creation of Data Models, and development of Dashboards for Analytical Research. ● Utilized advanced Power BI concepts such as Q&A visuals, composite models, personalization, and Power Apps integration to enable seamless data visualization. ● Deployed complex RLS and OLS security measures in reports to enhance data security and integrity. ● Executed 10+ projects for Director and Executive-level stakeholders, enhancing productivity and reducing turnaround time. ● Prepared data for predictive analytics in various domains including Law Firm, Credit, Risk, and Anomaly Detection. ● Facilitated the migration of 14 Tableau reports to Power BI and 30 Qlik reports to Power BI, employing T-SQL and SSIS packages for ETL transformation. ● Utilized tools such as Tabular editor, SQL Tracer, DAX Studio, and Accessibility Insights to optimize report performance and meet SLAs of below 10 second. ● Innovated an innovative app automating ADO documentation, generating test cases, conducting code reviews, and analyzing SQL, Python, and C# code using Open AI (GPT turbo 3.5 model), .NET Console app and Azure, enhancing developer productivity by 27%.

Software Engineer -1 at MAQ Software, Jul 2022 - Sep 2022 · 3 mos, Hyderabad, Telangana, India

React Devloper Associate at The Entrepreneurship Networks at TEN: Labs, Jun 2021 - Sep 2021 · 4 mos, Satna, Madhya Pradesh, India

GeeksGeeks Campus Connect SPOC,IIIT, Bhagalpur at GeeksforGeeks, Jan 2021 - Jun 2022 · 1 yr 6 mos, Bhagalpur, Bihar, India

President(IIIT-Bhagalpur Student Chapter) at GeeksforGeeks, Dec 2020 - Jul 2021 · 8 mos, Bhagalpur, Bihar, India

Codechef Student Chapter (Lead) at CodeChef, Oct 2020 - Sep 2021 · 1 yr, India

GeeksforGeeks Student Chapter Lead(IIIT,Bhagalpur) at GeeksforGeeks, Sep 2020 - Jul 2021 · 11 mos, Bhagalpur, Bihar, India

Education

Bachelor of Technology, Computer Engineering, Indian Institute of Information Technology Bhagalpur

High School, New Era High School,Panchgani