Headline

Azure Data Engineer | Databricks · PySpark · ADLS Gen2 · SQL · Medallion Architecture · Delta Lake | Capital Markets | 3+ Yrs @ TCS | Immediate Joiner

About

Data Engineer with 3+ years of experience at Tata Consultancy Services, specializing in building and deploying production-grade ETL/ELT pipelines on Azure Databricks for a capital markets client following the Medallion Architecture. My core work revolves around designing Bronze, Silver, and Gold layer pipelines using PySpark, Spark SQL, and Delta Lake that includes handling incremental data ingestion from ADLS Gen2 via Auto-Loader and from Azure SQL Server, with all credentials securely managed through Azure Key Vault. I build transformation logic across all layers, from raw data cleaning, validation, and merge logic in Bronze and Silver, to business-level aggregations, joins, and window functions powering the trading and reporting desk in Gold and orchestrate everything using Databricks Workflow Jobs with centralized logging, retry logic, and automated email alerting. On the deployment side, I manage the complete CI/CD lifecycle using Azure DevOps that feature branching, Pull Request reviews, and 4-stage release approvals across Dev, Test, Stage, and Production environments. A CI/CD governance model I helped establish was later adopted as a standard across project squads at TCS. I also use Python for utility functions like null handling, special character replacement, and exception management alongside my PySpark work. Prior to my capital markets project, I worked on a Databricks migration engagement which includes rebuilding legacy ETL workflows into the Medallion Architecture, which cut data processing time by 35% and established reusable patterns for credential management that the broader team adopted as standard. I have worked in 10-day Agile sprints, collaborating with business analysts, testing teams, ADF ingestion engineers, and client-side senior developers throughout the delivery lifecycle with zero SLA breaches maintained over 18 consecutive months in production. 🎓Microsoft Certified: Azure Data Fundamentals (DP-900) Open to Azure Data Engineer opportunities across diverse domains and industries

Location

Mumbai

Work experience

Azure Data Engineer at Tata Consultancy Services, Feb 2024 - Sep 2025 · 1 yr 8 mos, India, • Designed and delivered 20+ production-grade ETL/ELT pipelines across bronze, silver, and gold layers in Azure Databricks using PySpark and Spark SQL, enabling daily Gold-layer analytics for the capital markets trading and reporting desk. • Built Auto-Loader ingestion from ADLS Gen2 for CSV and Parquet files and Azure SQL Server; secured all credentials via Azure Key Vault Job_common_Notebooks — achieving zero credential exposure in production. • Reduced downstream data defects by 30% by implementing merge logic, window functions, aggregations, and standardized Spark SQL data quality validation frameworks. • Cut pipeline failure detection time from 2+ hours to under 15 minutes via Databricks Workflow Jobs with centralized logging, retry logic, and automated email alerting. • Led CI/CD releases via Azure DevOps with feature branching, PR reviews, and 4-stage approvals (Dev → Test → Stage → Prod), maintaining zero unplanned production outages. • Resolved 15+ ADF production incidents with ingestion team, reducing pipeline downtime by 20% and maintaining zero SLA breaches over 18 months.

Data Engineer at Tata Consultancy Services, Aug 2022 - Jan 2024 · 1 yr 6 mos, Mumbai, • Cut data processing time by 35% by migrating legacy ETL workflows to Databricks Medallion Architecture, rebuilding Bronze, Standard View, and silver pipelines in PySpark, Spark SQL, and Auto-Loader. • Established reusable Job Common Notebook pattern for Azure Key Vault credential management — adopted as team standard across all migrated pipelines. • Enabled root-cause analysis in under 10 minutes for any failure via structured logging and Databricks Workflow Job orchestration with granular audit trails. • Deployed all changes via Azure DevOps CI/CD with feature branching and PR review gates across Dev, Test, Stage, and Production environments.

Associate System Engineer at Tata Consultancy Services, Feb 2022 - Jul 2022 · 6 mos, Mumbai, • Completed TCS onboarding and ILP training • Performed SQL-based ETL, data extraction and validation (joins, aggregations, subqueries) using MySQL and SSMS. • Supported ETL pipeline design and data reconciliation, transitioning into a full data engineering role

Education

Bachelor of Engineering - BE, Mechanical Engineering, North Maharashtra University