Headline

Data Engineer | Azure & Databricks | ADF, ADLS, PySpark, SQL | TCS | Databricks Certified Data Engineer Associate

About

I am a Data Engineer specializing in building scalable, production-grade data pipelines on the Azure and Databricks ecosystem, with experience in delivering end‑to‑end solutions for enterprise clients at Tata Consultancy Services (TCS). I work across the full SDLC, from requirement analysis and design to development, testing, deployment and post‑production support, ensuring reliable and high‑quality data delivery for analytics and reporting. In my current role as a Data Engineer, I design and implement reusable data quality rules, automated validation checks and medallion architecture‑based pipelines (Bronze–Silver–Gold) in Databricks, achieving up to 99.9% data accuracy and reducing downstream issues. I orchestrate multi‑source data ingestion using Azure Data Factory and Databricks Jobs, integrating 15+ heterogeneous sources (REST APIs, flat files, relational DBs, streaming feeds) while meeting strict SLAs for batch and incremental loads. My technical toolkit includes Azure Data Factory, Azure Data Lake Storage, Azure Synapse, Databricks (Delta Lake, Unity Catalog, Auto Loader), Apache Spark, PySpark, SQL, Python and Kafka. I focus on performance optimization (partitioning, broadcasting, autoscaling) and robust governance using Unity Catalog and Azure RBAC to ensure secure, compliant and scalable data platforms. I have also worked as a Data Analyst Intern, where I used SQL, Adobe Analytics, Power BI and Azure Data Lake to analyze large‑scale datasets, automate reporting and deliver insights that improved campaign performance and operational efficiency. This blend of engineering and analytics helps me understand both data infrastructure and business outcomes.

Location

Pune District

Work experience

Big Data Engineer at Tata Consultancy Services, Jun 2023 - Present · 3 yrs, Pune District, Maharashtra, India,  Engineered a robust ETL pipeline using Apache Spark and Kafka, processing 500+ million records daily with 99.99% data accuracy and reduced data latency by 60% for real-time analytics.  Engineered scalable data pipelines integrating data lakes, data warehouses, and cloud components. Optimized performance by reducing data scanned by 3x using Parquet and columnar formats, significantly cutting storage and query costs.  Designed and maintained DAGs in Apache Airflow to orchestrate complex ETL workflows, reducing manual intervention by 90% and improving pipeline reliability and scheduling efficiency.  Boosted Spark pipeline efficiency by leveraging broadcast joins, caching, and predicate pushdown, cutting processing time by 30% for daily workloads exceeding 2 million records.  Achieved a 40% increase in Kafka data ingestion speed, optimizing the streaming pipeline, and reducing overall processing time by 25% across various data pipeline components.  Led performance tuning initiatives that optimized queries to yield a 5x improvement in data processing speed.  Achieved and maintained an impressive 99.8% data pipeline up-time by leveraging Spark and Python to ingest streaming and transactional data from 8 primary sources, ensured seamless data flow and minimized disruptions.  Actively contributed to Agile-based delivery methodology – participated in sprint planning, daily standups and retrospectives; delivered features iteratively while maintaining alignment with business goals and technical roadmap.

Trainee at PW (PhysicsWallah), Mar 2023 - Jun 2023 · 4 mos, Remote, Built dashboard in Power BI/Excel to track performance trends, Documented analytics process and presented insights.

Data Analyst at Tata Consultancy Services, Jan 2023 - Mar 2023 · 3 mos, Analyzed customer purchase behavior and engagement patterns across client devices using Adobe Analytics, SQL, and Power BI, delivering actionable insights that refined campaign targeting and enhanced product engagement by 20%. Utilized Azure Data Lake to process and manage large-scale datasets up to 3 TB. Conducted exploratory data analysis using Python, Azure Data Explorer (Kusto) and Insight Analytics to uncover key trends, detect anomalies and provide actionable insights. Streamlined and automated reporting workflows, reducing manual effort by 10 person-hours per month and enhancing overall operational efficiency. Developed 5+ interactive Excel dashboards using complex datasets to track key performance indicators, providing 10 stakeholders with actionable insights that boosted decision-making speed by 15%. Delivered incremental analysis to evaluate the effectiveness of offers and promotions, utilizing statistical data models to compare performance across test and control groups, resulting in a 15% uplift in conversion rates. Analyzed large datasets with SQL to identify trends and draw insights, resulting in a 29% increase in operational efficiency. Prepared and delivered weekly/monthly business performance reports to stakeholders, highlighting key trends, insights and actionable recommendations, enabling data-driven decisions and improving reporting efficiency by 20%.

Education

Bachelor of Technology - BTech, electrical engineering, samrat ashok technological institute

Bachelor of Technology, Electrical and Electronics Engineering, E Cell SATI