Sujoy Das profile photo 1 on Tal

Sujoy Das

Freelance Azure & Databricks Data Engineer | Building Reliable ETL, Lakehouse and Streaming Pipelines | PySpark, ADF, SQL, Kafka

Bengaluru

5+ years of work experience

Ascendion
Jun 20264 mosPresent

Bengaluru

Working as an Azure and Databricks Data Engineer on an enterprise manufacturing and traceability platform. • Built a Databricks App with Streamlit for user-driven data input and dynamic traceability report generation. • Developed ingestion pipelines across BigQuery, SFTP, XML, Oracle, and Delta Lake. • Applied incremental loading and secure cross-cloud connectivity patterns. • Supported Kafka-based near-real-time ingestion and streaming workflow monitoring. • Orchestrated production pipelines and dependencies using Automic. • Implemented data quality and reliability controls for downstream reporting. • Used Lakeflow Connect to integrate Databricks tables with Oracle systems.
Arthya Tech Solutions Private Limited - Schreiber ICC
Mar 20242 yrs 4 mosJun 2026

Bengaluru

Product Lifecycle Management (PLM): • Engaged in migration of data from legacy warehouse by utilizing Qlik Replicate as well as Azure Data Factory for product life cycle management (PLM). • Migrated US and Global PLM dashboards to Power BI, improving reporting efficiency and repointed data connections to Azure Databricks in preparation for the retirement of Schreiber's Oracle BAW (Business Analytics Warehouse). • Migrated the logic for creating architected tables necessary for the PLM project to Azure Databricks and integrated these tables into Automic workflows for automated daily execution, improving data management efficiency. • Leveraged Lakehouse architecture within Azure Databricks to analyze large PLM datasets effectively which enhanced our ability to extract valuable insights for project improvement.
Freelance
Feb 20231 yr 1 moFeb 2024

Remote

1. Led the migration of on-premise big data workloads to Azure Databricks, resulting in a 30% improvement in processing times. 2. Implemented data partitioning strategies, significantly improving query performance and enhancing overall data retrieval speed. 3. Revamped existing ETL pipelines in Azure Data Factory, introducing parameterization and dynamic mapping for enhanced flexibility. 4. Achieved a 25% reduction in pipeline execution times by optimizing data flow activities and orchestrating dependencies. 5. Implemented a robust security framework by integrating Azure Key Vault into data pipelines, ensuring the secure management of connection strings and sensitive information. 6. Successfully orchestrated data movement between Azure and AWS cloud platforms, enabling seamless integration of data from diverse sources. 7. Collaborated with external APIs and third-party cloud data lakes, establishing a unified data integration architecture.
Larsen & Toubro
Sep 20225 mosJan 2023

• Developed and implemented Azure Data Factory pipelines to automate data ingestion per week, from 4 project sites through HTTPS eradicating complete manual workload • Defined and ensured data quality checks within Azure Data Factory, ensuring data compliance and facilitating accurate analysis • Cleaned and joined diverse datasets using Azure Databricks to extract useful insights • Loaded final data into SQL Database for access by data analysts, empowering informed decision-making based on data-driven insights • Partnered closely with data analysts to track work progress and resource consumption, fostering collaboration and optimizing data analysis workflows, resulting in 40% improved workflow
Larsen & Toubro
Aug 20211 yr 1 moAug 2022

• Technical documentation of ₹200Cr project for complex electrical systems, ensuring clear communication across different teams and stakeholders. • Spearheaded initiatives that increased recievables by 50% across two consecutive quarters, securing a combined improvement of ₹30 crore and optimizing cash flow. • Analyzed project documentation and implemented streamlined workflows and strategies, accelerating project execution by 40% compared to standard methods. • Successfully led a team of 5 in deploying pilot digital initiatives developed by R&D, fostering collaboration and driving successful integration of new technologies. • Initiated and implemented project management knowledge, leading to standardized practices across the team which improved project success rates by 10%. • Took part in developing project cost estimates and resource allocation plans, enabling effective budget forecasting and adherence, and improving financial control.
Sep 20204 mosDec 2020

Remote

• Prepared and implemented lesson plans covering required course topics. • Diversified teaching techniques and learning tools help students with differing ability levels and varied learning modalities. • Mentored students on individual basis to help develop math skills. • Encouraged critical thinking and problem-solving to help students develop mathematical reasoning. • Achieved conversion of 5 demo sessions into full time courses.

Sujoy’s AI Stack

claude code
claude code
gemini
gemini
github copilot
github copilot
hugging face
hugging face
openai
openai
ChatGPT
ChatGPT

Education

Indian Institute of Technology, Delhi

Master of Technology - MTech, Construction Management

Indian Institute of Technology, Delhi

20192 yrs2021

Masters, Big Data Analytics

TrendyTech

20231 yr2024

Join Sujoy and 10k+ other professionals on

Tal Boss

Tal is the app where bosses meet engineers & builders directly, without middlemen

Download Tal Boss

Sujoy Das on Tal

Headline

Freelance Azure & Databricks Data Engineer | Building Reliable ETL, Lakehouse and Streaming Pipelines | PySpark, ADF, SQL, Kafka

About

I am an Azure and Databricks Data Engineer with 4+ years of experience building, modernizing, and optimizing enterprise data platforms. I help organizations turn data from APIs, SFTP, XML, relational databases, BigQuery, and event streams into reliable, analytics-ready datasets. My core areas of work include: • Azure Databricks and PySpark development • Azure Data Factory pipeline design and modernization • Delta Lake and Lakehouse implementation • Batch and near-real-time data ingestion • SQL and Spark performance optimization • Cross-cloud ingestion from Google BigQuery to Azure • XML and SFTP integration • Data quality, incremental loading, orchestration, and monitoring • Azure Key Vault and secure pipeline configuration • Streamlit and Databricks Apps for operational data applications I am available for freelance and consulting engagements involving ETL development, Databricks optimization, ADF troubleshooting, Lakehouse architecture, migration to Azure, and streaming pipeline prototypes. Email: sujoy2412@gmail.com

Location

Bengaluru

AI stack

claude code

gemini

github copilot

hugging face

openai

ChatGPT

Work experience

Senior Associate Engineer at Ascendion, Jun 2026 - Present · 4 mos, Bengaluru, Working as an Azure and Databricks Data Engineer on an enterprise manufacturing and traceability platform. • Built a Databricks App with Streamlit for user-driven data input and dynamic traceability report generation. • Developed ingestion pipelines across BigQuery, SFTP, XML, Oracle, and Delta Lake. • Applied incremental loading and secure cross-cloud connectivity patterns. • Supported Kafka-based near-real-time ingestion and streaming workflow monitoring. • Orchestrated production pipelines and dependencies using Automic. • Implemented data quality and reliability controls for downstream reporting. • Used Lakeflow Connect to integrate Databricks tables with Oracle systems.

Data Engineer at Arthya Tech Solutions Private Limited - Schreiber ICC, Mar 2024 - Jun 2026 · 2 yrs 4 mos, Bengaluru, Product Lifecycle Management (PLM): • Engaged in migration of data from legacy warehouse by utilizing Qlik Replicate as well as Azure Data Factory for product life cycle management (PLM). • Migrated US and Global PLM dashboards to Power BI, improving reporting efficiency and repointed data connections to Azure Databricks in preparation for the retirement of Schreiber's Oracle BAW (Business Analytics Warehouse). • Migrated the logic for creating architected tables necessary for the PLM project to Azure Databricks and integrated these tables into Automic workflows for automated daily execution, improving data management efficiency. • Leveraged Lakehouse architecture within Azure Databricks to analyze large PLM datasets effectively which enhanced our ability to extract valuable insights for project improvement.

Data Engineer at Freelance, Feb 2023 - Feb 2024 · 1 yr 1 mo, Remote, 1. Led the migration of on-premise big data workloads to Azure Databricks, resulting in a 30% improvement in processing times. 2. Implemented data partitioning strategies, significantly improving query performance and enhancing overall data retrieval speed. 3. Revamped existing ETL pipelines in Azure Data Factory, introducing parameterization and dynamic mapping for enhanced flexibility. 4. Achieved a 25% reduction in pipeline execution times by optimizing data flow activities and orchestrating dependencies. 5. Implemented a robust security framework by integrating Azure Key Vault into data pipelines, ensuring the secure management of connection strings and sensitive information. 6. Successfully orchestrated data movement between Azure and AWS cloud platforms, enabling seamless integration of data from diverse sources. 7. Collaborated with external APIs and third-party cloud data lakes, establishing a unified data integration architecture.

Data Engineer at Larsen & Toubro, Sep 2022 - Jan 2023 · 5 mos, • Developed and implemented Azure Data Factory pipelines to automate data ingestion per week, from 4 project sites through HTTPS eradicating complete manual workload • Defined and ensured data quality checks within Azure Data Factory, ensuring data compliance and facilitating accurate analysis • Cleaned and joined diverse datasets using Azure Databricks to extract useful insights • Loaded final data into SQL Database for access by data analysts, empowering informed decision-making based on data-driven insights • Partnered closely with data analysts to track work progress and resource consumption, fostering collaboration and optimizing data analysis workflows, resulting in 40% improved workflow

Senior Engineer at Larsen & Toubro, Aug 2021 - Aug 2022 · 1 yr 1 mo, • Technical documentation of ₹200Cr project for complex electrical systems, ensuring clear communication across different teams and stakeholders. • Spearheaded initiatives that increased recievables by 50% across two consecutive quarters, securing a combined improvement of ₹30 crore and optimizing cash flow. • Analyzed project documentation and implemented streamlined workflows and strategies, accelerating project execution by 40% compared to standard methods. • Successfully led a team of 5 in deploying pilot digital initiatives developed by R&D, fostering collaboration and driving successful integration of new technologies. • Initiated and implemented project management knowledge, leading to standardized practices across the team which improved project success rates by 10%. • Took part in developing project cost estimates and resource allocation plans, enabling effective budget forecasting and adherence, and improving financial control.

Mathematics Tutor at KidAptivity Technologies, Sep 2020 - Dec 2020 · 4 mos, Remote, • Prepared and implemented lesson plans covering required course topics. • Diversified teaching techniques and learning tools help students with differing ability levels and varied learning modalities. • Mentored students on individual basis to help develop math skills. • Encouraged critical thinking and problem-solving to help students develop mathematical reasoning. • Achieved conversion of 5 demo sessions into full time courses.

Education

Master of Technology - MTech, Construction Management, Indian Institute of Technology, Delhi

Masters, Big Data Analytics, TrendyTech