Headline

Data Engineer at Comcast

About

Data Engineer with 5 years of experience across data engineering, big data, and software engineering, building large-scale data platforms on Databricks and AWS. Specializes in event-driven streaming ingestion (Spark Structured Streaming, Auto Loader, S3-SNS-SQS), Medallion architecture, and Delta Lake governed by Unity Catalog. Strong in Python, SQL, and PySpark, with proven delivery of ETL/ELT pipelines, data quality and governance, pipeline monitoring, and performance and cost optimization across 50+ production data feeds.

Location

Chennai

AI stack

claude code

claude

gemini

github copilot

grok

ChatGPT

Work experience

Data Engineer 2 at Comcast, Sep 2024 - Present · 2 yrs 2 mos, Chennai, Tamil Nadu, India, I build and operate large-scale data platforms on Databricks and AWS, turning 50+ diverse data feeds into reliable, governed data products. • Engineer event-driven streaming ingestion using Spark Structured Streaming and Databricks Auto Loader with an S3 → SNS → SQS pattern, incrementally landing 50+ data feeds into Delta Lake. • Design and operate Medallion (Bronze / Silver / Gold) pipelines across 40+ Databricks jobs, onboarding sources spanning cloud, identity, endpoint, network, and SaaS using Terraform (IaC). • Build and maintain Vector (VRL) pipelines for collection, transformation, and routing of events between source systems, S3, Splunk, and Databricks. • Tune performance and cost with partitioning, Z-ordering / liquid clustering, Delta optimization, and incremental processing — lowering compute usage and query latency. • Build job and pipeline monitoring for success/failure, freshness, and volume health, cutting incident detection and triage time. • Enforce data quality and governance with schema-drift detection, validation checks, and a central data dictionary for lineage and metadata. • Built a Databricks-native detection framework that replaced legacy Splunk correlation searches — config-driven, versioned rules on a scheduled runner with deduplication, suppression, and audit history. Earlier in this role, I led the data engineering for xAIde, an enterprise AI chatbot: automated multi-source pipelines (SharePoint, GitHub, databases) that cut manual refresh cycles by 90%, plus a hybrid retrieval + dynamic-SQL backend using Azure OpenAI (GPT-4o), LangChain, and RAG over Snowflake with row-level security.

Software Engineer at Gigamon, Aug 2022 - Dec 2023 · 1 yr 5 mos, Chennai, Tamil Nadu, India, Worked on security and cryptography for the GigaVUE product line in a RHEL 8 environment. • Led the OpenSSL migration from 1.0.2 to 3.0.7, resolving compatibility issues and updating dependent open-source packages (curl, OpenLDAP, Apache HTTPd, ssmtp, thrift). • Implemented user-password encryption with OpenSSL and contributed to crypto features (user-selectable SSH crypto options, automated trust-store upgrades). • Remediated CVEs surfaced by Qualys, Tenable, and Black Duck scans, and upgraded open-source packages to keep the product secure and compliant. • Supported the Python 2.7 → 3.8 upgrade and resolved sustenance bugs, internal defects, and Coverity issues. • Maintained GigaVUE images for FIPS certification (BAH, SafeLogic). Recognized with the GigaSTAR Team Award.

Data Engineer at Cognizant, Jul 2021 - Jul 2022 · 1 yr 1 mo, • Built ETL / data-integration workflows with SQL Server Integration Services (SSIS) — extraction, transformation, and loading using data-flow components (Source, Destination, Lookup, Conditional Split, Data Conversion). • Wrote and optimized complex SQL in MS SQL Server; created and managed tables, views, stored procedures, and functions. • Scheduled and automated SSIS packages with SQL Server Agent; applied performance tuning, error handling, and data-quality checks. • Designed Tableau dashboards and reports, and partnered with cross-functional teams to translate requirements into data solutions.

Data Engineer Intern at Cognizant, Feb 2021 - Jul 2021 · 6 mos, • Gained hands-on experience with the Hadoop ecosystem and Apache Spark for processing and analyzing large datasets. • Built database structures and automation scripts with MySQL and Python. • Explored Microsoft Azure (application deployment, virtual machines, storage) and strengthened problem-solving and collaboration in a team environment.

Data Science Intern at LIVEWIRE India, Dec 2019 - Feb 2020 · 3 mos, Chennai, Tamil Nadu, India

Education

Post Graduate Program, Data science and Engineering, Great Lakes Institute of Management

Bachelor of Engineering - BE, Electronics and communication engineering, Sri venkateswara college of engineering