Headline

Data Engineer Intern @ Nielsen | PySpark, AWS, Airflow, Kafka | B.Tech CSE ’26

About

Data Engineer building production-grade pipelines at Nielsen. I work with PySpark, AWS (EMR, EKS, S3, Step Functions), Airflow, and Grafana — migrating legacy ETL systems to modern cloud-native architectures. What I've done: Migrated 15+ Talend jobs to PySpark, built pipelines handling 2M+ records/day, designed data quality frameworks catching 95% of anomalies, and reduced infrastructure costs by 30%. Tech: Python | SQL | PySpark | AWS | Airflow | Docker | Kubernetes | Grafana Open to full-time Data Engineer / Backend Engineer roles starting mid-2026. Let's connect.

Location

Bengaluru

AI stack

hugging face

langchain

ollama

openai

Work experience

Data Engineer internship at Nielsen, Mar 2025 - Present · 1 yr 4 mos, • Migrating 15+ legacy Talend ETL jobs to PySpark on AWS EMR on EKS, reverse-engineering undocumented workflows and reducing infrastructure costs by 30% and job runtime by 45%. • Engineered batch data processing pipelines handling 2M+ records daily across 3 input sources, implementing partitioning and caching strategies that improved Spark job performance by 40%. • Orchestrated end-to-end pipeline execution using AWS Step Functions with 5-stage state machines, retry logic, and error handling, achieving 99.5% daily run reliability. • Built real-time monitoring with Grafana dashboards tracking 12+ pipeline metrics and configured Airflow DAGs for scheduled execution and alerting across S3 data lake layers. • Developed a data quality framework analyzing 4 months of production data across 20+ columns, designing detection strategies for gradual drift vs sudden drops that caught 95% of anomalies pre-production. • Collaborated with a team of 8 engineers to map Talend workflow logic to PySpark transformations, documenting 10+ migration patterns adopted as team-wide best practices.

Education

Bachelor of Technology - BTech, Computer Engineering, Lovely Professional University