Headline

Data Engineer | PySpark, Airflow, AWS | ETL/ELT Pipelines | Scalable Analytics Systems

About

Data Engineer building high-throughput ETL/ELT pipelines on AWS. I design and orchestrate data pipelines that ingest, transform, and deliver analytics-ready datasets at scale. Most recent: built a Sales Analysis Pipeline processing 3M+ rows per batch, reducing manual reporting effort by 70% and enabling real-time business intelligence. What I do: • Engineer end-to-end data pipelines using PySpark and Airflow (TaskFlow API) • Build data ingestion systems that aggregate multi-source APIs (GA, Ads, email services, internal APIs) • Design ETL workflows that cut report generation time by 80%+ while improving data quality • Architect monitoring and structured logging for pipeline reliability and observability What I solve for: Data accessibility: Turn fragmented, real-time data sources into unified, trusted datasets Speed: Automate reporting workflows that took hours, now running in minutes Reliability: Implement data quality checks, alerting, and failure detection that catch issues before they hit analytics Technical Stack: Python, PySpark, Airflow, AWS (S3, EC2, RDS, Lambda), Pandas, Grafana, structured logging, data quality frameworks Currently: Open to Data Engineer roles at companies building data infrastructure, analytics platforms, or data-driven products. Interested in: ML infrastructure, real-time streaming, data warehouse optimization, and solving data engineering at scale.

Location

Patna

Work experience

Software Engineer - SDE 1 (Data Infrastructure) at Zerberus.ai, Feb 2025 - Sep 2025 · 8 mos, London, England, • Designed compliance data architecture: Built FastAPI backend service to aggregate, normalize, and serve compliance check results and policy generation outputs; architected data models to support real-time querying across 10K+ compliance rules. • Implemented data ingestion pipeline: Created service to stream compliance check execution results into structured storage; enabled audit trails and compliance reporting queries that scaled to 100K+ daily events. • Optimized data query performance: Reduced compliance check lookup latency by 40% through indexing strategy and query optimization; improved end-user experience for policy generation product.

Data Engineer (Associate) at Taiyō.AI, Aug 2024 - Nov 2024 · 4 mos, Berkeley, California, United States, • Built data processing pipeline for ML training data: Engineered Python scripts to extract, clean, and prepare training data from multiple sources; processed and validated 500K+ records to ensure data quality for model training. • Optimized data pipeline performance: Refactored data processing workflows to reduce execution time by 60%, enabling faster iteration cycles for model experimentation and improving CI/CD pipeline throughput.

Back End Developer at Xamtac Consulting, Feb 2023 - Aug 2024 · 1 yr 7 mos, • Architected Grafana-based monitoring and observability: Built structured logging framework and monitoring dashboard to track pipeline health, data freshness, and API performance; reduced failure detection time from 6+ hours to 15 minutes (80% improvement), enabling faster incident response. • Optimized API and database queries: Analyzed slow queries and API bottlenecks; implemented caching strategy and query optimization that reduced latency by 45% and improved system throughput.

Backend Developer Intern at Xamtac Consulting, Feb 2023 - May 2023 · 4 mos, Chicago, Illinois, United States

Education

Btech(CSE), cse, Amity University Patna