Headline

Data Engineer | Software Engineer

About

As a Software Engineer at Teikametrics, I specialize in building and optimizing real-time data pipelines for large-scale e-commerce analytics. Using Scala, Apache Spark Streaming, Databricks, Delta Lake, and SQL, I design scalable incremental data workflows, develop complex data transformations, and improve system reliability, performance, and efficiency. In addition to data engineering, I have built AI-powered analytics systems that help organizations identify operational pain points, uncover anomalies and trends, and better understand the signals within their data. These systems combine large-scale data processing with AI-driven analysis to turn complex datasets into actionable insights, enabling teams to diagnose issues faster, streamline systems, and make more informed decisions.

Location

Bengaluru

AI stack

anthropic apis

claude code

claude

langchain

langfuse

openai

Work experience

Software Engineer at Teikametrics, Jun 2025 - Present · 1 yr 4 mos, Remote, • Built and optimized real-time data pipelines using Scala and Apache Spark Streaming on Databricks for large-scale e-commerce analytics. • Designed incremental and fault-tolerant streaming workflows using checkpointing and Delta Lake. • Implemented complex SQL transformations and aggregations on high-volume datasets. • Optimized Spark jobs to improve performance and reduce compute cost. • Maintained Java-based Maven projects supporting scalable data applications. • Automated deployment and operational tasks using shell scripts and Makefiles.

Associate Engineer at Crayon Data, Aug 2023 - Jun 2025 · 1 yr 11 mos, Chennai, Tamil Nadu, India, • Designed and deployed end-to-end ETL pipelines using Scala and Apache Spark, orchestrated with Airflow and Jenkins. • Built schema auto-mapping frameworks in Python to simplify and speed up client onboarding. • Developed metadata crawlers for S3 to extract Parquet and Delta Lake schemas, improving data governance. • Created NLP-based data validation tools enabling non-technical users to define and enforce dataset rules. • Deployed custom LLMs on Kubernetes with FastAPI and CLI access for scalable inference. • Supported large-scale production pipelines with performance tuning, monitoring, and issue resolution.

Education

Bachelor's degree, Information and Communication Technology, Shanmugha Arts, Science, Technology & Reserch Academy (SASTRA), Thanjavur