---
title: "Amit Kumar Soni - Principal Data Engineer in Bengaluru | Tal"
description: "Meet Amit Kumar Soni, Principal Data Engineer | Architecting High-Throughput Streaming & Distributed Ingestion Frameworks | Scala • Java • Python • Spark • Kafka • Kubernetes | Ex-Mercedes, Tesco, Dell Hyper-Scale Data Platforms based in Bengaluru. AI stack includes claude code, codex, and github c..."
image: "https://tal.club/meet/amit-kumar-soni/share-image"
---

# Amit Kumar Soni on Tal

## Headline

Principal Data Engineer \| Architecting High\-Throughput Streaming & Distributed Ingestion Frameworks \| Scala • Java • Python • Spark • Kafka • Kubernetes \| Ex\-Mercedes, Tesco, Dell Hyper\-Scale Data Platforms

## About

Principal Data Engineer with 13\+ years of expertise designing, developing, and operating enterprise\-scale distributed data platforms and high\-throughput ingestion architectures\. Specialized in engineering production\-hardened streaming grids and decoupled multi\-tenant data layers capable of processing high\-frequency server telemetry and complex transactional logs at scales exceeding millions of events per second \(EPS\)\. Deep technical authority in the Apache Spark, Apache Kafka, and cloud\-native ecosystems \(Kubernetes, Docker, Azure\)\. Highly skilled in identifying and permanently eliminating engine\-level processing bottlenecks—including severe data skew, shuffle memory thrashing, and serialization overhead—to drastically compress compute footprints and enforce strict near\-real\-time SLAs\. Combines low\-level distributed systems engineering and hands\-on optimization with technical roadmap strategy, cross\-team architecture governance, and engineering mentorship\. Actively extending data boundaries by designing type\-safe, asynchronous REST data planes in Scala, remote execution loops via Ammonite, and structured, deterministic Agentic AI data orchestration layers\. Core Technical Competencies: • Core Data Engineering: Distributed Systems Design, Multi\-Tenant Platforms, Event\-Driven Streaming, High\-Throughput Batch Infrastructures, Cloud\-Native Topologies, Large\-Scale Data Migration\. • Data Processing & Storage Formats: Apache Spark \(Core Execution / Partition Tuning\), Spark Structured Streaming, Apache Kafka \(High\-Throughput Topology Tuning\), Custom Kafka Connectors, Delta Lake, Hadoop HDFS, Presto, Hive\. • Programming Ecosystems: Scala, Core Java, Python, SQL, Advanced Linux Shell \(Bash\), Ammonite\. • Infrastructure & Platform Observability: Kubernetes, Docker, Microsoft Azure, Azure Databricks, Unity Catalog Governance, ELK Stack \(Elasticsearch, Logstash, Kibana\), Automated Telemetry\. • AI Data Engineering: Structured LLM Orchestration Patterns, Pydantic Contracts, Hybrid Retrieval & Reranking Frameworks, RAG Pipelines\.

## Location

Bengaluru

## AI stack

claude code

codex

github copilot

langgraph

ollama

OpenAI Agents SDK

## Work experience

Principal Data Engineer at Kyndryl India, Jul 2022 \- Present · 4 yrs 5 mos, Bengaluru, Karnataka, India, • Architected and defined the technical roadmap for multi\-tenant enterprise data products utilizing Scala, Spark, and Delta Lake, seamlessly decoupling complex real\-time Kafka streams and legacy transactional engines\. • Compressed production data processing runtimes by 62\.5% \(from 4 hours to 90 minutes\) by re\-engineering distributed cluster topologies and optimizing partitioning schemas for high\-cardinality transactional log streams\. • Reduced production incident triage and recovery times by 60% by designing cross\-platform resilience mechanisms, strict data\-validation gates, and unified telemetry topologies across multi\-tenant data platforms\. • Slashed manual infrastructure deployment tasks by 75% \(saving 26 hours annually\) by engineering fully automated Azure infrastructure triggers and system deployment alerts using Bash\. • Reduced upstream data egress latency to near\-real\-time across the Tesco Common Egress Framework \(CEF\) by architecting custom Apache Kafka Connectors to ingest diverse data streams and standardize message schemas across heterogeneous sources\. • Eliminated configuration drift and cut deployment overhead by building a type\-safe Scala REST API control plane, an automated Scala CLI thin client, and Ammonite remote execution loops integrated with the ApiCurio Schema Registry\. • Engineered a high\-concurrency, multi\-threaded near\-real\-time data distribution backend using Java and Scala, leveraging Spark Structured Streaming to deliver low\-latency analytical endpoints that abstract complex streaming pipelines\. • Enforced centralized security controls across consumer touchpoints, standardizing OAuth2 authentication, strict RBAC authorization boundaries, real\-time backend audit logging, and containerized Docker execution profiles\. • Architected a repository Q&A and code\-intelligence platform utilizing hybrid retrieval \(dense/sparse vector indexes\), reranking frameworks, and deterministic LLM orchestration loops enforced via Pydantic\.

Senior Team Lead at Persistent Systems, Mar 2020 \- Jul 2022 · 2 yrs 5 mos, Bengaluru, Karnataka, India, • Scaled cloud\-native data ingestion to handle multi\-terabyte telemetry footprints from Dell Support Assist Cloud servers, managing peak throughputs exceeding millions of events per second \(EPS\) via Apache Spark grids deployed on Kubernetes\. • Eliminated severe memory thrashing and network serialization bottlenecks caused by heavily nested JSON server telemetry data structures; tuned custom Spark partitioning and cluster resource limits to permanently eradicate data skew and shuffle thrashing\. • Constructed production\-hardened component frameworks featuring custom validation logic, failure isolation gates, and transactional boundaries over Kubernetes clusters to protect downstream databases from corrupted analytical payloads\. • Engineered centralized logging and proactive observability architectures by integrating the ELK Stack \(Elasticsearch, Logstash, Kibana\), optimizing telemetry index search speeds and shrinking pipeline troubleshooting cycles\.

Senior Data Engineer at Tech Mahindra, Sep 2017 \- Mar 2020 · 2 yrs 7 mos, Bengaluru, Karnataka, India, • Analyzed business requirement documents and collaborated with multiple teams to close gaps and meet business needs\. • Migrated data from diverse sources to HAAS, implementing configuration changes for scalable Big Data solutions across multiple portfolios\. • Architected and automated existing data flows, improving efficiency and reducing manual intervention\. • Developed reporting systems and Hive queries to extract business\-critical data points\. • Wrote Shell and Python scripts to automate workflows via Control\-M\. • Built and optimized Spark workflows using Scala and PySpark for large\-scale data processing and automation\.

Senior Software Engineer at Big Data BizViz, Jan 2017 \- Jul 2017 · 7 mos, Bengaluru, Karnataka, India, • Developed new APIs and features for existing products, enhancing functionality and customer experience\. • Customized and integrated product solutions for multiple clients using product APIs, ensuring high adoption and satisfaction\. • Designed and implemented Java APIs and UI components for seamless end\-to\-end solutions\. • Developed optimized Hive queries, SQL, and PL/SQL / PostgreSQL scripts for data loading, transformation, and customization from Oracle and PGSQL databases\. • Collaborated with professional services teams in customer\-facing projects, delivering scalable and reliable data solutions\.

Senior Software Engineer at Accenture, Jun 2015 \- Jan 2017 · 1 yr 8 mos, Bangalore, • Built scalable data pipelines using Pig, Hive, MapReduce, and Sqoop, processing large structured & unstructured datasets\. • Developed custom UDFs and optimized Hive queries, improving data processing efficiency\. • Automated workflows and reporting via Oozie and Bash, enhancing operational efficiency\. • Collaborated with cross\-functional teams to ensure data quality, availability, and timely delivery for client requests\.

## Education

Bachelor of Engineering, Computer Science and Engineering, Bansal Institute Of Research And Technology

[View the original profile](https://tal.club/meet/amit-kumar-soni)

```json
{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "Person",
      "@id": "https://tal.club/meet/amit-kumar-soni#person",
      "name": "Amit Kumar Soni",
      "url": "https://tal.club/meet/amit-kumar-soni",
      "image": "https://tal.club/meet/amit-kumar-soni/profile-photo.jpg",
      "description": "Meet Amit Kumar Soni, Principal Data Engineer | Architecting High-Throughput Streaming & Distributed Ingestion Frameworks | Scala • Java • Python • Spark • Kafka • Kubernetes | Ex-Mercedes, Tesco, Dell Hyper-Scale Data Platforms based in Bengaluru. AI stack includes claude code, codex, and github c...",
      "jobTitle": "Principal Data Engineer",
      "worksFor": {
        "@type": "Organization",
        "name": "Kyndryl India"
      },
      "hasOccupation": {
        "@type": "Occupation",
        "name": "Principal Data Engineer"
      },
      "homeLocation": {
        "@type": "Place",
        "name": "Bengaluru"
      },
      "alumniOf": [
        {
          "@type": "EducationalOrganization",
          "name": "Bansal Institute Of Research And Technology"
        }
      ],
      "knowsAbout": [
        "claude code",
        "codex",
        "github copilot",
        "langgraph",
        "ollama",
        "OpenAI Agents SDK"
      ],
      "mainEntityOfPage": {
        "@id": "https://tal.club/meet/amit-kumar-soni#webpage"
      }
    },
    {
      "@type": "ProfilePage",
      "@id": "https://tal.club/meet/amit-kumar-soni#webpage",
      "url": "https://tal.club/meet/amit-kumar-soni",
      "name": "Amit Kumar Soni on Tal",
      "description": "Meet Amit Kumar Soni, Principal Data Engineer | Architecting High-Throughput Streaming & Distributed Ingestion Frameworks | Scala • Java • Python • Spark • Kafka • Kubernetes | Ex-Mercedes, Tesco, Dell Hyper-Scale Data Platforms based in Bengaluru. AI stack includes claude code, codex, and github c...",
      "dateModified": "2026-11-11T08:35:56.058Z",
      "isPartOf": {
        "@id": "https://tal.club/#website"
      },
      "mainEntity": {
        "@id": "https://tal.club/meet/amit-kumar-soni#person"
      },
      "primaryImageOfPage": {
        "@type": "ImageObject",
        "url": "https://tal.club/meet/amit-kumar-soni/profile-photo.jpg",
        "contentUrl": "https://tal.club/meet/amit-kumar-soni/profile-photo.jpg",
        "caption": "Amit Kumar Soni profile photo on Tal"
      },
      "inLanguage": "en-IN"
    }
  ]
}
```