Headline

Data Engineer | FinTech · BFSI | Python · SQL · PySpark · Airflow · AWS · GCP | Glue · Lambda | Redshift · BigQurey | ETL/ELT, CDC, Data Warehousing, Data Modeling | Data Quality, Governance & Security | Immediate Joiner

About

Data Engineer, 3 years 3 months, available immediately. Open to Data Engineer and Analytics Engineer roles: PAN India or remote. I build and run production data platforms for microfinance lending and BFSI clients. At Gojo I was one of two data engineers on a seven-client platform and sole owner of five client production environments, all onboarded independently: ingestion, dimensional data modelling, KPI reporting, and a change data capture pipeline behind a live mobile banking app. Small team, real production, independent ownership of every pipeline I shipped. Reliability and data security are where I spend my attention. Idempotent, incremental ingestion backed by watermark stores. Reconciliation gates that halt a bad cycle before it propagates. Distributed locking for concurrency control. Monitoring and alerting that fires before a stakeholder notices. Cross-account data movement secured with allow-listed VPC CIDR blocks, IAM least privilege roles and AWS Secrets Manager. The work I'm proudest of is a defect. Deduplication was keyed on a timestamp column holding identical values for one primary key, so the pipeline quietly kept the wrong record. Nothing errored. Root cause analysis traced it, I re-keyed onto the change sequence column and put a validation gate in front of the watermark. I'm comfortable close to the business too running discovery and requirements gathering with cross-functional stakeholders, translating business requirements into data structures, mappings and transformations, and reconciling internal and external data sources via source-to-target mapping. Documentation and runbooks are part of the job. Core: Python · SQL (window functions, CTEs, BigQuery stored procedures, query optimization) · PySpark · Apache Spark · AWS Glue · AWS Lambda · Amazon Redshift · Amazon S3 · AWS DMS · Amazon RDS · Amazon DynamoDB · Amazon Athena · EventBridge · CloudWatch · IAM · Secrets Manager · VPC · AWS CLI · Apache Airflow · PostgreSQL · MySQL · Google BigQuery · Cloud Composer · Cloud Functions · Looker Studio · Docker · Git · CI/CD · Linux · shell scripting · ETL/ELT · data ingestion · data integration · data transformation · orchestration · CDC · data modelling · OLAP/OLTP · data warehousing · data lake · data mart · medallion architecture · data quality · data validation · data governance · data security · access control · RBAC Domain: BFSI · fintech · lending · microfinance · financial services · core banking · mobile banking Feel free to reach out me out: adityavaidya222000@gmail.com

Location

Indore

AI stack

claude

deepseek

gemini

grok

openai

perplexity

Work experience

Data Engineer at Gojo & Company, Inc., Nov 2023 - Feb 2026 · 2 y 3 mo, Shibuya-ku, Tokyo, Japan, Sole owner of 5 of 7 client environments, as 1 of 2 data engineers. Independently onboarded all 5 onto a medallion architecture spanning 500+ source tables and 20 GB/day using an S3 data lake, AWS DMS, a three-job Glue (PySpark) ETL framework for distributed processing, Glue Data Catalog, Athena and Redshift. Ended duplicate loads and reruns by making incremental ingestion idempotent with DynamoDB and Redshift watermark stores, cutting scan volume via columnar Parquet and partitioning across 5 environments. Sustained sub-5-minute latency at 99% measured uptime behind a live mobile banking app by delivering an event-driven serverless CDC pipeline across 3 environments: DMS to S3, S3 events trigger Lambda, psycopg2 upserts to PostgreSQL on RDS. Kept Lambda inside the AWS always-free tier (1M requests, 400,000 GB-seconds monthly) by micro-batching CDC with boto3 StartAfter, plus a self-proposed DynamoDB lock for concurrency control. Restored data correctness across 3 CDC environments by root-causing a silent defect no alert caught: dedup keyed on a timestamp column with identical values for one key, re-keyed onto the change sequence behind a validation gate. Cut business KPI reporting from 3-4 days to under 3 hours through automation, 10+ reports per client across 5 clients, 20-30 KPIs each, on EventBridge Lambdas; extended to social performance reporting for all 7. Designed the single source of truth for loan and portfolio KPIs across 5 clients, Redshift star schema, SCD Type 2 daily snapshot fact table, denormalized data mart, by running discovery with cross-functional stakeholders across internal and external data sources. Owned production support, monitoring, data governance and data security across 5 client AWS environments: secured cross-account movement with allow-listed VPC CIDR blocks, IAM least privilege roles, RBAC and AWS Secrets Manager, and cut incident response and troubleshooting to under 1 hour via CloudWatch and SNS alerting.

Data Engineer at Canopus Infosystems Pvt. Ltd., Aug 2022 - Jul 2023 · 11 mo, Indore, Madhya Pradesh, India, Sole data engineer on a client analytics engagement, building a governed cloud data warehouse from external SaaS data sources. Consolidated 40+ external Salesforce objects (approx. 2 GB/day) into Google BigQuery by building a serverless Python and Pandas Cloud Function performing chunked REST API integration against the Salesforce Bulk API 2.0 using SOQL, with credentials handled outside code. Cut ingestion scope and storage cost on the Google Ads feed by curating the BigQuery Data Transfer Service pull to the 7 business-relevant tables (approx. 1 GB/day) rather than replicating the full source. The client's marketing analyst reported a 10% uplift in campaign conversion rate from the consolidated data. Improved data integrity across downstream reporting by modeling 3 BigQuery layers, raw, processed and aggregate, with SQL-based deduplication, cleansing and data quality validation logic. Eliminated partial-data risk across all 40+ ingested objects by orchestrating the workflow as an Apache Airflow DAG on Cloud Composer: extraction, a data load verification gate, then aggregation queries in dependency order, with job dependency management throughout. Made business intelligence self-serve across all 3 modeled layers by exposing governed aggregate tables to Looker Studio dashboards, removing the analyst's dependency on engineering for routine reporting.

Education

Bachelor of Technology - BTech, Mechanical Engineering, Sri Aurobindo Institute of Technology, Indore

Public links

leetcode: https://leetcode.com/Aditya_vaidya