Overview
Work History
Education
Skills
Timeline
ACHIEVEMENTS
Generic

Darshan A V

Bangalore

Overview

3
3
years of professional experience

Work History

Software Development Engineer

MoEngage
11.2023 - Current
  • Real-Time User Ingestion Pipeline: Built a real-time data ingestion pipeline capturing user collection changes via change streams. Streamed oplog events to Kafka and ingested data into S3 using Apache Flink. Enabled reliable real-time analytics and downstream processing.
  • Centralized Airflow PostgreSQL Cleanup: Developed a centralized Airflow DAG to automate PostgreSQL cleanup across multiple clusters. Enforced retention policies and reclaimed 100GB storage per cluster. Improved database performance and reduced storage overhead using Airflow, Python, and Kubernetes.
  • Change Stream Query Validation: Designed a Kafka-driven validation system to compare legacy and new Trino queries. Automated correctness and performance validation, with results stored in S3. Supported safe query migrations and reduced production risk.
  • Real-Time Data Warehouse Event Ingestion: Redesigned a real-time event ingestion pipeline by separating backdated and live events. Eliminated resource contention and improved scalability and stability. Implemented using Java, Kafka Streams, Kafka Connect, S3, and Kubernetes.
  • On-Call & Platform Support: Provided 24x7 on-call support for data platform services. Maintained reliability of Airflow, Kafka, and real-time pipelines. Monitored and resolved production issues using Apache Flink, Kubernetes, Grafana, PagerDuty, and PostgreSQL.

Software Development Engineer

Jio Platforms Limited
08.2023 - 11.2025
  • MRI Data Anonymisation Pipeline: Created a pipeline to anonymize 10TB of MRI DICOM data using NiFi, Apache Spark, and Kafka, storing the output in Apache Ozone and integrating via Boto3 API. Achieved an 80% reduction in ingestion time compared to previous workflows.
  • Scalable SAP Ingestion Pipeline: Built a real-time pipeline in Scala to ingest change data from 700+ SAP tables using GoldenGate, Kafka, and Spark, encrypting PII via HMAC and storing results in Hive on Ozone.
  • Histopathology Data Ingestion: Engineered a high-throughput pipeline in Python with multi-threading to process and convert 10 PB of .jp2 images to .tiff format, using NiFi, Kafka, and Hadoop/Hive, leveraging multi-stage Docker builds, a lag-based load balancer, and Kubernetes.
  • CT Data Anonymization Pipeline: Automated ETL pipeline to anonymize 15TB of CT data by removing personal information tags in DICOM images, using NiFi, Spark, Kafka, and HDFS.reducing job runtime by 60%.

Education

Bachelor of Technology - Computer Science and Engineering

National Institute of Technology Karnataka
Surathkal, Karnataka
05-2023

Skills

Languages: Java, Scala, Python, SQL, C, JavaScript, Shell scripting

Tools and Technologies: Spring Boot, Apache Spark, Apache Flink, Apache Kafka, AWS, Kafka Connect, MongoDB, Athena, MySql, Hadoop, HDFS, ETL, Airflow, Linux, Docker, Kubernetes, CI/CD, Argo, Git, Grafana, Prometheus

Timeline

Software Development Engineer

MoEngage
11.2023 - Current

Software Development Engineer

Jio Platforms Limited
08.2023 - 11.2025

Bachelor of Technology - Computer Science and Engineering

National Institute of Technology Karnataka

ACHIEVEMENTS

  • Employee Rating: A
  • : Recognized among top 10% of employees with A
  • Rating.
  • Competitive Programming: Solved 1000+ problems on LeetCode, Codeforces, and other platforms.
Darshan A V