Rishabh D
Bangalore,
Summary
Results-driven Data Engineer with a successful track record in the internet industry. Demonstrated expertise in the Data Engineering tech stack, including PySpark, Apache Pinot, Python, SQL, Kafka, Databricks, and Apache Airflow. Proven ability to navigate large datasets and design scalable solutions, contributing to the success of data-driven initiatives in dynamic environments.
Overview
7
7
years of professional experience
1
1
Certification
Work History
Senior Data Engineer
MiQ Digital
Bengaluru
05.2024 - Current
- Led the Campaign Management capability, spearheading the design, development of the Data engineering charter as a part of broader re-architecture of the in-house tool - LAB.
- Built reusable, maintainable components leading to quicker onboarding of DSP, Adserver data onto the Lake leading to upwards of 50% efficiency in terms of engineering bandwidth.
- Integrated StarTree (Apache Pinot) for user facing analytics by framing optimal queries for various use-cases fully load tested achieving significant gains in terms of throughput, latency.
- Mentored a team of 4 leading to quick dev onboarding to the team as well as achieving overall velocity of the capability's deliverables.
Data Engineer
Razorpay Software Private Limited
Bengaluru
07.2022 - 05.2024
- Built SQL and Python DBT models for core payment flow holistic views separating out transformation layer leading to 50% improvement in backfill process and debugging data issues.
- Ingested data from various third party sources into S3 Data Lake which led to 80% reduction in time to make the data accessible across catalogs like delta and hive compared to the older API pull flow.
- Experience with Kafka to produce erroneous events from a Spark application to Lake and pushing them to Prometheus leading to 30% improvement in data debugging and error detection.
- Migrated various Spark jobs from Databricks to AWS EMR.
Data Engineer
Enquero (A Genpact Company)
Bengaluru
03.2021 - 07.2022
- Extracted expenses data using Python parallelism and concurrency which led to 400x faster data ingestion.
- Developed data pipelines in the People, Finance and L&D domains including implementation of SCD types, Dimensional Bus matrix, frameworks leading to 50% quicker onboarding of data.
- Created data sources including snapshots, consumption views using complex SQL queries for the BI team to consume and build reports upon.
Product Engineer
CBREX Technologies
Bengaluru
08.2019 - 01.2021
- Designed database schemas and involved in writing complex SQL queries for analytics report generation.
- Involved in driving the tech stack to Spring Boot, developing REST APIs from scratch, and also writing the authentication and authorization for the same.
RPA Developer Intern
Unilever Industries
Bengaluru
01.2019 - 06.2019
- Worked as a RPA developer using automation tools for automating tasks. Involved in optimal VM scheduling of RPA processes.
Education
Bachelor of Engineering - Computer Science
C.M.R Institute of Technology
06.2019
Skills
- PySpark
- Apache Pinot
- Databricks
- SQL
- Python
- Apache Airflow
- Apache Kafka
- Trino
- DBT
- Apache Hadoop
- Spark Streaming
- Spark Internals
- AWS
Certification
- Big Data Engineering Certification by TrendyTech, 01/12/20 - 01/05/21
- Coursera - Modernizing Data Lakes and Data Warehouses with Google Cloud, 01/07/23 - 01/07/23
- Coursera - Google Cloud Big Data and Machine Learning Fundamentals, 01/07/23 - 01/07/23
- Coursera - Building Batch Data Pipelines on Google Cloud, 01/08/23 - 01/08/23
- Coursera - Building Resilient Streaming Analytics Systems on Google Cloud, 01/09/23 - 01/09/23
Timeline
Senior Data Engineer
MiQ Digital
05.2024 - Current
Data Engineer
Razorpay Software Private Limited
07.2022 - 05.2024
Data Engineer
Enquero (A Genpact Company)
03.2021 - 07.2022
Product Engineer
CBREX Technologies
08.2019 - 01.2021
RPA Developer Intern
Unilever Industries
01.2019 - 06.2019
Bachelor of Engineering - Computer Science
C.M.R Institute of Technology