Rahul Gautam
Sr. Data Engineer
Shimla
Summary
Detail-oriented data Engineer designs, develops and maintains highly scalable, secure and reliable data structures. Accustomed to working closely with system architects, software architects and design analysts to understand business or industry requirements to develop comprehensive data models. Proficient at developing database architectural strategies at the modeling, design and implementation stages.
Overview
9
9
years of professional experience
4
4
years of post-secondary education
Work History
Sr. Data Engineer
Apollo.io
08.2022 - Current
- Built ETL framework to ingest data from Application database, in batch, into data warehouse, enabling analytics and reporting on Application datasets.
- Designed and Implemented config driven ingestion framework to get data from External data source, via APIs.
- Implemented the ELT via DBT.
- Implemented the data quality monitoring tool via dbt elementary.
- Implemneted the integration for extrnal data real time into Snowflake via snowpark.
- Improvbements into snowflake, automated db house keeping.
- Governance
Analytical Developer Data Engineer
Airbnb
Gurugram
05.2021 - 08.2022
- Automated the schema management for
data warehouse tables. Solution was GIT-managed, hence version controlled, and improved team efficiency by 19%, and opened ways for away team members to work on team's platform. - Remodeled existing data model into revamped model that was optimized for storage and compute. Achieved this using intermediate aggregated tables. This resulted in $2400/month savings in team's budget, improvement of 34%.
- Migrated legacy HQL-based codebase to optimized Scala-based pipelines. This helped with compute costs savings of $900/month.
- Implemented Data Quality Framework for anomaly detection and reporting. Proactively identified 233 issues in span of 7 months, improving reliability of datasets.
- Led design and implementation of in-house Push reporting framework for insights delivery through various channels.
Data Engineer
Amazon
Bengaluru
08.2019 - 04.2021
- Delivered data pipelines handling terabytes scale data, loaded daily into data warehouse.
- Migrated legacy data ingestion(SQL-based system) to Spark
based pipelines, on AWS EMR, using Scala for programming and Airflow for orchestration, resulting in availability of data in near real time. - Led implementation of Away team enablement, to simplify and automate general data warehousing requests for customers, leading to 27% improvement in team efficiency.
- Designed data model for transactional data for Amazon Music, overhauling existing data models, leading to effective storage and retrieval of data.
- Implemented automated data archival system, that offloaded data from
redshift to S3 for archival, saving $100k/yr in overall costs.
Senior Associate Technical Consultant
SAS R & D
Pune
01.2018 - 08.2019
- Led global deployment of SAS Anti-Money Laundering Solutions for 5 banking clients across different geographies.
- Designed and optimized ETL framework for efficient data warehousing and analytics in Oracle/MySQL, leveraging SAS Visual Analytics for predictive modelling.
- Established AML monitoring scenarios for financial institutions, enhancing transaction surveillance and compliance with integrated alert system for case management.
- Improved data pipeline efficiency using SAS Data Integration Studio and Oracle tuning, and optimizations lead to 44% reduction in processing times for data loads.
- Developed customized reports for AML-based analytics, which was appreciated by clientele, and was integrated into base product.
Project Engineer
Wipro Infotech
Mumbai
09.2015 - 01.2018
- Developed ETL pipeline to land banking data from disparate source systems into Central Data Warehouse(CDW).
- Integrated warehouse with third-party solution, SAS DDS, for Automated reporting for ADF reporting to RBI, and generating internal reporting on Credit Risk, Market Risk and asset-liabilities management system. Implemented SAS Anti-Money laundering solution for banking clients, for screening potentially illicit transactions and alerting investigators.
- Optimized whole data flow applications between Bank's Oracle Data Warehouse and SAS systems. Reduced processing time by 90% for overall data load.
- Mentoring new joiners and new team members on data
engineering practices in SQL and Oracle DB.
Education
Bachelor of Technology - Electronics And Communications Engineering
Jawaharlal Nehru Government Engineering College
Sundernagar, Mandi, Himachal Pradesh, India 08.2011 - 07.2015
Skills
Spark, Scala
Data Modelling, Data Warehousing
Cloud Services, GCP
Spark, Scala
Data Governance
Data Pipeline Design ETL/ELT
Batch/Real Time data processing
Timeline
Sr. Data Engineer
Apollo.io
08.2022 - Current
Analytical Developer Data Engineer
Airbnb
05.2021 - 08.2022
Data Engineer
Amazon
08.2019 - 04.2021
Senior Associate Technical Consultant
SAS R & D
01.2018 - 08.2019
Project Engineer
Wipro Infotech
09.2015 - 01.2018
Bachelor of Technology - Electronics And Communications Engineering
Jawaharlal Nehru Government Engineering College
08.2011 - 07.2015