Rachit Kapoor
Chandigarh
Summary
Motivated and experienced Data Engineer with over 3 years of experience developing and managing datalake, building ETL pipelines around it. Proven track record of helping companies to improve sustainability, reduce overall costs, and configure the best approaches for the most seamless and effective production.
Overview
3
3
years of professional experience
Work History
Senior Product Engineer
Loyalty Juggernaut Inc
Hyderabad
08.2021 - Current
- Led enhancement and improvement efforts for ETL pipelines within a Redshift-based datalake, utilizing Airflow, Spark, Python, and DMS. The enhanced ETL pipeline can now handle data up to 25 TB in one run without manual intervention.
- Designed and automated the end-to-end ETL flow, enabling seamless handling of migrations with just a click of a button. ETL configuration issues decreased by 70%
- Designed and implemented a pipeline to manage the creation of a Delta Lake, following the medallion architecture, which includes gold, bronze, and silver layers. AI query performance increased manyfold
- Actively developed and implemented pipelines around datalake, including Data Export functionality, to seamlessly transfer data from the datalake to external systems, leveraging technologies such as Airflow, Flask, AWS Batch, and Python.
- Developed an AI pipeline aimed at predicting customer churn, leveraging AWS Batch, Airflow, Amazon SageMaker, and AWS DynamoDB. Led to a 20% increase in customer retention
- Led and developed initiatives to efficiently manage partitions within PostgreSQL and seamlessly integrate them into our ETL pipeline, consequently enhancing query performance and responsiveness
- Led and developed a transformative change in ETL processes, transitioning DMS (Extract component of ETL) ownership to the Data Lake team through automated setup. Demonstrated hands-on leadership, ensuring comprehensive control over the ETL pipeline.
- Led and developed a DAG model to enhance capabilities of our query executor pipelines, allowing us to seamlessly create dependencies.
Education
Bachelor of Science - Computer Science Engineering
Chitkara University
Rajpura, Punjab07.2021
Skills
- Python
- Airflow
- Hadoop
- Apache Spark
- ETL
- AWS Batch
- AWS EMR
- AWS EC2
- Amazon Redshift
- AWS Lambda
- AWS S3
- AWS RDS
- Flask
- Docker
Accomplishments
- Qualified as an Infosys Certified Software Programmer
- Received the 'Juggernaut of the Month' award in commendation of my unwavering and exceptional contributions.
Extra Curriculars and Interests
- A guitar novice
- An avid hiker and walker.
- Occasional video game player.
- A basketball neophyte
Timeline
Senior Product Engineer
Loyalty Juggernaut Inc
08.2021 - Current
Bachelor of Science - Computer Science Engineering
Chitkara University
Similar Profiles
SADVITHA GONASADVITHA GONA
Product Specialist at Loyalty JuggernautProduct Specialist at Loyalty Juggernaut
Shubham PatilShubham Patil
Senior Production Engineer at Loyalty JuggernautSenior Production Engineer at Loyalty Juggernaut
Chris ZahariasChris Zaharias
Chief Revenue Officer at Loyalty JuggernautChief Revenue Officer at Loyalty Juggernaut