RAHUL KUMAR
Data Engineer
Bengaluru
Summary
Experienced Azure Data Engineer with expertise in designing, implementing, and optimizing data pipelines on Microsoft Azure. Proficient in utilizing key tools including Azure Data Factory, Synapse Analytics, Databricks, and SQL Database. Skilled in ETL, data warehousing, and real-time data processing. Proven collaborator, adept at working closely with teams to deliver scalable, secure, and compliant data solutions that drive valuable business insights and foster innovation.
Overview
7
7
years of professional experience
4
4
years of post-secondary education
4
4
Certifications
3
3
Languages
Work History
Engineer II
Lululemon
05.2023 - Current
- Developed end-to-end, optimized, and metadata-driven pipelines in ADF to load and process large volumes of data from disparate sources into Snowflake.
- Developed scripts to call APIs at scale using libraries like aiohttp and requests, including handling rate-limiting and parallel processing.
- Designed and implemented distributed data processing notebooks using Azure Databricks, PySpark for real-time and batch data processing.
- Leveraged Kafka to enable real-time data ingestion and processing, developed SQL scripts, procedures to ingest data in SCD type2, type1 pattern.
- Processed and transformed data in multiple file formats such as Parquet, Delta Lake, and Iceberg to support diverse analytics use cases.
- Hands-on experience in creating and maintaining CI/CD pipelines using tools like Git, and Azure DevOps, adhering to agile development methodologies.
- Working with all the stakeholders the understand the requirement and implement it
Data Engineer
Tiger Analytics
08.2022 - 05.2023
- Worked as a data engineer for a US based client, responsible for data ingestion from API to data lake
- Develop databricks notebook for transformation of data and scheduling the jobs in databricks and adf
- Worked closely with data analyst for gathering the requirements and implement it
- Manage the Devops release pipeline for deploying the code to higher environments
Consultant II
Neudesic Technologies
06.2021 - 08.2022
- Worked as a consultant for US based client, responsible for data ingestion (flat files) to Azure Sql DB
- Develop end to end pipelines in ADF for integration and movement of data
- Develop Azure function using python for cleansing of the files
- Writing stored procs for loading, transforming data
System Engineer
Tata Consultancy Services Ltd
09.2018 - 06.2021
- Worked as a consultant for US based client, responsible for processing and analyzing raw data(.xls format to csv,parquet,delta) using Pyspark, ADF, Blob, ADLS, Spark SQL, Hive, Databricks
- Developed end to end pipelines in ADF for integration and movement of data
- Developed optimized queries for transforming the data using pyspark, python, SQL
- Worked closely with the stakeholders to understand the requirement and implement it
Education
B.Tech - Information Technology
Kalyani Government Engineering College
Kalyani, West Bengal, India 05.2014 - 06.2018
Skills
- Databricks
- Microsoft Azure
- Python (Programming Language)
- SQL
- Snowflake
- Stakeholder Management
- Power BI
- PySpark
- Apache Hive,
- Data Ingestion
- Big data analysis
- Azure Data Factory
- Data Lake
- CI/CD
- Data Warehousing
Certification
Snowflake Snowpro Certification, 11/01/23, Present
Accomplishments
- Reduced data processing time by 30% using optimized pipelines
- Increased data pipeline efficiency by 25% with Python scripts.
- Improved data accuracy by 40% through robust ETL processes.
- Streamlined API data integration, making 70K requests a day
Timeline
Engineer II
Lululemon
05.2023 - Current
Data Engineer
Tiger Analytics
08.2022 - 05.2023
Consultant II
Neudesic Technologies
06.2021 - 08.2022
System Engineer
Tata Consultancy Services Ltd
09.2018 - 06.2021
B.Tech - Information Technology
Kalyani Government Engineering College
05.2014 - 06.2018
Similar Profiles
Owen PadilloOwen Padillo
Warehouse Associate at LululemonWarehouse Associate at Lululemon
Matthew WoodardMatthew Woodard
Assistant Manager of Operations at LululemonAssistant Manager of Operations at Lululemon
Andrew MontoyaAndrew Montoya
Warehouse Worker at LululemonWarehouse Worker at Lululemon