Summary
Overview
Work History
Education
Skills
Certification
Accomplishments
Languages
Timeline
Generic

Rahul Kumar

Bengaluru

Summary

Senior Azure Data Engineer with extensive experience in architecting, developing, and optimizing end-to-end data solutions on Microsoft Azure. Expert in leveraging Azure Data Factory, Synapse Analytics, Databricks, and Azure SQL Database to build robust ETL pipelines, scalable data warehouses, and real-time data processing systems. Adept at ensuring data quality, security, and compliance across complex environments. Proven ability to collaborate cross-functionally to deliver insight-driven, high-performance data platforms that support strategic business goals and innovation.

Overview

7
7
years of professional experience
1
1
Certification

Work History

Engineer II

Lululemon
05.2023 - Current
  • Developed end-to-end, optimized, and metadata-driven pipelines in ADF to load and process large volumes of data from disparate sources into Snowflake.
  • Developed scripts to call APIs at scale using libraries like aiohttp and requests, including handling rate-limiting and parallel processing.
  • Designed and implemented distributed data processing notebooks using Azure Databricks, PySpark for real-time and batch data processing.
  • Leveraged Kafka to enable real-time data ingestion and processing, developed SQL scripts, procedures to ingest data in SCD type2, type1 pattern.
  • Processed and transformed data in multiple file formats such as Parquet, Delta Lake, and Iceberg to support diverse analytics use cases.
  • Hands-on experience in creating and maintaining CI/CD pipelines using tools like Git, and Azure DevOps, adhering to agile development methodologies.
  • Working with all the stakeholders the understand the requirement and implement it

Data Engineer

Tiger Analytics
08.2022 - 05.2023
  • Company Overview: Worked as a data engineer for a US based client.
  • Responsible for data ingestion from API to data lake.
  • Develop databricks notebook for transformation of data and scheduling the jobs in databricks and adf.
  • Worked closely with data analyst for gathering the requirements and implement it.
  • Manage the Devops release pipeline for deploying the code to higher environments.
  • Worked as a data engineer for a US based client.

Consultant II

Neudesic Technologies
06.2021 - 08.2022
  • Company Overview: Worked as a consultant for US based client.
  • Responsible for data ingestion (flat files) to Azure Sql DB.
  • Develop end to end pipelines in ADF for integration and movement of data.
  • Develop Azure function using python for cleansing of the files.
  • Writing stored procs for loading, transforming data.
  • Worked as a consultant for US based client.

System Engineer

Tata Consultancy Services Ltd
09.2018 - 06.2021
  • Company Overview: Worked as a consultant for US based client.
  • Responsible for processing and analyzing raw data(.xls format to csv,parquet,delta) using Pyspark, ADF, Blob, ADLS, Spark SQL, Hive, Databricks.
  • Developed end to end pipelines in ADF for integration and movement of data.
  • Developed optimized queries for transforming the data using pyspark, python, SQL.
  • Worked closely with the stakeholders to understand the requirement and implement it.
  • Worked as a consultant for US based client.

Education

B.Tech - Information Technology

Kalyani Government Engineering College
Kalyani, West Bengal
06.2018

Skills

  • Azure Databricks
  • Microsoft Azure
  • Python
  • SQL
  • Snowflake
  • Stakeholder Management
  • Power BI
  • PySpark
  • Apache Hive
  • Azure Functions
  • Data Ingestion
  • Big data analysis
  • Azure Data Factory
  • Data Lake
  • CI/CD
  • Data Warehousing
  • Prompt Engineering

Certification

  • Snowflake Snowpro Certification, 11/01/24, Present
  • DP-203: Microsoft Azure Data Engineering Associate, 01/01/22, 01/01/23
  • DP-900: Microsoft Azure Data Fundamentals, 01/01/22, 01/01/23
  • AZ-900: Microsoft Azure Fundamentals, 03/01/22, 03/01/22

Accomplishments

  • Reduced data processing time by 30% using optimized pipelines.
  • Increased data pipeline efficiency by 25% with Python scripts.
  • Improved data accuracy by 40% through robust ETL processes.
  • Streamlined API data integration, making 70K requests a day.

Languages

English
Hindi
Bengali

Timeline

Engineer II

Lululemon
05.2023 - Current

Data Engineer

Tiger Analytics
08.2022 - 05.2023

Consultant II

Neudesic Technologies
06.2021 - 08.2022

System Engineer

Tata Consultancy Services Ltd
09.2018 - 06.2021

B.Tech - Information Technology

Kalyani Government Engineering College

Similar Profiles

CREATE PROFILE
Rahul Kumar