Big data engineer with 2 Years of experience in data engineering & big data domain and Azure service like ADF, Databricks, Storage Account and Event-Hub.
Databricks Professional and Azure certified Data Engineer
Experience in creating and scheduling ETL jobs with Azure Data Factory.
Expertise in Transforming and Migrating - Business Logics and KPIs using Databricks techs like Workflows, PySpark , Spark-SQL and DLT.
Sound knowledge of Python Framework Django.
Overview
2
2
years of professional experience
4
4
years of post-secondary education
5
5
Certifications
2
2
Languages
Work History
Data Engineer
Pentaho Migration Project ( Celebal Technologies)
Jaipur
07.2022 - Current
Gathered Raw Data from Hadoop , DB2 and Oracle using Azure Data Factory pipeline and stored in Raw-Zone container
Created Medallion Architecture (Raw, Intermediate and Global) to stored respective external table.
Migrated SQL and JavaScript code into Spark-SQL and PySpark code.
Created validation libraries to validation transformed data by Databricks notebook.
Created multiple jobs pipelines through ADF and integrated Databricks notebook to respective pipeline.
Global and Intermediate layer data sent to the NAS layer and PowerBI team.
Data Engineer
Finance Digi-Hub Project ( Celebal Technologies)
Jaipur
09.2021 - 06.2022
Gathered Raw Data from Fullstory tool using API Integration and Email Automation (IMAP) from Adobe experience tool
Performed Data Transformations such as filtering , pivoting, sorting, optimization and aggregation using Databricks.
Build and Implemented custom python function or classes for data validation.
Created Medallion Architecture and Layers (Bronze, Sliver & Gold) for stored external table into respective layers.
Performed Aggregation operation on Sliver Layer, Stored data into Gold Layer and send to the PowerBI team.
Education
Bachelor of Technology - Computer Science
Poornima University
Jaipur
08.2017 - 07.2021
Certification
Databricks Certified Data Engineer Professional
Timeline
Databricks Certified Data Engineer Professional
04-2023
Databricks Certified Associate Developer for Apache Spark 3.0