1 year of experience in Big data engineering, worked on data-related projects on Databricks, AWS, Azure, and Power BI etc.
Overview
1
1
year of professional experience
1
1
Certification
Work History
Data Engineer
Reflections Info Systems
Trivandrum
12.2021 - Current
Data Ingestion Pipelines:WebScrapping/DBFS->ETL(Databricks)->Azure DataLake.Get the data from Various Website or CSV/JSON files in dbfs and performed Quality Checks and Transformation Using Pyspark then load this data into Azure DataLake
Unified DataWarehouse:Azure DataLake->ETL(Databricks)->DeltaLake.Read CSV data from DataLake then performed SCD and Transformation on it finally load into delta lake
DataWarehouse(AWS):S3->ETL(Glue)->Redshift.Read Data from S3 using Crawler in GLUE then performed SCD and Transformation Using Pyspark and SparkSQL then stored data into REDSHIFT
CDC Real-time Streaming Pipeline:PostgreSQL->Azure Event Hub->Databricks(ETL)->Delta Lake.Captured data from database using Streamsets into EVENT hub then read this data into databricks notebook performed Extraction from JSON format and Transformation using Pyspark load into delta lake with soft delete.
Power BI Reports:Data Read->ETL(Query editor)->Data modelling and DAX->Visualization
DataMarts:Created DataMarts from Warehouse according to business requirements
Databricks Notebook Orchestration using Apache AIRFLOW
Machine Learning Intern
Grroom
Mumbai
09.2021 - 12.2021
Education
Bachelor of Computer Science And Engineering - Computer Science
Jiwaji University, Gwalior, Madhya Pradesh ,India
09.2021
77%
12th -
Govt. Mixed Hr, Sec. Jammu, India
05.2017
77%
10th -
Govt. Mixed Hr. Sec. Jammu
05.2015
86.20%
Skills
SQL
Spark
Python
Data Warehousing & Marts
Change data capture
Kafka,Event Hub
Airflow,Docker,Linux
Streaming
WebScrapping
AWS(EMR,Glue,S3,RedShift)
Power BI(Data Modelling,DAX,Reports)
Databricks
Azure(DataLake,ADF)
Certification
Azure Data Fundamental DP-900
Timeline
Data Engineer - Reflections Info Systems
12.2021 - Current
Machine Learning Intern - Grroom
09.2021 - 12.2021
Jiwaji University - Bachelor of Computer Science And Engineering, Computer Science