Summary
1 year of experience in Big data engineering, worked on data-related projects on Databricks, AWS, Azure, and Power BI etc.
Overview
1
1
year of professional experience
1
1
Certification
Work History
Data Engineer
Reflections Info Systems
Trivandrum
12.2021 - Current
- Data Ingestion Pipelines:WebScrapping/DBFS->ETL(Databricks)->Azure DataLake.Get the data from Various Website or CSV/JSON files in dbfs and performed Quality Checks and Transformation Using Pyspark then load this data into Azure DataLake
- Unified DataWarehouse:Azure DataLake->ETL(Databricks)->DeltaLake.Read CSV data from DataLake then performed SCD and Transformation on it finally load into delta lake
- DataWarehouse(AWS):S3->ETL(Glue)->Redshift.Read Data from S3 using Crawler in GLUE then performed SCD and Transformation Using Pyspark and SparkSQL then stored data into REDSHIFT
- CDC Real-time Streaming Pipeline:PostgreSQL->Azure Event Hub->Databricks(ETL)->Delta Lake.Captured data from database using Streamsets into EVENT hub then read this data into databricks notebook performed Extraction from JSON format and Transformation using Pyspark load into delta lake with soft delete.
- Power BI Reports:Data Read->ETL(Query editor)->Data modelling and DAX->Visualization
- DataMarts:Created DataMarts from Warehouse according to business requirements
- Databricks Notebook Orchestration using Apache AIRFLOW
Machine Learning Intern
Grroom
Mumbai
09.2021 - 12.2021
Education
Bachelor of Computer Science And Engineering - Computer Science
09.2021
77%
12th -
05.2017
77%
10th -
05.2015
86.20%
Skills
- SQL
- Spark
- Python
- Data Warehousing & Marts
- Change data capture
- Kafka,Event Hub
- Airflow,Docker,Linux
- Streaming
- WebScrapping
- AWS(EMR,Glue,S3,RedShift)
- Power BI(Data Modelling,DAX,Reports)
- Databricks
- Azure(DataLake,ADF)
Certification
Azure Data Fundamental DP-900
Timeline
Data Engineer - Reflections Info Systems
12.2021 - Current
Machine Learning Intern - Grroom
09.2021 - 12.2021
Jiwaji University - Bachelor of Computer Science And Engineering, Computer Science
Govt. Mixed Hr, Sec. Jammu - 12th,
Govt. Mixed Hr. Sec. Jammu - 10th,
Similar Profiles
ANJU PRASADANJU PRASAD
Quality Assurance Engineer at Reflections Info systems Pvt LtdQuality Assurance Engineer at Reflections Info systems Pvt Ltd
VIGNESH DURAIVELUVIGNESH DURAIVELU
Associate Manager - IT Delivery at Reflections Info Systems PVT LTDAssociate Manager - IT Delivery at Reflections Info Systems PVT LTD
ABHIMANYU PSABHIMANYU PS
Senior System Administrator at Reflections Info Systems PVT LTDSenior System Administrator at Reflections Info Systems PVT LTD
Anand SankarAnand Sankar
Software Engineer at Reflections info systemsSoftware Engineer at Reflections info systems
ASHWITA GAJAREASHWITA GAJARE
Data Engineer at KFin Technologies Pvt. Ltd.Data Engineer at KFin Technologies Pvt. Ltd.