ABHISHEK VAISH

A Data Engineer with 2+ years of experience in the field and a passion for all things in data. One of my biggest achievements so far is to complete a successful migration, improving the data processing and data quality. Having strong communication and technical background. Capable of mastering new technologies.

Spotify End to End Data Pipeline

• Use Spotify Developer Account to pull data from API using AWS Glue.
• Converting the API results into JSON files and uploading them to AWS S3.
• Connect S3 with Azure Databricks using IAM.
• Use KMS to store keys and Databricks Secrets to pull secret keys on Databricks.
• Use Medallion Architecture to transform the data.
• Store the data into DataLake as parquet format after performing all the Data Transformation.

IPL Dataset - Data Pipeline and Data Analysis

• Use IPL Dataset from Kaggle which is initially present in CSV format.
• Upload files to AWS S3 bucket and connect it with Snowflake Stage.
• Use External Tables to load the files into Raw Tables.
• Perform Data Transformation to add more information to Data.
• Create Views for the complex calculation which will be used in reporting.
• Design reports using DAX query.
• Create Dashboard and provide filters to visualize data using Power BI.

• Astronomer Certification for Apache Airflow Fundamentals
• DBT Fundamentals at DBT Labs
• Badge 1: Data Warehousing Workshop at Snowflake University