Project Name : AB In Bev
Role : Data Engineer
Technologies : ADLS Gen2, ADB, ADF, Azure blob Storage, PySpark
Responsibilities
- Involved in collecting the requirements from the Client
- Analyze quantitative and qualitative data to uncover information to drive desired changes
- Created Pyspark Scripts for Data Transformation based on provided Rules
- Worked on ADB (Azure Data Bricks) for Data Validation and to perform Transformation on Data using Pyspark, SQL, Python Read Data from DBMS,
- Read multiple tables, dataframes for comparison using Pyspark in ADB
- Mapping the columns from Oracle DB to PostgreSQL as per Business requirement
- Performed ETL using Azure Data Factory, Scheduling of Pipelines, created Linked Services, datasets
Project Name : DHL E Commerce Solutions
Role : Data Engineer
Technologies : Azure data factory, Azure data bricks, Azure Sql Db, PySpark, Azure
Responsibilities :
- Developed Python Scripts for data validation and data Correctness and obfuscation code using PySpark
- Performed Transformation on datasets with Pyspark
- Sentiment Sore Analysis, Key Phrase Extraction using Azure Cognitive Services
- Implemented the Optimization Techniques and Best Practices in the Azure Data Factory
- Writing Stored Procedures and functions to extract data from staging database
- Developed a Chat Bot using Microsoft BOT Framework integrated QnA Maker Service and Luis Service.