Mandar Deokar

Project Name: Conservice.

Domain : US Utility

Role : Azure Date Engineer

• Migrating the entire project to Microsoft Fabric, including migrating ADF pipelines to Fabric.
• Migrating the data warehousing system to Microsoft Fabric using lakehouses and medallion architecture for optimized data storage and processing.
• Setting up the Breeze Framework for systematic processing of data and a scalable approach.
• Converting complex stored procedures into PySpark notebooks for data transformation.
• Working with schema-based lakehouse to handle security in data lakehouse.
• Optimizing PySpark code to reduce processing times for notebooks.
• Developing a deployment pipeline on Microsoft Fabric to deploy the project to various workspaces. Using Variable Library to deploy items and reduce manual intervention.
• Analyzing requirements and developing transformation stored procedures to address specific business needs.
• Developing and managing Azure Data Factory (ADF) pipelines, a framework for data ingestion from multiple sources, including on-premises systems, NetSuite, Salesforce, and ADP, into Azure Data Lake Storage.
• Implementing efficient data ingestion processes for both full and incremental data loads.
• Using Direct Query and Import Mode in Power BI for Report Processing. Making changes to use Direct Lake Mode for faster Power BI refresh.
• Designing and developing dimensional and fact tables to support a scalable data warehouse architecture.
• Deploying the SQL database objects across UAT and Production environments.

Project Name : Colonial Oil

Domain : US Utility

Role: Azure Data Engineer.

• Setting up the infrastructure and Breeze Framework for the migration of the legacy database to Microsoft Fabric.
• Creating a notebook for data processing from an existing stored procedure and clients' input.
• Scheduling fabric capacity to improve cost efficiency. Vacuuming Delta Table and Lakehouse for cleanup of data, and implementing Delta Table time travel to handle failures in the pipeline and maintain accuracy of data.
• Working with the client for Salesforce and manual file processing, and setting up the system process for the same.

Project Name: JW Pepper

Domain: U.S. Retail

Role: Associate Data Developer.

• Analyzing requirements and developing transformation stored procedures to address specific business needs.
• Used Talend to understand the project requirements, and created procedures based on the same.
• Developing and managing Azure Data Factory (ADF) pipelines, a framework for data ingestion from multiple sources, including on-premises systems, Oracle, DB2, and IBM, into Azure Data Lake Storage.
• Built and maintained ADF Dataflows to give REST API calls, getting payloads, and then transforming complex, nested JSON data into a structured format compatible with the data warehouse (DWH) model.
• Implementing efficient data ingestion processes for both full and incremental data loads.
• Designing and developing dimensional and fact tables to support a scalable data warehouse architecture, and integrating multiple systems into one.
• Deploying the SQL database objects, as well as ADF pipelines, across UAT and production environments.
• Implemented data quality checks to ensure accuracy and consistency.
• Documented technical specifications and user guides for stakeholders.
• Developed scripts to automate repetitive tasks such as backups or populating test databases with sample data.