Kishor Kumar Saini

Project: (US Health Insurance Client)

1. Consumption View Audit & Alert Framework (Azure Data Factory):

• In this project, we are checking the ingestion of all Hive tables and applying the necessary transformations using business logic.
• The transformed data and audit details were stored in Azure Synapse Analytics.
• Sent alerts for successful pipeline runs and failures.

2. Migration from ADF to Databricks (Azure Databricks & PySpark):

• Worked on migrating Azure Data Factory dataflows using Databricks PySpark.
• Created DataFrames using PySpark and applied various transformations based on business requirements.
• Developed a generic codebase using parameters and reusable PySpark code.
• Developed parameterized workflows on Databricks.

3. Spark Streaming & Event Hub (Azure Databricks & PySpark):

• Received real-time patient information messages from Event Hub.
• Developed framework to load data from Bronze layer to Silver layer and enable user-specified transformations.
• Gold Framework Development: Design and develop the Gold Framework to load data from the Silver layer to the Gold layer.
  Create insightful metrics from the data.
• Sent insightful metrics to Cosmos DB API for mobile app usage.

Project: Integrated Business Planning (US Health Insurance Client)

• Responsible for developing the Framework, which loads the data from several sources up to the Silver Layer, while allowing users to apply transformations as desired.
• Designed and developed the Gold Framework to load the data from silver to gold layer, while also generating outbound files for Data Science Teams' consumption.
• Load data objects from external sources to the Gold Layer using the Framework while working with DG, DA, and DM Team.

Project: Capture and Refine Tool

• I was responsible for creating operators for the ETL tool. Operators included Validation, Transformation, filter, and Ingestion.
• Development of a migration script to migrate metadata from the source environment to the test and production environment.
• Technologies used: Azure, Python, DBT.