Himanshu Gupta

• Working on a migration project from on-premise to Microsoft Azure Cloud
• Using Tech Stack PySpark, Databricks, Oracle, Delta Tables, Azure Data Factory, Datalake.
• Creating end to end data pipelines from Data ingestion to Datawarehouse load with updated data.
• Responsible for Spark Job performance optimizations to manage cost
• Implementation of CI/CD Pipelines using Github for Code and Pipeline Deployments.
• Involved in Production environment setup and code deployments.

• Developed a Datawarehouse using Big data tools and technologies on MS Azure cloud for leading Retail client.

• Used Azure Datafactory,Azure Databricks, Azure Datalake storage and Azure Synapse.

• Developed ETL transformations(Joins,Aggregations etc.) using spark for Data Processing.

• Integrated multiples sources like SQL,Oracle with Azure Datalake using Azure Data Factory

• Employed data cleansing methods, significantly Enhanced data quality.

• Spark Job Optimizations for running the jobs with maximum efficiency

• Performed Unit testing and Data validation

• Developed a Datawarehouse using Big data tools and technologies on MS Azure cloud for leading Retail client
• Developed ETL transformations(Joins,Aggregations etc.) using spark for Data Processing
• Integrated multiples sources like SQL,Oracle with Azure Datalake using Azure Data Factory
• Implementation of Spark Scala Code based on client requirements
• Implementation of Definitions and UDF's within the code
• Created Spark Streaming as well as Spark Batch processing jobs and ran on Azure DataBricks Platform

• Worked on migration project to transform banking data from on premise database to Google Big Query(GCP)
• Used Snowflake Data Model and performed transformations using Spark
• Developed Spark code on scala to create ETL transformations and Aggregations
• Implementation of the Business Logics in the Job.

• Decommissioning of Informatica to Hadoop Big data of from ERP System stored to Hadoop ecosystem
• Implementation of the Business Logic used on Informatica to spark jobs
• Processing and Cleansing of data on spark and then applying transformations in optimized way
• Validations of data from Informatica with hive.