Kumar Snehil

• The client places the requirements from three different source systems on a daily basis. Spark consumes data from ADLS, Snowflake, and Web API, with these three sources of data ingestion occurring on a daily basis.
• The three Spark jobs created will consume the data from all three sources, perform multiple transformations in the Azure Databricks notebook, and write data into intermediate ADLS.
• After a final Spark job runs, which joins all three intermediate tables into a single master Azure Databricks notebook, present in the output table.
• For the orchestration tool, we used Azure Data Factory (ADF) to trigger the notebook at a suitable business hour, where the Azure data pipeline runs and provides the final output.
• Ad hoc analysis and debugging, job monitoring, and data quality checks, and the business queries the data from that table.
• Analytics are done on the final data by being consumed in the Tableau dashboard.
  

Project 1

• Developed data models that enhanced business decision-making processes.
• Implemented data governance protocols to ensure industry compliance.
• Conducted analyses of key issues, enabling effective root cause identification.
• Collaborated with cross-functional teams to gather requirements for data initiatives.
• Sourced raw data from third-party systems in ADLS for Foundation Data Product.
• Transformed and processed data in Azure Databricks for reporting purposes.
• Delivered final data to downstream business users for GI Daily, Weekly, and Commission Reports.

Project 2

• Worked with the Central Migration team for Azure Cloud migration from v1 to v2, which includes datasets, pipelines, Python codes and custom shell scripts migration from HDFS to ADLS services.
• The process involved three types of testing (local, unit, and rapid) testing with different volumes of data.
• The project involved stringent timelines, customer follow-ups, and achieved it within deadlines.

Updated ADLS data monthly, managing approximately 900 columns of data.

A dynamically created catalog is generated daily based on the available columns.

• Sorted DataFrame according to ADLS column schema specifications before writing to Hive tables.
• Ensured seamless integration of ADLS for efficient reporting.
• Executed daily processes to maintain accurate and timely data availability.
• Optimized workflows for dynamic table creation and data management.
• Engaged in continuous learning to stay updated with new technologies and programming techniques.