Sujeet Singh Juneja
Azure Data Engineer
Pune
Summary
Certified Azure Data Engineer with 4 years of experience in the different Azure services, especially ADF (Azure Data Factory) and Azure Databricks. Responsible for implementing and designing architecture in the cloud to bring in digital transformation.
Overview
5
5
years of professional experience
2
2
Certification
Skills
- Database: SQL server
- Cloud Platform: Azure Databricks, Azure Data Factory, Azure DevOps
- PySpark
- Python
- CI/CD
- Data Warehouse Concepts
- Azure Delta Lake Concepts
- Other tools: QlikView (Reporting tool), SSIS
Work History
Azure Data Engineer
Exusia India Pvt Ltd
01.2023 - Current
- Built Azure Data Factory (ADF) pipeline to execute dynamic column mapping using a user-configurable lookup file
- The procedure was made more effective and efficient, and the work required to execute mapping for various data sources was reduced by this pipeline
- To transfer data from Netezza and DB2 to ADLSGen2 in parquet format, a configuration-driven framework was designed and implemented using a variety of Azure Data Factory (ADF) operations
- This required less manual effort
- Delivered data after performing certain transformations based on the business requirement using data flow activity in ADF
- Developed pipeline that can be quickly integrated with other pipelines to obtain a list of unprocessed files
- Only the new files from the source can be processed using this pipeline
- The processing costs are decreased with the aid of this pipeline
- Implemented a pipeline that was designed to process only the files that had been changed or added since the last pipeline runs
- Designed and schedule Databricks notebook to automate the external table creation for the new onboarded data in
- ADLS or refresh the existing tables
- This utility is based on the configuration table and also maintains the audit of the tables created or refreshed
- Implemented change data capture (CDC) using the data flow component in Azure Data Factory (ADF)
- Designed the data extraction utility in ADF to push the processed data business storage account based on the load behavior.
Azure Data Engineer
Nagarro Software Pvt Ltd
09.2021 - 12.2022
- Design and development of a common component framework architecture on Azure Data Factory
- Created complete end-to-end orchestration ADF pipelines for data ingestion, data warehouse, and snowflake data load
- Making performance improvements to the current common component framework in an effort to speed up processing
- Have completed environment setup for data processing by building data factory pipelines, databases, and clusters in Databricks with the appropriate metastore, among other things
- Adding new features to the current framework in case the data management team has new requirements
- Created data factory pipelines for data ingress and egress that support external locations like ADLSGen2, S3, and
- SFTP
- Worked with the data management team to put up data quality checks to deal with the incoming faulty records
- Migration of ADF Pipelines and Databricks notebook using CI/CD pipelines to a higher environment
- Also used CI/CD pipelines to update the secret in the KeyVault
- Knowledge of implementing Spark concepts using SparkSQL or PySpark
- Responsible for providing the client, onshore colleagues, and senior leaderships with a technical overview of the design and architecture.
Data Warehouse Specialist
Milliman India Pvt Ltd
06.2020 - 09.2021
- Designed ADF pipeline to extract parquet file of on-premises SQL Server table’s data into Azure Cloud
- Performed operations like Data reconciliation, validation, and error handling after extracting data into SQL Server
- Created a linked service in ADF to execute the notebooks present in Azure Databricks
- Worked on writing ETL using Spark SQL in Databricks
- Worked on creating tests for data validation using deequ in Databricks.
MedInsight Intern
Milliman India Pvt Ltd
06.2019 - 05.2020
- Designed SSIS packages to transfer data from flat files to SQL server
- SSIS transformations such as data conversion, lookup, derived column, conditional split etc
- Were used
- Wrote checks to verify the integrity of file data
- Worked with large volumes of files and building automation around it to improve file processing efficiency
- Data analysis/finding error when the process of data gets failed and providing proper documentation around the findings.
Certification
Microsoft Certified: Azure Data Engineer Associate (DP-203)
Microsoft Certified: Azure Fundamentals (AZ-900)
Similar Profiles
Harpreet MehtaHarpreet Mehta
Partner - Head of Global Operations at Exusia India Pvt Ltd.Partner - Head of Global Operations at Exusia India Pvt Ltd.
SOWMYA ARAVINDSOWMYA ARAVIND
Vice President at WELLS FARGO INDIAVice President at WELLS FARGO INDIA
Himanshu JainHimanshu Jain
Associate Consultant at ExusiaAssociate Consultant at Exusia
Nikit KhopadeNikit Khopade
Key Account & Marketing Manager at Kuvos Tech (A DYNAMIK LABS GROUP OF COMPANY)Key Account & Marketing Manager at Kuvos Tech (A DYNAMIK LABS GROUP OF COMPANY)
Sudarshan MhalasSudarshan Mhalas
Lead Engineer at Ozone Financial Technology India Pvt Ltd (Ozone API)Lead Engineer at Ozone Financial Technology India Pvt Ltd (Ozone API)