Vikas Gullapelli

Azure Data Engineer

JP MORGAN Services Pvt Ltd

11.2020 - Current

Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources like Azure SQL, Blob storage
Successfully reduced data processing time by 50% through performance tuning and optimization techniques
Designed and developed and deployed multiple end-to-end data pipelines for ingesting high volume, high veracity data to the Azure cloud by leveraging Python, Azure Databricks and ADF (Azure Data Factory)
Implemented Azure DevOps build and release pipelines to deploy Notebooks, Jobs, python files to Azure Databricks and to deploy Linked services, datasets, pipelines etc
To Azure Data Factory
Created ADF pipelines using activities – Copy, Lookup, For Each, Data bricks (notebook), Get Metadata, Execute Pipeline, Stored Procedure, If condition, Filter, Until etc
Good working experience in query optimization & performance tuning of stored procedures & spark
Worked on unity catalog in Azure Databricks
Monitored the scheduled Azure Data Factory pipelines and configured the alerts to get notifications of failure pipelines through Azure Logic App
Proficient in implementing and managing ETL processes using Azure services including Azure Storage, Databricks, PySpark
Ensured data security and compliance by implementing encryption, access control, and key management solutions using Azure Key Vault, adhering to regulatory requirements and industry best practices
Developed ADF pipelines and triggers to load the data from various source systems into target tables by executing Databricks notebooks and used ADLS Gen 2 for data storage and retrieval
Writing the stored procedures, functions, triggers and creating indexes and Having Good programming knowledge on T-SQL Statements
Collaborated extensively with internal teams, including data engineering and product teams, to lead solution design and proof-of-concept discussions
Developed ADF pipelines and python scripts for carrying out data quality checks and configured email notifications using logic apps to notify users once tables were refreshed successfully
Worked on Notebook in Azure Databricks with Job Cluster scheduled Batch Jobs
Monitoring ADF Pipelines for Daily/weekly/monthly loads and identifying the failed pipelines and do the analysis for the failures
Writing the stored procedures, functions, triggers and creating indexes and Having Good programming knowledge on T-SQL Statements
Collaborated extensively with internal teams, including data engineering and product teams, to lead solution design and proof-of-concept discussions
Strong experience in creating complex pipeline jobs, scheduling triggers, Mapping data flows using Azure Data Factory
Worked on Notebook in Azure Databricks with Job Cluster scheduled Batch Jobs
Monitoring ADF Pipelines for Daily/weekly/monthly loads and identifying the failed pipelines and do the analysis for the failures