Venkat sai kumar K
Data Engineer
Hyderabad
Summary
Adept Azure Data Engineer with a proven track record at Mindtree, UST Global, enhancing data processing and analysis through innovative Azure Data Lake and Azure Data Bricks solutions. Demonstrated expertise in automating and optimizing data workflows, achieving a 30% reduction in execution time. Collaborates effectively across teams, showcasing strong problem-solving skills and a commitment to data integrity.
Overview
5
5
years of professional experience
Work History
Azure Data Engineer
LTI Mindtree
07.2022 - Current
- Enhanced data quality by performing thorough cleaning, validation, and transformation tasks.
- Designing the workflow for sending the mail by using the logic apps in Azure.
- Automating on SQL DW Start and Stop using automation account and runbook
- Collaborated on ETL (Extract, Transform, Load) tasks, maintaining data integrity and verifying pipeline stability.
- Collecting the structured and semi-structured data from the different sources and after collecting the raw data,running the ETL pipelines to cleanse the data and convert it as per the business need.
- Responsible for data ingestion, data cleansing, data standardization and data transformation.
- Implementing various spark transformation and actions using Databricks on raw data as per the business requirement.
- Involved in extracting data from traditional RDBMS to ADLS using Azure data factory.
- Involved in the development of a well optimized ADF pipeline which has reduced the execution time by 30%
Data Engineer
UST Global
10.2019 - 07.2022
- Collaborated with cross-functional teams for seamless integration of data sources into the company''s data ecosystem.
- Designed and implemented a data lake solution on Azure using Azure Data Lake Storage and Azure Data bricks to enable real-time data processing and analysis.
- Developed data ingestion pipelines and data transformation workflows using Python.
- Responsible for creating and maintaining data pipelines in Azure Data factory, Pipeline components such as Integration Runtime, Linked Services, Datasets and Activities.
- Created nested pipelines with various triggering mechanisms. Transforming data with Azure Data Flows.
- Developed Data bricks notebooks and Job Compute.
Optimized data pipelines for performance and scalability. - Implemented data quality checks and data validation routines to ensure accuracy and consistency of data
Azure Cloud Engineer
UST Global
10.2019 - 07.2022
- Worked on PowerShell on Azure services by creating scripts (availability sets & Disks, VM Images, resizing Virtual machines, blobs and file shares.
- Automated using powershell scripts VM stoppage, multiple Virtual machine creations
- Working experience with Git and GitHub repositories
- implemented Azure Active directory RBAC rules.
- Worked on Azure Virtual Network multi-tier design with Subnets and Network Security Groups using powershell and Terraform
- Having good Knowledge in Azure monitor, Azure activity logs, Azure backup report and application insights
Education
Post Graduate Diploma in Business Management -
04.2001 -
B.Tech - Electronics And Communications Engineering
04.2001 -
Skills
Azure Data Factory
Azure Data Bricks
Azure Data Lake
Azure Devops
Azure keyvault
Spark SQL
PySpark
Powershell
Azure Synapse Analytics
Dataware House
Hive
HDFS
Sqoop
Timeline
Azure Data Engineer - LTI Mindtree
07.2022 - Current
Data Engineer - UST Global
10.2019 - 07.2022
Azure Cloud Engineer - UST Global
10.2019 - 07.2022
University Of Hyderabad - Post Graduate Diploma in Business Management,
04.2001 -
JNTUA - B.Tech, Electronics And Communications Engineering
04.2001 -