Venkatesh Punagani
Azure Data Engineer
Austin
Summary
With over 12 years of experience in Data Engineering and a specialization in Azure's Data ecosystem and Python development, I am enthusiastic about creating data-driven solutions that have notably enhanced business processes. Accustomed to working closely with system architects, software architects and design analysts to understand business or industry requirements to develop comprehensive data models. Proficient at developing database architectural strategies at the modeling, design and implementation stages.
Overview
12
12
years of professional experience
4
4
years of post-secondary education
Work History
Senior Data Engineer
Croyant Technologies Private Limited
8 2023 - 01.2024
- Facilitated project planning by effectively estimating technical tasks, resulting in superior sprint execution and 100% on-time delivery.
- Experience designing and implementing data pipelines using Azure Databricks for data cleaning, transformation, and loading into Azure Synapse Analytics
Knowledge of implementation and deployment of analytics solutions at client architecture. - Experience performing Data Wrangling and Exploratory Data Analysis on required data and metrics
- Established and enforced naming standards for cloud-based data warehousing solutions, ensuring consistency and clarity across projects.
- Responsible for development, support, maintenance, and implementation of a complex project module.
- Facilitation of end-to-end solution discussions.
- Creation of integration requirements in the form of features and user stories.
- Provide expertise in area and advanced knowledge of applications programming and ensure application design adheres to the overall architecture blueprint.
- Utilize advanced knowledge of system flow and develop standards for coding, testing, debugging, and implementation.
- Reengineered existing ETL workflows to improve performance by identifying bottlenecks and optimizing code accordingly.
Senior Data Engineer
Xebia IT Architects Pvt. Ltd
03.2022 - 08.2023
- Implement data integration and data transformation solutions using Azure services like Azure Data Factory, Azure Databricks, and Azure Synapse Analytics.
- Optimize and tune data solutions for performance, scalability, and cost-effectiveness.
- Ensure data quality, integrity, and security throughout data lifecycle.
- Monitor and troubleshoot data pipelines to identify and resolve issues.
- Collaborate with data scientists and analysts to provide them with necessary data and infrastructure for their work.
- Analyzed complex data and identified anomalies, trends, and risks to provide useful insights to improve internal controls.
IT Consultant
Ericsson India Pvt. Ltd
Bengaluru
04.2019 - 11.2021
- Design and implement end-to-end data solutions on Azure cloud platform.
- Collaborate with stakeholders to gather data requirements and understand business needs.
- Developed data pipelines and workflows to efficiently ingest, process, transform, and store data.
- Involved in Deploying data pipelines from lower environment to higher environment.
- Provided technical guidance and mentoring to application development teams throughout all phases of software development life cycle.
Data Engineer
IBM India Pvt. LTD
Bengaluru
05.2016 - 04.2019
- Identify and resolve issues concerning data management to improve data quality.
- Migrating data from API (Which we got data from ALBATROS Application) to Azure Data Lake.
- Maintaining data in Various Zones to perform transformation.
- Designed various ingestion and processing patterns based on use cases.
- Written Pyspark code to create dataframes and operate the source data.
- Involved in ETL Error Handling logic in Production Environment in azure.
- Converted On premises stored procedure into Pyspark dataframes.
Senior software engineer
Capgemini INDIA Private Limited
09.2013 - 05.2016
- Develop and implement pipelines that extract, transform, and load data into an information product that helps the organization reach its strategic goals.
- Focus on ingesting, storing, processing, and analyzing large datasets.
- Create scalable, high-performance web services for tracking data
Software Engineer
Aktrix Techonologies Private Limited
Bengaluru
07.2011 - 08.2013
- Design, develop, and maintain databases and database objects, such as tables, views, stored procedures, and triggers
- Develop and maintain database security, including user access and permissions
- Monitor database performance and troubleshoot issues
- Develop and maintain ETL processes to move data between databases and systems
- Develop and maintain data warehouse structures
- Develop and maintain data models and data dictionaries
- Develop and maintain data integration processes
- Develop and maintain reporting and analytics solutions
- Develop and maintain data quality and integrity processes
- Contributed to open-source projects, sharing knowledge with the broader community while gaining valuable insights from other experienced professionals.
Education
Bachelor of Technology - Information Technology
Annamacharya Institute of Technology & Sciences
India 08.2007 - 05.2011
Skills
Data Modelling
undefinedProfessional Description
10 Years experience in IT Industry with 5 years of experience in Azure Data Engineering. Well versed experience in creating pipelines in Azure Cloud Data Factory V2 using different activities like Move and Transform, Copy, Filter, ForEach etc., Scheduling pipelines and monitoring the pipelines. Good Experience in Azure Data Lake, Azure Data Factory, Azure SQL and Azure Blob, Azure Synapse Analytics, Azure Data Bricks. Creating pipeline and defining end to end data driven workflows using Azure factory. Create, edit data factories and child resources including Datasets, Linked services, Pipelines, Triggers and Integration runtimes. Good experience in ETL transformations using Data flows. Configuring email alerts using Logic Apps and web activity in ADF. Familiar with Data Warehousing using Data Extraction, Data Transformation, Data Loading (ETL) Hands on Experience in Agile/SCRUM Methodology. Having experience in Airlines, Health care/medical and Insurance domains. Designing, building, and deploying, publishing data analysis reports/dashboards for large volumes of data. Hands-on experience in Python programming and Spark components like Spark-core and Spark-SQL. Having experience to work and involved in few POC implementations. Experience in deploying the pipelines from Dev environment to Production Environment using Git Repos and Azure Devops.
Projects
Diabetes UK, Datahub Developer, Azure Synapse Analytics, Azure Data Lake Storage Gen2, Pyspark, Azure Devops, Git, Developed pipeline for common ingestion framework in Azure Synapse. Extract data from different sources like SFTP, NFS, API and store them in ADLS Gen2. Involved in SQL performance optimization techniques. Developed Pyspark notebook based on business rules in Synapse. Written Parametrized UDF’s for data Validation in Synapse. Involved in migration of data from one layer to another like Silver, Bronze, Gold Etc... Experienced in deploying the pipelines from lower environment to higher environment. Created Databases, Tables, Views based on requirement in Azure Synapse. Co-ordinated with business stakeholders on the requirements of Project. Etihad Airways, Senior Consultant, Azure Data Factory, Azure Data Bricks, Azure Synapse, Spark, Pyspark, Migration of data from Cloudera to Azure Data Lake Storage Gen2. Extract data from Web API to ADLS Gen2 using Azure Data Factory. Developing Azure Data Bricks notebook based on business logic. Using Logic apps and web activity for email alerts for developed pipelines in ADF. Involved in Production deployment activities. Migrated Complex transformations from Cloudera to Azure Environment. Involved and completed all POC’s within the timelines. Created external tables in Azure Synapse. Experience in using version control systems. Configuring Azure Key Vault to store secrets and using them in Datafactory and Data Bricks. Scheneider Electric, IT Consultant, Azure, Hadoop, HDFS, Hive, Spark, python, Databricks, Data Factory, Migrating data from the API (Which we got the data from ALBATROS Application) to Azure Data Lake. Maintaining the data in Various Zones to perform the transformations. Designed various ingestion and processing patterns based on use cases. Written the Pyspark code to create dataframes and operate the source data. Involved in ETL Error Handling logic in Production Environment in azure. Converted On premises stored procedure into Pyspark dataframes. Built complex data ingestion/processing frameworks using Azure Databricks/Python/Pyspark. Documenting the flow of process/data from different applications Involve in project related and architectural calls. Currently working on a POCs in Project Implementations. Advanced Analytics Euro Clear, Big data Developer, Azure, Hadoop, HDFS, Hive, Spark, python, Databricks, Data Factory, Worked closely with the business analysts to convert the Business Requirements into Technical Requirements and prepared low and high level documentation. Imported required tables from RDBMS to HDFS using Sqoop. Developed data pipeline using Flume, Sqoop and map reduce and Spark to ingest customer behavioral data and purchase histories into HDFS for analysis. Develop and run Map-Reduce jobs on a multi Peta byte YARN and Hadoop clusters which processes billions of events every day, to generate daily and monthly reports as per user's need. Developed Apache Spark Applications by using Scala, python and Implemented Apache Spark data processing project to handle data from various RDBMS and Streaming sources.
Timeline
Senior Data Engineer
Xebia IT Architects Pvt. Ltd
03.2022 - 08.2023
IT Consultant
Ericsson India Pvt. Ltd
04.2019 - 11.2021
Data Engineer
IBM India Pvt. LTD
05.2016 - 04.2019
Senior software engineer
Capgemini INDIA Private Limited
09.2013 - 05.2016
Software Engineer
Aktrix Techonologies Private Limited
07.2011 - 08.2013
Bachelor of Technology - Information Technology
Annamacharya Institute of Technology & Sciences
08.2007 - 05.2011
Senior Data Engineer
Croyant Technologies Private Limited
8 2023 - 01.2024
Similar Profiles
Vasuda KrishnamurthyVasuda Krishnamurthy
Financial Analyst at EMC SOFTWARE AND SERVICES INDIA PRIVATE LIMITED/Dell, Technologies India Private LimitedFinancial Analyst at EMC SOFTWARE AND SERVICES INDIA PRIVATE LIMITED/Dell, Technologies India Private Limited
RAKESH JRAKESH J
Network Engineer at EQL FINANCIAL TECHNOLOGIES PRIVATE LIMITED (Formerly name of INDIAFILINGS PRIVATE LIMITED)Network Engineer at EQL FINANCIAL TECHNOLOGIES PRIVATE LIMITED (Formerly name of INDIAFILINGS PRIVATE LIMITED)
NARENDRA JILKARNARENDRA JILKAR
Assistant Manager at Scootsy Logistics Private Limited (Swiggy Instamart - Bundl Technologies Private Limited)Assistant Manager at Scootsy Logistics Private Limited (Swiggy Instamart - Bundl Technologies Private Limited)
Norman ScottNorman Scott
Influencer Marketing Manager at Uplift DeskInfluencer Marketing Manager at Uplift Desk
Chaz BregmanChaz Bregman
Lab Mentor at Freshmen Research InitiativeLab Mentor at Freshmen Research Initiative