Sudheer Reddy Basiredddygari
Senior Software Engineer
Bangalore
Summary
Seasoned Azure Data Engineer | Big Data Analytics & ETL Development Expert
A seasoned Azure Data Engineer with 7 years of experience in Big Data Analytics, ETL development, and data integration. Highly skilled in architecting and optimizing end-to-end data solutions using Azure Data Factory, Azure Databricks, Azure Synapse Analytics, and Microsoft Fabric. Expertise in managing large-scale data storage and processing with Azure Data Lake Storage (ADLS), and applying Spark SQL and PySpark for efficient data transformations and analytics. Adept at building automated, scalable data pipelines and workflows, leveraging Azure DevOps for seamless continuous integration and continuous delivery (CI/CD). Proven ability to design and implement optimized data systems that drive business insights, foster data-driven decision-making, and ensure high performance across cloud platforms.
Overview
8
8
years of professional experience
4
4
years of post-secondary education
4
4
Languages
Work History
Senior Software Engineer
EPAM Systems, Inc
07.2023 - Current
- Ingested files from SFTP in to Finance landing zone blob containers.
- Involved in connecting to various sources from microsoft fabric to load the data.
- Involved in building lake house using bronze, silver and gold layers using delta tables and extensively worked on optimizing the pyspark code.
- Involved in writing unit test cases for pyspark code using Pytest framework.
- Extensively worked on python exceptions and data structures list and dictonary.
- Worked on understanding the business logic and writing pyspark code to create views by joining dataframes and aggregating the data based on business logic.
- Extensively involved in code promotion to higher environments using azure devops CI/CD pipelines.
- Created synapse and fabric pipelines to schedule spark notebooks.
- Actively participated in PI planning events and provided key inputs and effort estimations.
- Mentored junior developers, fostering professional growth and enhancing team productivity.
Senior Software Engineer
CGI Inc.
06.2021 - 07.2023
- Designed and developed Dynamic Metadata-driven pipelines to perform data ingestion from Sharepoint, SFTP, Rest API, Azure BLOB and Relational Databases to different layers into the ADLS Gen2 using Azure Data Factory (ADF V2)
- Extensively worked on Data Extraction, Migration, Transformation, Cleansing, Loading data
- Extensively worked on ADF components - Integration Runtime, Linked Services, Data Sets, Pipelines, Activities and Triggers
- Good hands on experience on Azure Data Factory activities such as Copy, Lookup, Get Metadata, if condition, for each, Set Variable, Execute pipeline, web, Filter and wait
- Developed pyspark scripts to load data in to dataframes from adls using mount point and worked on transformations
- Extensively worked on optimizing pyspark application by reducing the data shuffles and other techniques
- Implemented Slowly Changing dimensions using DELTA Lake Merge functionality
- Deployed the codes to multiple environments with the help of Azure Devops CI/CD process
Associate Consultant
Capgemini
09.2019 - 06.2021
- Attended initial discussion meetings and Scrum meetings to collaborate with the Project Manager, Architect, Business Analyst and fellow Developers to understand the Business logic that will be implemented in the development
- Involved in getting the insights of domain knowledge
- Involved in loading data from on premise SQL server to Azure cloud
- Worked on writing Pyspark code to transform the ingested data.
- Responsible for setting end to end Data factory pipelines for full load and incremental load
Associate Consultant
Capgemini
11.2018 - 08.2019
- Developed Pipelines in Data Factory to copy the data from on premise to Data Lake, to enable the archival of processed files, to Process multiple files at the Same Time
- Created Linked Services, Datasets, Key Vault Secrets, Schedule based triggers
- Developed Notebooks in Data bricks using Pyspark to ingest the data from Data Lake to the Staging table after performing data quality checks and transformations
- Created stored procedures to handle the incremental data in Dimension and Fact tables
- Integrated all the notebook and stored procedure activities in the Pipeline
Software Engineer
Dev Information Technology Ltd.
07.2018 - 10.2018
- DATA-SSOT Project
Associate Software Engineer
Tech Mahindra
03.2017 - 07.2017
- Associate Software Engineer
Education
B.Tech -
Annamacharya Institute of Technology & Sciences
Andhra Pradesh 08.2011 - 05.2015
Skills
Pyspark
SQL
Python
Azure Synapse Analytics
Azure Data Factory
undefinedAwards
Sliver Award, 01/20/22, CGI Inc.
Timeline
Senior Software Engineer
EPAM Systems, Inc
07.2023 - Current
Senior Software Engineer
CGI Inc.
06.2021 - 07.2023
Associate Consultant
Capgemini
09.2019 - 06.2021
Associate Consultant
Capgemini
11.2018 - 08.2019
Software Engineer
Dev Information Technology Ltd.
07.2018 - 10.2018
Associate Software Engineer
Tech Mahindra
03.2017 - 07.2017
B.Tech -
Annamacharya Institute of Technology & Sciences
08.2011 - 05.2015
Similar Profiles
Mihir DashMihir Dash
Senior Test Automation Engineer (QA LEAD) at EPAM Systems, IncSenior Test Automation Engineer (QA LEAD) at EPAM Systems, Inc
Sachin Kumar BeheraSachin Kumar Behera
Software Developer A2 at EPAM Systems, IncSoftware Developer A2 at EPAM Systems, Inc
Alena StankevichAlena Stankevich
Support Services Coordinator at EPAM Systems, IncSupport Services Coordinator at EPAM Systems, Inc