Harinatha Reddy
Cloud Data Engineer
Hyderabad
Summary
Experienced Cloud and Big Data Engineer with over 9 years in roles such as Lead Programmer, Analyst, and Developer. Demonstrated skill in designing and optimizing ETL processes and constructing scalable data pipelines utilizing AWS services. Proficient in boosting team efficiency through effective leadership, collaboration, and strategic issue resolution.
Overview
9
9
years of professional experience
5
5
years of post-secondary education
Work History
Lead Data Engineer
Virtusa
07.2024 - Current
- Aligned business objectives with technical requirements through close collaboration with product owners providing insights based on available data sources.
- Mentored junior engineers on best practices in data engineering, fostering a culture of continuous learning and improvement within the team.
- Identified areas for improvement within existing ETL frameworks to maximize efficiency without sacrificing accuracy or integrity of results produced during transformations applied on ingested datasets.
- Utilized AWS services to create Data pipeline for Ingestion and Transformations and finally stored Data in Snowflake
Senior Data Engineer
IBM
12.2021 - 06.2024
- Working on Finance modernization from legacy systems in Auto insurance services
- Built End-to-end data pipelines using PySpark for setting up scalable infrastructure in AWS
- Extracted the data from the cross bucket as batches and Implemented data preparation logic to flatten the nested data.
- We did data quality checks along with schema validation hourly, Under the Transformation phase.
- We generated Audit output files containing total day-to-day transactions and premiums by consuming multiple data sources.
- Automated the entire project with help of AWS Event rules.
- Utilized AWS lambda service to store event progression in Dynamo DB and to trigger AWS Step functions.
- Used AWS SNS services to get notifications on incidents and Data Dog for Incident creation in ServiceNow
- Used Terraform to create infrastructure in AWS.
- Used CI/CD integration to deploy code in AWS.
Senior Data Engineer
Wipro
07.2021 - 12.2021
- Generated reports to the client in XML, CSV, and Excel format after validating the alert transactions based on the given conditions
- The generated Reports will be used for finding fraud transactions.
- Built End-to-end data pipelines using Apache Spark for setting up scalable infrastructure in AWS
- Generated Output XML, CSV, Excel files using multiple data sources like Input Excel, NTBS, Proficee by applying transformations logics
- Used CI/CD integration to deploy the jar's in AWS
- Used Airflow DAGs to run Jobs
Data Engineer
TCS
01.2019 - 06.2021
- Developed end-to-end data pipelines using Apache Spark and for setting up scalable infrastructure in AWS. Migrated/Ingested Terabytes of Big data with negligible error percentage.
- Developed a python code to extract the schema of the database and loaded it into AWS RDS.
- Developed a python code to generate config files which are having metadata of input tables.
- Developed a python script to generate Hive DDLs using config files.
- Developed a python script to generate lineage files.
- Developed the data quality framework for data/column check.
Developer
TCS
07.2016 - 12.2018
- I was part of developing the API Services using C# .Net which will connect to mainframe, document printing, and Communication down streams which act as intermediate for UI and exchange the data in JSON format.
- Worked on Agile based project in the development of services and completed it successfully within the time frame.
- Used Gerrit and Git tools for code repositories.
- Used Microsoft SQL Server to store data.
Education
B.Tech - Electronics and Communications
Vignan's University
Guntur, India 05.2011 - 05.2015
Post Graduate Diploma - Artificial Intelligence and Machine Learning
BITS Pilani
Hyderabad 11.2019 - 12.2020
Skills
AWS
PySpark
Python3
AWS Glue
EMR - Serverless
Step-Functions
Event Bridge
Lambda
S3
SNS
Terraform
Hudi
Athena
Workflows
Cloud Watch
SparkSql
PostgressSql
Hive
AWS EMR
EC2
AWS RDS
Dynamo DB
MS SQL
Git
Snowflake
ECS
Accomplishments
- Star of the month
- Best team member
- Service commitment award
Interests
Chess
Cricket
Videos Games
Music
Timeline
Lead Data Engineer
Virtusa
07.2024 - Current
Senior Data Engineer
IBM
12.2021 - 06.2024
Senior Data Engineer
Wipro
07.2021 - 12.2021
Post Graduate Diploma - Artificial Intelligence and Machine Learning
BITS Pilani
11.2019 - 12.2020
Data Engineer
TCS
01.2019 - 06.2021
Developer
TCS
07.2016 - 12.2018
B.Tech - Electronics and Communications
Vignan's University
05.2011 - 05.2015
Similar Profiles
Matheshwaran PMatheshwaran P
Software QA Test Engineer at VirtusaSoftware QA Test Engineer at Virtusa
Nikita SinghNikita Singh
Executive - Facilities and Sustainability at VirtusaExecutive - Facilities and Sustainability at Virtusa
NIFRAZ THAJUDEENNIFRAZ THAJUDEEN
Lead Software Engineer at VirtusaLead Software Engineer at Virtusa
Paul D. Conley IIIPaul D. Conley III
Senior Technical Writer at VirtusaSenior Technical Writer at Virtusa
SRINIVAS NEELAMSRINIVAS NEELAM
Senior Service Engineer at S SCIENTIFIC INSTRUMENTSSenior Service Engineer at S SCIENTIFIC INSTRUMENTS