RAJASHEKAR REDDY PEDDADANNAGARI
SENIOR DATA ENGINEER
Hyderabad,TS
Summary
Senior Data Engineer with 10+ years of experience designing and building scalable data platforms using Azure, Databricks (PySpark), Snowflake, and Apache Kafka. Proven expertise in developing 50+ production-grade ETL/ELT pipelines, processing TB-scale data, and delivering real-time data ingestion solutions. Strong background in migrating legacy systems (DataStage, Teradata) to cloud-native architectures, achieving up to 40% performance improvements and 30% cost optimization. Experienced in multi-cloud environments (Azure, AWS), Infrastructure as Code (Terraform), and building reliable, high-performance data engineering solutions. Recognized for driving end-to-end data platform modernization, optimizing data workflows, and collaborating with cross-functional teams to deliver enterprise-scale data solutions.
Overview
4
4
Certifications
10
10
years of professional experience
Work History
Senior Data Engineer
Optum
11.2021 - Current
- Designed and implemented 50+ production-grade Azure Data Factory (ADF) pipelines to ingest data from 10+ heterogeneous sources, improving data availability SLAs by ~35%.
- Built and optimized large-scale data processing systems using Azure Databricks (PySpark & Snowpark), handling TB-scale structured, semi-structured, and unstructured data.
- Led end-to-end migration of legacy ETL systems (DataStage, Teradata) to cloud-native Azure architecture (ADF + Databricks + Snowflake), reducing infrastructure costs by ~30%.
- Improved ETL performance by ~40% by redesigning transformation logic, partitioning strategies, and optimizing Spark execution plans.
- Developed reusable data ingestion and transformation frameworks, increasing engineering productivity by ~35% and standardizing pipeline development across teams.
- Designed and implemented Snowflake-based data platform, optimizing data loading strategies and compute utilization.
- Built event-driven pipelines using Azure Functions and Logic Apps, enabling near real-time data processing and automation.
- Designed and implemented real-time data ingestion pipelines using Apache Kafka, streaming high-volume data from multiple sources into Snowflake via Azure Databricks, enabling near real-time analytics and significantly improving data latency and availability.
- Utilized Jenkins in CI/CD workflows for deployment automation, later transitioning to Terraform (Infrastructure as Code) for standardized, scalable, and automated infrastructure provisioning.
- Provisioned and managed cloud infrastructure using Terraform, reducing deployment effort and improving consistency across environments.
- Configured and managed Azure resources including Storage Accounts (Blob, ADLS), Virtual Networks, Private Endpoints, and secure data access mechanisms.
- Collaborated with cross-functional teams to validate data, troubleshoot production issues, and ensure high reliability of data pipelines.
- Mentored interns and led development of a Spring Boot-based application, successfully deploying it to production.
Software Engineer
Legatotechnology
11.2019 - 11.2021
- Developed and maintained enterprise-grade ETL pipelines using Teradata, Informatica, and Snowflake, enabling ingestion and processing of large-scale healthcare datasets.
- Built and optimized data workflows integrating data from multiple on-premise and cloud sources (AWS S3, Snowflake), ensuring high data quality and consistency.
- Designed and executed complex SQL queries and SnowSQL scripts, improving query performance by 40–60%, significantly reducing processing time.
- Worked with key AWS services including:AWS Lambda for lightweight data processing and automation tasks
AWS Step Functions for orchestrating multi-step workflows
AWS S3 for scalable storage of raw and processed datasets
AWS CloudWatch for monitoring job execution and troubleshooting failures - Built and supported data ingestion pipelines leveraging S3 and Snowflake, enabling efficient batch data processing and storage.
- Performed query tuning and execution plan optimization, identifying bottlenecks and improving overall system performance.
Associate
Cognizant Technologies Solutions
10.2016 - 11.2019
- Developed and maintained ETL pipelines using Teradata and Informatica, supporting large-scale enterprise data processing and transformation.
- Worked extensively on Teradata data warehouse, designing and optimizing SQL queries for efficient data retrieval and transformation.
- Built data ingestion workflows to load data from multiple source systems into Teradata, ensuring data integrity and consistency.
- Performed query optimization and performance tuning, improving execution efficiency and reducing runtime for critical data pipelines.
- Assisted in designing data models and schema structures, supporting reporting and analytics use cases.
- Collaborated with business and technical teams to analyze requirements and translate them into scalable data solutions.
- Supported data validation, testing, and production deployment, ensuring high-quality and reliable data delivery.
Education
Bachelor of Computer Science Engineering -
Vasavi College of Engineering (Affiliated To OU)
04.2001 -
Skills
Azure Databricks
Azure Data Factory (ADF)
Snowflake (Snowpark, SnowSQL)
Apache Kafka & Real-time Data Streaming
ETL / ELT Pipeline Development
Terraform
Data Migration (DataStage/information → AWS/Azure / Snowflake)
Azure Functions & Logic Apps
AWS (Lambda, Step Functions, S3, CloudWatch)
Advanced SQL & Data Warehousing
Certification
Snowpro
Timeline
Azure Developer Associate
08-2023
Databricks Certified Data Engineer Associate
02-2023
Azure Fundamentals
08-2022
Snowpro
11-2021
Senior Data Engineer
Optum
11.2021 - Current
Software Engineer
Legatotechnology
11.2019 - 11.2021
Associate
Cognizant Technologies Solutions
10.2016 - 11.2019
Bachelor of Computer Science Engineering -
Vasavi College of Engineering (Affiliated To OU)
04.2001 -