Verified
This profile is verified using an email address.
Arun Kumar Thippareddy
Hyderabad
Summary
Data Engineer with 9+ years of IT experience, including 4+ years in Azure Data Engineering, 2 years in Big Data technologies, and 3 years in SAP BW/HANA. Experienced in designing and developing scalable ETL/ELT pipelines using Azure Data Factory (ADF), Azure Databricks, PySpark, ADLS Gen2, and Delta Lake. Strong expertise in SQL, Python, Spark, Hadoop, and data warehousing concepts with hands-on experience in incremental loading, data transformation, performance optimization, and cloud-based data solutions. Proven ability to deliver end-to-end data integration solutions in Agile environments.
Overview
1
1
Certification
11
11
years of professional experience
Work History
Azure Data Engineer
Cognitive Techware Pvt Ltd
Hyderabad, India
08.2024 - 06.2026
- Designed and implemented scalable ETL pipelines using Azure Data Factory to ingest, transform, and load data from diverse sources into Azure Data Storage services (Azure Data Lake, Azure Blob Storage, Azure SQL DB), with advanced data processing and analytics performed in Azure Databricks for optimized performance and efficiency.
- Developed and managed end-to-end data integration pipelines in Azure Data Factory by designing Linked Services, Datasets, and Pipelines to seamlessly extract and consolidate data from multiple sources (Azure SQL, Blob Storage, Azure Data Lake), while implementing automated workflow scheduling through triggers to ensure efficient and reliable data processing.
- Developed and managed end-to-end data integration pipelines in Azure Data Factory by designing Linked Services, Datasets, and Pipelines to seamlessly extract and consolidate data from multiple sources (Azure SQL, Blob Storage, Azure Data Lake), while implementing automated workflow scheduling through triggers to ensure efficient and reliable data processing.
- Designed and fine-tuned high-performance SQL queries to build scalable data solutions, ensuring optimal efficiency while meeting both technical requirements and business goals.
- Partnered with cross-functional stakeholders to elicit business requirements, analyze complex data challenges, and escalate mission critical issues to leadership for strategic resolution, ensuring alignment with organizational objectives.
- Collaborated with reporting teams to deliver customized, high-quality datasets optimized for analytics, empowering data-driven decision-making and actionable business insights.
- Engineered robust CI/CD pipelines using Azure DevOps to streamline version control, automate deployments, and ensure seamless integration of data solutions.
- Designed and integrated unit testing frameworks within Azure Data Factory pipelines to rigorously validate data transformations, guaranteeing accuracy and reliability across ETL workflows.
Azure Data Engineer
Awone AI Pvt Ltd
Hyderabad
04.2023 - 03.2024
- Designed and automated data ingestion pipelines in Azure Data Factory (ADF) to extract data from REST APIs, HTTP endpoints, CSV files, and loading raw data into Azure Blob Storage and ADLS Gen2 with optimized partitioning strategies.
- Built ETL workflows using ADF Data Flows to cleanse, aggregate, and transform raw data into analytics-ready formats, ensuring compliance with business logic and schema requirements.
- Structured and optimized data storage in Azure SQL DB and ADLS Gen2, implementing partitioning, compression, and Delta Lake formats to enhance query performance for downstream consumers.
- Secured pipeline credentials and connections by integrating Azure Key Vault to manage secrets for APIs, databases, and linked services, reducing exposure risks by 100%.
- Implemented CI/CD for ADF pipelines via Azure DevOps, automating deployments across environments (Dev/Test/UAT/Prod) using ARM templates, Git-based version control.
- Enhanced pipeline reliability by adding unit tests (data validation, schema checks) and monitoring alerts, reducing data quality issues by 30% post deployment.
- Optimized storage costs by configuring Blob Storage tiers (Hot/Cool) and lifecycle policies, cutting expenses by 40% without impacting data accessibility.
- Collaborated with BI teams to deliver curated datasets in Azure SQL DВ with row level security (RLS), enabling secure self-service analytics in Power BI.
Azure Data Engineer
Tiger Analytics
Hyderabad, India
06.2022 - 02.2023
- Primarily involved in Data Migration using SQL, SQL Azure, Azure Data Lake and Azure Data Factory.
- Developed and maintained scalable data pipelines using Azure Data Factory (ADF) to orchestrate ingestion from SQL Server sources into centralized data lakes.
- Designed ADF Data Flows with conditional logic, lookup activities, and parameterized datasets to handle multiple claim types and data variations across ingestion layers.
- Built layered storage in Azure Data Lake Storage (ADLS Gen2)—implementing raw, curated, and mastered zones with consistent partitioning strategies and metadata tagging.
- Leveraged Azure Databricks (PySpark) to transform and enrich large volumes of healthcare data, including entity matching, surrogate key generation, and late-arriving dimension handling.
- Configured Linked Services and Key Vault integrations to securely manage credentials and secrets used across ingestion and transformation jobs.
- Collaborated in building star schema models for a centralized data warehouse in Azure Synapse, supporting analytics on member-level and provider-level metrics.
- Optimized pipeline performance, reducing execution time by 30% and improving overall efficiency by monitoring activity run metrics through Azure Monitor and applying dynamic scaling in Spark clusters within Databricks.
- Created optimized Databricks notebooks using PySpark for constructing and managing Delta tables, leading to a 50% improvement in data processing efficiency and increase in SQL query performance for analytical workloads.
Data Engineer
Uptech Solutions Pvt Ltd
Hyderabad
04.2019 - 01.2022
- Designed and developed scalable big data applications using Apache Spark, Hadoop, Hive, and cloud platforms to process large-scale structured and unstructured datasets.
- Built and maintained batch and real-time data processing pipelines using Spark Streaming and Hadoop ecosystem components.
- Developed ETL workflows for data ingestion, transformation, and loading from multiple data sources into data lakes and data warehouses.
- Optimized Spark jobs by implementing partitioning, caching, and memory tuning techniques, improving processing performance and reducing execution time.
- Created reusable data processing frameworks and automated workflows to enhance operational efficiency and data reliability.
- Collaborated with cross-functional teams including data architects, business analysts, and data scientists to deliver data-driven solutions.
- Performed data quality validation, troubleshooting, and root cause analysis to ensure accuracy and consistency across datasets.
- Monitored cluster performance and resource utilization, implementing performance improvements and cost optimization strategies.
- Debugged production issues, resolved application defects, and provided ongoing maintenance and support for big data applications.
- Implemented security and governance standards, including access control, data encryption, and compliance requirements.
- Participated in code reviews, unit testing, and CI/CD deployment processes to ensure high-quality software delivery.
- Generated technical documentation, architecture diagrams, and operational procedures for knowledge sharing and project sustainability
SAP BW Consultant
Accenture
Hyderabad, India
04.2015 - 02.2018
- Developed SAP BW data models using InfoObjects, ADSOs, CompositeProviders, and InfoCubes.
- Designed and maintained ETL processes using DataSources, Transformations, DTPs, and Process Chains.
- Created BEx Queries and analytical reports for business users across multiple functional areas.
- Developed ABAP routines for complex transformations and custom business requirements.
- Optimized BW queries and data loading processes, improving reporting performance and system efficiency.
- Performed data validation, troubleshooting, and production support activities.
- Collaborated with business and technical teams to deliver scalable SAP BW reporting solutions.
Education
Master of Science -
Univeristy of Greenwich
LONDON08-2010
Bachelor of Technology -
SVIT
Hyderabad, India05-2008
Skills
Data/Cloud Data Engineering
Azure Data Factory (ADF)
Azure Databricks
Fabric
Azure Data Lake Gen2 (ADLs)
Data Lake Optimizations
SQL
PySpark
SQL Server
Python Programming
ETL Processes
Cloud-based Data Solutions
Microsoft Fabric
Azure DevOps
Certification
- DP-700 (Microsoft Fabric Azure Data Engineer Associate)
- DP-750(Azure Databricks Data Engineer Associate)
- DP-900 (Microsoft Azure Fundamentals)
- Databricks Lakehouse Fundamentals
- Databricks Generative AI, Fundamentals
Languages
English
Bilingual or Proficient (C2)
Telugu
Bilingual or Proficient (C2)
Hindi
Advanced (C1)
Timeline
Azure Data Engineer
Cognitive Techware Pvt Ltd
08.2024 - 06.2026
Azure Data Engineer
Awone AI Pvt Ltd
04.2023 - 03.2024
Azure Data Engineer
Tiger Analytics
06.2022 - 02.2023
Data Engineer
Uptech Solutions Pvt Ltd
04.2019 - 01.2022
SAP BW Consultant
Accenture
04.2015 - 02.2018
Master of Science -
Univeristy of Greenwich
Bachelor of Technology -
SVIT