Summary
Overview
Work History
Education
Skills
Certification
Accomplishments
Timeline
Generic

Shekar Raparthi

Hyderabad

Summary

Results-driven Databricks Developer with hands-on experience in building scalable data pipelines using PySpark, Spark SQL, and Databricks SQL. Skilled in Delta Lake, Lakehouse architecture (bronze–silver–gold), and automating workflows using Databricks Workflows. Strong ability to optimize data processes, ensure data quality and governance with Unity Catalog, and collaborate with cross-functional teams to deliver clean, reliable, analytics-ready datasets. Continuously learning and adapting to new Databricks features to drive innovation and improve performance.

Overview

4
4
years of professional experience
1
1
Certification

Work History

Systems Engineer

Tata Consultancy Services
Hyderabad
08.2025 - Current
  • Developed and maintained scalable ETL/ELT data pipelines using Databricks SQL and PySpark to support business reporting, analytics, and downstream consumption.
  • Built high-performance Spark applications using PySpark and Spark SQL for data extraction, transformation, and aggregation across multiple file formats (CSV, JSON, Parquet, Delta).
  • Designed and maintained bronze–silver–gold Lakehouse architecture, ensuring clean, curated, and production-ready data layers aligned with business requirements.
  • Built streaming and incremental ingestion pipelines using Auto Loader, Structured Streaming, and Delta Live Tables (DLT) to support real-time and near-real-time data processing.
  • Optimized data workflows using Delta Lake features such as ACID transactions, schema evolution, time travel, and Z-Ordering to ensure accuracy, reliability, and query performance.
  • Implemented and managed Databricks Workflows/Jobs to automate ingestion, transformation, validation, and maintenance tasks, improving operational efficiency.
  • Collaborated with cross-functional teams to gather data requirements, resolve complex data issues, and translate business needs into technical data solutions.
  • Performed continuous performance tuning and cost optimization by improving Spark job execution plans, optimizing joins, partitioning strategies, caching, and cluster configurations.
  • Ensured data quality, security, and governance using Unity Catalog, role-based access controls, audit logging, and documentation of end-to-end processes for knowledge sharing.

Azure Data Engineer

Accenture
Hyderabad
12.2021 - 08.2025
  • Designed and implemented scalable ETL pipelines with Azure Data Factory and Azure Databricks, integrating multiple data sources and enhancing data processing efficiency by 40%.
  • Developed optimized PySpark and Spark SQL jobs for large-scale data transformation and processing, cutting manual intervention by 50%.
  • Built and automated ADF workflows including linked services, datasets, triggers, and integration runtimes for seamless data movement.
  • Integrated ADLS Gen2, external databases, and Azure Synapse Analytics to provide unified data access and support analytical reporting.
  • Collaborated with cross-functional teams to deliver clean, structured datasets for analytics and business decision-making.

Education

B. Com - Computer Applications

Osmania University
Hyderabad, India
08.2021

Skills

  • Programming languages: Python and SQL
  • Data engineering tools: PySpark, Azure Data Factory, Azure Databricks, and Azure Synapse Analytics
  • Tools and technologies: Power Apps, Power Automate, and SharePoint

Certification

  • Databricks Certified Data Engineering Associate
  • AZ-900: Azure Fundamentals
  • DP-203: Azure Data Engineer Associate
  • DP-100: Microsoft Certified Azure Data Scientist Associate

Accomplishments

  • Top 5 finalist in the Innovation Contest: Developed a tool using Power Apps, Power Automate, and SharePoint to reduce manual work and enhance efficiency, recognized as one of the top five innovations in an internal contest
  • Process Automation Achievement: Implemented automation solutions that streamlined operations, reduced manual effort, and increased productivity
  • Technology-driven impact: Contributed to digital transformation by leveraging the Microsoft Power Platform for process optimization and workflow automation

Timeline

Systems Engineer

Tata Consultancy Services
08.2025 - Current

Azure Data Engineer

Accenture
12.2021 - 08.2025

B. Com - Computer Applications

Osmania University
Shekar Raparthi