Summary
Overview
Work History
Education
Skills
Websites
Certification
Timeline
Generic

Pavan K

Bangalore

Summary

Accomplished Senior Data Engineer at Ernst & Young GDS, specializing in Azure Databricks and data governance. Expert in designing robust ingestion frameworks and automating data quality checks. Proven ability to collaborate effectively with business teams, ensuring compliance and enhancing data integrity. Strong analytical skills complemented by a commitment to excellence.

Overview

10
10
years of professional experience
1
1
Certification

Work History

Senior Data Engineer

Ernst & Young GDS
Bangalore
01.2021 - Current
  • Designed Unity Catalog-based ingestion framework to move data from ADLS into volumes, then process into raw and staging Delta tables.
  • Implemented data governance using Unity Catalog schemas, grants, and access control for secure data sharing.
  • Automated schema validation and data quality checks using PySpark before loading into staging.
  • Collaborated with business teams to identify and resolve data anomalies in incoming datasets.
  • CICD development using Databricks Asset bundles.

Senior Data Engineer

Ernst & Young GDS
Bangalore
11.2021 - 04.2025
  • Built ADF pipelines for config-driven ingestion and transformation into governed Unity Catalog Delta tables.
  • Managed data lineage and audit logging via Unity Catalog for compliance reporting.
  • Implemented data quality validation rules and automated error segregation in PySpark.
  • Developed a config-driven framework to dynamically populate control tables post ingestion.

IT Analyst / PySpark Developer

Tata Consultancy Services (TCS)
Londrina
06.2019 - 11.2021
  • Developed generic ingestion templates in PySpark for batch and incremental loads using Unity Catalog raw and curated layers.
  • Created .whl library packages for reusable PySpark functions and deployed in Databricks.
  • Implemented SCD Type 2 logic for historical data tracking in Delta tables.

Data Engineer

Tata Consultancy Services (TCS)
01.2016 - 01.2021
  • Developed generic ingestion templates in PySpark for batch and incremental loads using Unity Catalog raw and curated layers.
  • Created .whl library packages for reusable PySpark functions and deployed in Databricks.
  • Implemented SCD Type 2 logic for historical data tracking in Delta tables.

Earlier Roles

Tata Consultancy Services (TCS)
01.2016 - 01.2019
  • Python developer for web scraping automation using Selenium.
  • L3 support handling code deployments, migrations, and production issue resolution.
  • Developed automation scripts for ARM template deployments and connection updates in ADF.

Education

M.Tech - Thermal Science

Amrita Vishwa Vidyapeetham
Bangalore, India
01.2015

Skills

Cloud platforms and services

  • Azure Databricks and Data Factory
  • Azure Data Lake Storage (ADLS)
  • Data orchestration with Airflow
  • Programming languages: Python, SQL, PySpark

Scripting tools: Shell script, PowerShell

Version control: Git, Bitbucket

  • Data governance and security strategies
  • Unity Catalog management
  • Data visualization: Tableau, Power BI
  • Project management with Azure Devops ,Jira

Certification

  • Microsoft Certified: Azure Data Engineer Associate (DP-203)
  • Microsoft Certified: Fabric Analytics Engineer Associate (DP-600)
  • Databricks Certified Data Engineer Associate
  • Microsoft Certified: Azure Fundamentals (AZ-900)

Timeline

Senior Data Engineer

Ernst & Young GDS
11.2021 - 04.2025

Senior Data Engineer

Ernst & Young GDS
01.2021 - Current

IT Analyst / PySpark Developer

Tata Consultancy Services (TCS)
06.2019 - 11.2021

Data Engineer

Tata Consultancy Services (TCS)
01.2016 - 01.2021

Earlier Roles

Tata Consultancy Services (TCS)
01.2016 - 01.2019

M.Tech - Thermal Science

Amrita Vishwa Vidyapeetham
Pavan K