Pavan K
Bangalore
Summary
Accomplished Senior Data Engineer at Ernst & Young GDS, specializing in Azure Databricks and data governance. Expert in designing robust ingestion frameworks and automating data quality checks. Proven ability to collaborate effectively with business teams, ensuring compliance and enhancing data integrity. Strong analytical skills complemented by a commitment to excellence.
Overview
10
10
years of professional experience
1
1
Certification
Work History
Senior Data Engineer
Ernst & Young GDS
Bangalore
01.2021 - Current
- Designed Unity Catalog-based ingestion framework to move data from ADLS into volumes, then process into raw and staging Delta tables.
- Implemented data governance using Unity Catalog schemas, grants, and access control for secure data sharing.
- Automated schema validation and data quality checks using PySpark before loading into staging.
- Collaborated with business teams to identify and resolve data anomalies in incoming datasets.
- CICD development using Databricks Asset bundles.
Senior Data Engineer
Ernst & Young GDS
Bangalore
11.2021 - 04.2025
- Built ADF pipelines for config-driven ingestion and transformation into governed Unity Catalog Delta tables.
- Managed data lineage and audit logging via Unity Catalog for compliance reporting.
- Implemented data quality validation rules and automated error segregation in PySpark.
- Developed a config-driven framework to dynamically populate control tables post ingestion.
IT Analyst / PySpark Developer
Tata Consultancy Services (TCS)
Londrina
06.2019 - 11.2021
- Developed generic ingestion templates in PySpark for batch and incremental loads using Unity Catalog raw and curated layers.
- Created .whl library packages for reusable PySpark functions and deployed in Databricks.
- Implemented SCD Type 2 logic for historical data tracking in Delta tables.
Data Engineer
Tata Consultancy Services (TCS)
01.2016 - 01.2021
- Developed generic ingestion templates in PySpark for batch and incremental loads using Unity Catalog raw and curated layers.
- Created .whl library packages for reusable PySpark functions and deployed in Databricks.
- Implemented SCD Type 2 logic for historical data tracking in Delta tables.
Earlier Roles
Tata Consultancy Services (TCS)
01.2016 - 01.2019
- Python developer for web scraping automation using Selenium.
- L3 support handling code deployments, migrations, and production issue resolution.
- Developed automation scripts for ARM template deployments and connection updates in ADF.
Education
M.Tech - Thermal Science
Amrita Vishwa Vidyapeetham
Bangalore, India01.2015
Skills
Cloud platforms and services
- Azure Databricks and Data Factory
- Azure Data Lake Storage (ADLS)
- Data orchestration with Airflow
- Programming languages: Python, SQL, PySpark
Scripting tools: Shell script, PowerShell
Version control: Git, Bitbucket
- Data governance and security strategies
- Unity Catalog management
- Data visualization: Tableau, Power BI
- Project management with Azure Devops ,Jira
Certification
- Microsoft Certified: Azure Data Engineer Associate (DP-203)
- Microsoft Certified: Fabric Analytics Engineer Associate (DP-600)
- Databricks Certified Data Engineer Associate
- Microsoft Certified: Azure Fundamentals (AZ-900)
Timeline
Senior Data Engineer
Ernst & Young GDS
11.2021 - 04.2025
Senior Data Engineer
Ernst & Young GDS
01.2021 - Current
IT Analyst / PySpark Developer
Tata Consultancy Services (TCS)
06.2019 - 11.2021
Data Engineer
Tata Consultancy Services (TCS)
01.2016 - 01.2021
Earlier Roles
Tata Consultancy Services (TCS)
01.2016 - 01.2019
M.Tech - Thermal Science
Amrita Vishwa Vidyapeetham