Debarun Chatterjee
Manager - Data And AI
Bangalore
Summary
Dynamic Data Engineering Manager / Architect with 13+ years of experience in building, scaling, and managing modern data platforms on Azure, AWS, and Databricks. Proven expertise in cloud data architecture, lakehouse design, ETL modernization, and real-time analytics. Skilled in leading global teams to deliver robust, high-performance data ecosystems that drive actionable business insights.
Specialist in Databricks (Spark, Delta Lake, MLflow), with a deep focus on data lakehouse architecture, cost optimization, and end-to-end cloud migration strategies.
Overview
13
13
years of professional experience
3
3
Certifications
Work History
Manager - Data and AI
EY GDS
02.2021 - Current
- Designed and implemented an enterprise-grade Databricks Lakehouse on Azure, integrating 40+ data sources, and reducing data latency by 70%.
- Led a 20-member global data engineering team, delivering scalable pipelines using ADF, Databricks, and Delta Lake.
- Architected an automated ingestion framework (CDC) using championed CI/CD for Databricks notebooks using Azure DevOps and Terraform-based environment provisioning. and Delta Live Tables), improving reliability by 99.9%.
- Championed CI/CD for Databricks notebooks using Azure DevOps, and Terraform-based environment provisioning.
- Collaborated with stakeholders to design and execute data governance frameworks, ensuring compliance and data integrity across multiple platforms.
- Enhanced customer satisfaction by resolving disputes promptly, maintaining open lines of communication, and ensuring high-quality service delivery.
Lead Data Engineer
Tata Consultancy Services
01.2019 - 02.2021
- Built and managed Spark-based ETL pipelines on AWS EMR, handling over 2 TB of daily data volume.
- Developed a data quality framework using Python and Spark SQL, improving data accuracy by 95%.
- Led data architecture initiatives, integrating cloud solutions to optimize data storage and retrieval, driving operational efficiency.
- Automated daily ETL validation and reporting using Airflow and Lambda, reducing manual overhead by 80%.
Senior Data Engineer
Deloitte USI
02.2017 - 12.2018
- Designed and implemented scalable data pipelines using Scala and Apache Spark, significantly improving data processing efficiency by 70%.
- Led cross-functional teams in the migration of on-premises data solutions to cloud-based platforms, enhancing accessibility and reducing costs.
- Developed automated ETL processes that streamlined data integration from multiple sources, resulting in improved data accuracy and reliability.
- Mentored junior data engineers on best practices in data architecture and system optimization, fostering a collaborative learning environment.
Associate - Projects
Cognizant Technologies Solutions
11.2012 - 01.2017
- Collaborated with cross-functional teams to develop project plans, ensuring alignment with strategic business objectives.
- Ensured adherence to HIPAA and internal compliance standards across 15+ data pipelines and systems, leading to 100% audit compliance and zero security incidents during annual reviews.
- Utilized ticketing systems (e.g., ServiceNow, Jira) to track, prioritize, and report incidents, ensuring accurate and timely updates for stakeholders and users.
- Participated in Agile ceremonies, contributing to sprint planning and retrospectives to improve team efficiency and collaboration.
Education
Masters of Pharmacy - Pharmacy
Birla Institute Of Technology
Mesra, Ranchi 04.2001 -
Bachelors Of Pharmacy - Pharmacy
West Bengal University Of Technology
Kolkata, India 04.2001 -
Skills
Cloud Platforms: Azure (ADF, Synapse, Data Lake, Event Hub, Key Vault) AWS (S3, Glue, Lambda, Redshift, Athena)
Data Processing: Databricks, Apache Spark, PySpark, Delta Lake, Airflow
Data Warehousing: Snowflake, Synapse Analytics, Redshift
Programming: Python, SQL, Scala
DevOps & CI/CD: Azure DevOps, GitHub Actions, Terraform, Docker
Architecture: Data Lakehouse, ETL/ELT Design, Data Governance, CDC Pipelines
Leadership: Data Strategy, Team Building, Project Delivery, Stakeholder Management
GenAI & LLM Integration: Retrieval-Augmented Generation (RAG) on Databricks, Vector Embeddings, LangChain, OpenAI APIs
Certification
Databricks certified developer associate
Timeline
Databricks certified developer associate
08-2025
Microsoft Certified: Azure Data Engineer Associate
09-2023
Manager - Data and AI
EY GDS
02.2021 - Current
AWS Certified Solutions Architect – Associate Certification
08-2020
Lead Data Engineer
Tata Consultancy Services
01.2019 - 02.2021
Senior Data Engineer
Deloitte USI
02.2017 - 12.2018
Associate - Projects
Cognizant Technologies Solutions
11.2012 - 01.2017
Masters of Pharmacy - Pharmacy
Birla Institute Of Technology
04.2001 -
Bachelors Of Pharmacy - Pharmacy
West Bengal University Of Technology
04.2001 -