Kaustav Banerjee

Dynamic and results-oriented Azure DataOps and MLOps Architect with over 13+ years of IT experience, including 8+ years in technical leadership roles. Proven expertise in Azure Data Engineering, Application Development, MLOps, and LLMOps, with a strong track record of building and leading high-performing cloud data engineering teams.

Currently leading an Azure Data Engineering team at Capgemini Invent, driving innovation and operational excellence in cloud-native data solutions. Adept at designing and developing ETL pipelines, data models, and data architectures using both on-premises and cloud-based tools.
Skilled in:
Azure Data Factory (pipelines, datasets, linked services, integration runtime, custom activities using C#)
Azure Databricks (PySpark, Spark job orchestration, ADLS Gen2 integration)
Azure Storage Solutions (Blob Storage, Data Lake Store, Cosmos DB, Storage Tables)
MLOps/LLMOps using Azure ML Studio, mlflow, PyTorch, TensorFlow, pandas, Azure DevOps, Kubernetes, and Docker Swarm
On-premises tools: SSIS, SSAS, REST APIs, Power BI
Big Data: Hadoop, Hive QL
Agile/Scrum methodologies
Experienced in building CI/CD pipelines, deploying machine learning models, and implementing data quality frameworks using C# and Hive QL. Passionate about fostering innovation, collaboration, and continuous improvement in cloud data engineering environments.

• Bengali, Read, Write, Speak
• English, Read, Write, Speak
• Hindi, Read, Write, Speak

• DataOps and MLOps Architect, Enterprise, 09/01/23 to Present, Architecting and designing Data Pipelines using Azure Databricks, Azure DataLake, pyspark, Collaborated with data scientists to optimize model performance and reduce inference latency., Deployed ML models on Azure using Docker and Kubernetes, ensuring high availability and scalability., Monitored model performance and system health using Azure Monitor and Log Analytics., Designed and implemented CI/CD pipelines for ML models using Azure DevOps., Designed and implemented CI/CD pipelines for data pipelines.
• Cloud Architect, Enterprise, 02/01/23 – 09/30/23, Working closely with Enterprise Architect’s team to implement FinOps practice., Designed a solution to generate a report which is taking data of cost from three different clouds (Azure, AWS and OCI) to understand monthly and weekly expenses., Implemented DevOps CI CD Pipeline.
• Team Lead (Azure Data Engineering Team), Energy and Resources, 09/01/21 – Present, Architecting and designing Data Pipelines using Azure Databricks, Azure DataLake, pyspark., Designing data pipeline using ADF V2., Data Pipeline development of streaming data., Bronze Silver Gold Architecture., Implementation of Application insight and Azure Monitoring., Architecting and developing Azure DevOps pipelines.
• Team Lead (Modern Data Platform), Pharmaceutical, 04/01/21 – 09/30/21, Architecting and designing Data Pipelines using Azure Databricks, Azure DataLake, pyspark., Designing data pipeline using ADF V2., Implementation of Application insight and Azure Monitoring., Architecting and developing Azure DevOps pipelines.
• Technology Consultant (Azure Data platform), Retail CPG, 04/01/20 – 04/30/21, Architecting and developing different projects using Azure components., Architecting and developing Data Pipelines., Implementation of Application insight and Azure Monitoring., Architecting and developing Azure DevOps pipelines.
• Data Engineer, Retail CPG, 05/01/18 – 04/30/20, Creating Azure Data Factory V2 Pipelines., Creating Azure Databricks Notebook using SCALA., C# Function App., Implementation Application insight and Azure Monitoring., Creating SSAS tabular project and deployment into AAS., Creating Power BI reports., Creating CI CD for different Modern Data-Platform Projects.
• Big Data Analyst, Retail CPG, 02/01/18 – 05/01/18, Creating Azure Data Factory V1 Pipelines., Logic App creation for scheduling and pipeline trigger., Creating HQL Files., Creating Data Quality Framework for HDInsight Cluster., Creating C# Custom Activity Roles and Responsibilities., Preparing PowerShell script for deployment.
• Cloud Engineer, Utility, 02/01/17 – 02/28/18, Creating Azure Data Factory V1 Pipelines., Logic App creation for scheduling and pipeline trigger., Creating HQL Files., Creating Data Quality Framework for HDInsight Cluster., Creating C# custom activity.
• C# MVC Developer, Utility, 06/01/16 – 02/28/17, C# MVC Application development JQuery for client-side operation., API Consumption., Deployment into Azure., Implementing Azure DevOps as code repository., Deployment using Azure DevOps.
• Android Development, Utility, 07/01/15 – 06/30/16, C# MVC Application development Jquery for client-side operation., API Consumption., Deployment into Azure.
• SSIS Developer, Utility, 10/01/14 – 06/30/15, Pulling on-premises data from external servers to SQL Server., Add and configure an OLE DB connection manager., Add a Data Flow task., Error Framework designing.
• MS CRM Integration Developer, Utility, 11/01/13 – 10/31/14, Creating WCF services to integrate between SAP, CRM and GIS., Created Packages, Stored procedure, Triggers.
• C# .NET Developer, India Govt, 01/01/12 – 11/30/13, C# MVC Application development.