Vivek Khandelwal
Bengaluru
Summary
Lead Data & AI Engineer with 11+ years of experience designing and delivering scalable data platforms and AI-driven solutions on Azure. Specialized in Azure Databricks, PySpark, real-time streaming, and Generative AI (RAG-based architectures). Proven track record of building production-grade machine learning and GenAI systems, including dispute prediction models with confidence scoring and enterprise chatbot platforms using vector databases. Experienced in leading cross-functional teams, driving architectural decisions, and translating complex technical solutions into business impact. Passionate about building intelligent, scalable, and future-ready data ecosystems.
Overview
11
11
years of professional experience
Work History
Engineering Manager, Data Analytics
A.P. Moller Maersk
03.2022 - Current
AI-Powered Meeting Intelligence , Customer Incentive Platform,
Commercial Waivers Platform , AI-Based Auto Indexing Solution, Ocean Disputes Platform
- Worked with project stakeholders to understand, define and meet project requirements.
- Built scalable ingestion and preprocessing workflows to handle audio inputs, metadata enrichment, and structured storage.
- Implemented validation checks and structured output formatting to ensure data quality, and consistent knowledge generation.
- Developed scalable batch and streaming data pipelines using Spark and Delta Lake to process customer and shipment data.
- Designed an AI-driven pipeline integrating speech-to-text, NLP processing, and structured PDF generation for automated meeting intelligence.
- Built ML models for shipping trend prediction and incentive optimization, ensuring reliable feature engineering and data validation.
- Implemented data quality checks, performance tuning, and Power BI reporting for real-time analytics and sales insights.
- Architected near real-time data pipelines using Spark Structured Streaming and Kafka for waiver reporting and decision systems.
- Designed robust data models, schema enforcement (Delta), and automated data validation to ensure accuracy and consistency.
- Deployed ML-based recommendation engines, driving ~$100M annual revenue impact through workflow automation.
- Engineered ingestion pipelines to process structured and unstructured data (PDF, Word, images) with scalable parsing frameworks.
- Implemented multilingual NLP models with data normalization, metadata tagging, and quality validation layers.
- Optimized indexing workflows, improving processing accuracy and delivering cost realization equivalent to 100 FTEs.
- Leading a team of 6 engineers to build enterprise-grade streaming and batch pipelines using Azure Databricks, ADLS, Spark, and Kafka.
- Designed end-to-end ingestion frameworks with schema evolution, Delta Lake enforcement, incremental loads, and CDC handling.
- Established data quality frameworks, monitoring, performance optimization, and Power BI dashboards for reliable business analytics.
SDE-III
Gap Inc
01.2022 - 03.2022
- Led software engineering initiatives focused on retail modernization in North America.
Executed development and modernization strategies for inventory systems to improve functionality.
Senior Analyst
Sopra Steria
09.2019 - 10.2020
- Executed Invoice Matching V16 implementation for UK grocery retailer, modernizing outdated platform.
Senior System Engineer
Accenture Services Pvt Ltd
02.2018 - 09.2019
- Worked as Senior Software Engineer For UK Based Fashion Retailer
Senior System Engineer
Infosys Ltd
02.2015 - 02.2018
- Functioned as SME for RIB and RMS modules
Engineer
CMC Ltd
10.2014 - 02.2015
- Functioned as SQL Engineer specializing in MP Treasury project.
Education
Bachelor of Technology - Electronics & Comms.
Suresh Gyan vihar University
Skills
- Microsoft Azure (ADLS, AKS, Azure OpenAI)
- Azure Databricks & Delta Lake
- Python
- PySpark & Apache Spark
- Kafka (Real-time Streaming Pipelines)
- Generative AI (RAG, LangChain, LlamaIndex)
- Vector Databases (Redis / Redisearch)
- Machine Learning & MLflow
- Kubernetes & Docker
- Stakeholder Management & Technical Leadership
Projects
Interactive Knowledge Gen Al Agent for System Training
A.P Moller maersk
• Designed and prototyped a n Al-powered meeting processing tool that converts
product demos and discussions into concise PDF summaries with key decisions, next
steps, FAQs, and relevant screenshots.
• Integrated speech-to-text and NLP pipelines to transcribe recordings, extract
highlights, and automatically generate structured FAQs for onboarding, training, and
support teams.
• Developed automated process visualization by generating draw.io flow diagrams
from meeting transcripts, illustrating user journeys, decision points, and data flows.
• Enhanced cross-team knowledge sharing b y creating plug-and-play outputs for
knowledge bases, reducing dependency o n lengthy recordings and improving
accessibility.
Commercial Waivers Platform
• Led the design and implementation o f robust, scalable pipelines for near real-time
reporting on commercial waivers, significantly improving data accessibility and
decision-making speed.
• Spearheaded the deployment of machine learning models that automatically
provide recommendations for waiver approvals and rejections, utilizing historical
data, revenue per customer, and other parameters.
• Drove process automation that enabled annual revenue savings o f 100 million USD
by optimizing workflow efficiencies and automating the waiver decision-making
process.
• Collaborated with cross-functional stakeholders to ensure alignment between
technical solutions and business objectives, while managing project timelines,
resource allocation, and risk mitigation.
• Ensured scalability and sustainability of solutions by implementing best practices in
machine learning model management, data processing, and pipeline orchestration.
Tech Stack: Azure Databricks, PySpark, Python, PostgreSQL, Kafka, Power BI, JIRA,
Workday, Scrum, Agile methologies, Capacity Planning
Awards
- Star Award, A.P Moller Maersk
- Employee of Quarter, A.P Moller Maersk,
- ACE Award, Accenture Ltd, Honored with ACE award for outstanding performance in Project delivery and Client satisfaction.
- INSTA Award, Infosys Ltd, Awarded with INSTA award for client delivery and outstanding performance
Social
- linkedin.com/in/vivek6648
- github.com/vivek6648
Timeline
Engineering Manager, Data Analytics
A.P. Moller Maersk
03.2022 - Current
SDE-III
Gap Inc
01.2022 - 03.2022
Senior Analyst
Sopra Steria
09.2019 - 10.2020
Senior System Engineer
Accenture Services Pvt Ltd
02.2018 - 09.2019
Senior System Engineer
Infosys Ltd
02.2015 - 02.2018
Engineer
CMC Ltd
10.2014 - 02.2015
Bachelor of Technology - Electronics & Comms.
Suresh Gyan vihar University