Shivam Kumar

Senior Data Engineer

Deloitte USI

09.2022 - Current

Led the development, optimization, and customization of the analytics product to elevate audit quality and enable enterprise-wide data-driven decisions, leveraging cost-effective models to maximize impact.
Delivered end-to-end, scalable data pipelines and workflows across batch and streaming use cases, improving execution times, resource consumption, and audit analytics turnaround.
Utilized PySpark, Azure Databricks, SQL, and Tableau to extract, transform, load and visualize audit data from SAP, Oracle, and other enterprise systems through resilient data pipelines.
Implemented data governance with Unity Catalog in Azure Databricks to manage fine-grained access, lineage, and compliance across datasets and analytics assets.
Built and maintained production-grade pipelines using Azure Data Factory and Azure Databricks, ensuring reliable ingestion, transformation, and loading to Azure Data Lake storage.
Developed Databricks PySpark jobs to systematically clean and standardize data, including null checks, value normalization, and schema validation.
Implemented an automated data quality validation framework, executing 30+ rule-based checks with PySpark and in-platform DIC rulebooks to ensure accuracy and regulatory compliance.
Implemented Delta Lakehouse architecture on Parquet to enable ACID transactions, enforce schema integrity, and deliver modern warehouse capabilities for reliable analytics and governance.
Conducted performance tuning and troubleshooting of Databricks jobs, improving efficiency, scalability, and cost-effectiveness.
Applied Spark optimization techniques such as caching, map, reduceByKey, and repartitioning, to accelerate data processing and reduce compute overhead. Addressed data skew with salting technique and minimized shuffle costs using broadcast joins.
Built and deployed an Azure OpenAI–powered RAG GENAI chatbot that streamlines review of 10,000+ audit documents while achieving 95% accuracy in anomaly detection and risk assessment.
Hands on experience in building interactive, user-friendly dashboards in Tableau and PowerBI that drive data-driven decisions, translating complex data into actionable insights.
Implemented CI/CD pipelines in Azure Databricks using Azure DevOps, integrating Git for version control, automated testing, and continuous deployment.
Worked in Agile Methodologies.
Mentored junior Spark developers and data engineers, fostering best practices in coding standards, performance optimization, and cloud data engineering.
An Excellent Team Player with a good problem-solving approach strong communication, leadership skills, and ability to work in a time-constrained and team-oriented environment and independently with minimal supervision to meet deadlines.

Similar Profiles

GURUPRASATH KRGURUPRASATH KR

Senior Consultant at Deloitte USI Consulting - Deloitte DigitalSenior Consultant at Deloitte USI Consulting - Deloitte Digital

Akhtar KhanAkhtar Khan

DC Senior Consultant at Deloitte Consulting India Private Limited (Deloitte USI)DC Senior Consultant at Deloitte Consulting India Private Limited (Deloitte USI)