Abhishek Kumar
Data Engineer
Bangalore
Summary
- Data Engineer with 9+ years of experience designing and architecting scalable batch and near real-time data pipelines and cloud-native data platforms across Utilities and Finance domains.
- Demonstrated expertise in building high-performance, cost-efficient data solutions, including large-scale migrations (65TB+ SAP BW/HANA to Azure), robust ETL pipelines
- Built reusable ingestion frameworks that enable enterprise analytics and reporting significantly reducing data source onboarding time and improving development efficiency.
- Implemented robust data governance practices using Unity Catalog, enabling secure access control and compliance.
- Proven track record of driving performance improvements, including achieving up to 87% reduction in processing time through advanced optimization techniques in Spark and SQL.
- Strong collaborator with business stakeholders, analytics, and data science teams to deliver data-driven insights and ML-ready datasets.
- Hands-on experience with CI/CD pipelines using GitHub Actions, and building reusable, production-grade data components to standardize and scale data platforms.
- Brings a strategic mindset with deep technical expertise to design scalable, reliable, and high-impact data solutions that drive business value.
Overview
10
10
years of professional experience
Work History
Data Engineer
Shell India Markets Private Ltd.
11.2023 - Current
- Designed and delivered multiple end-to-end data products within the Enterprise Data Platform (EDP) team, leveraging Databricks Delta Live Tables for ETL pipelines, SQL functions for reporting logic, and Unity Catalog with row-level security for secure data governance.
- Developed scalable data ingestion pipelines, enabling seamless integration of multi-source data (SAP, APIs, databases, SharePoint), enabling seamless integration into Azure.
- Designed and implemented a centralized Common Ingestion Framework to standardize data onboarding across platforms.
- Built reusable ingestion modules, significantly reducing data source onboarding time, and improving development efficiency.
- Established standardized pipeline design patterns, ensuring consistency, scalability, and maintainability across teams.
- Implemented robust data governance practices using Unity Catalog, enabling secure access control, and compliance.
- Created interactive Power BI dashboards that transformed raw data into actionable insights for business stakeholders.
- Redesigned platform architecture and applied Databricks performance optimizations, achieving significant efficiency gains, and simplifying long-term maintainability.
- Enabled secure virtual consumption of data products across subscriptions and platforms, ensuring scalability and compliance.
- Transitioned the Databricks workspace to a higher CIDR range to resolve subnet conflicts with minimal downtime, preserving data integrity, and optimizing cluster performance.
- Built reusable components and a scalable foundation for the Common Ingestion Framework and Databricks applications.
- Migrated over 65 TB of SAP data into Azure, unlocking improved performance, scalability, and analytics capabilities.
- Recognized with the Service Recognition Award (2025) for impactful contributions and platform improvements.
Data Engineer
IBM
09.2022 - 10.2023
- Implemented SAP functionalities including hierarchies and currency conversion by integrating Databricks and Power BI.
- Leveraged expertise in Delta Live Tables (DLT), DBT, and Unity Catalog to deliver secure, scalable, and high-performing data solutions.
- Optimized a critical report using dynamic SQL functions, reducing processing time from 300+ seconds to 39 seconds — an 87% improvement in efficiency.
- Enhanced Spark job performance, ingestion pipelines, and storage efficiency through advanced tuning techniques and diagnostics with the Spark UI.
- Spearheaded three continuous improvement initiatives, driving $168,000 in cost savings and boosting team productivity.
- Built Power BI solutions that transformed raw datasets into actionable business intelligence.
- Implemented CI/CD pipelines with GitHub Actions, enabling automated deployments and streamlined development workflows.
Module Lead
Mindtree
11.2021 - 09.2022
- Handled multiple reads and writes with Azure Data Lake and Blob storage.
- Expertise in writing Spark RDD transformations, Actions, Data Frames, Case classes and Struct Types for the required input data and performed the data transformations.
- Loaded and transformed large sets of semi structured data likes XML, JSON, Avro, ORC, Parquet.
- Have experience in creating pipeline jobs, schedule triggers using Azure data factory.
- Built the data pipeline using Azure Service like Data Factory to load the data from Legacy SQL server to Azure Data Base using Data Factories.
- Performed ETL using Azure Databricks. Was responsible for Optimizing Spark SQL queries that helped in saving Cost to the project.
- Code & peer review of assigned task.
- Unit testing and Bug fixing.
- Involved in working on the Data Analysis, Data Quality and data profiling for handling the business that helped the Business team.
Technology Analyst
Infosys
05.2016 - 11.2021
- Exposure in design and development of solutions for Big Data using the Hadoop eco system technologies (HDFS, Hive, Sqoop, Apache Spark).
- Leveraged Sqoop to import the data from RDBMS into HDFS and worked schema evolution by using AVRO file format.
- Created HIVE table on top of the AVRO file and AVRO schema to handle schema evolution Hands on experience in creating Apache Spark RDD transformations on data sets in Hadoop data lake.
- Involved in working on the Data Analysis, Data Quality and data profiling for handling the business that helped the Business team.
- Code & peer review of assigned task.
- Unit testing and Bug fixing.
- Business Requirement mapping to System Capacity and release.
- Identify, manage and implement technology enhancement/migration like Server, Database and Product upgrades
- Provide quality review and technical oversight for key Data, Integration, Unit Testing, and Environment management deliverables.
- Track all Incidents/ issues logged against applications to their completion. Report Incident Status/ Application health status to senior management and executives on both client and Infosys side.
- Customized Development and implementation of Customer Services applications Perform Root Cause Analysis and plan fixes for system issues using BMC Remedy.
- Plan and implement pro-active and automated monitoring of critical applications.
Education
Graduate -
01-2016
GPA: 70%
Skills
Azure Databricks
Azure Data Factory
Azure Data Lake
Unity Catalog
Hive
Apache Spark
Delta Live Tables (DLT)
DBT
Power BI
Python
SQL
Data pipeline design
Data modeling
Timeline
Data Engineer - Shell India Markets Private Ltd.
11.2023 - Current
Data Engineer - IBM
09.2022 - 10.2023
Module Lead - Mindtree
11.2021 - 09.2022
Technology Analyst - Infosys
05.2016 - 11.2021
Satyabhama University - Graduate,