Verified
This profile is verified using an email address.
Lav Kumar Shukla
Bengaluru
Summary
Dynamic Senior Data Engineer with a proven track record at IBM, specializing in Kafka and Spark development. Successfully led teams to enhance data processing efficiency and optimize ETL workflows. Adept at cross-functional collaboration and problem-solving, driving impactful data architecture solutions that support high-volume data ingestion and ensure system reliability.
Overview
10
10
years of professional experience
Work History
Senior Data Engineer
IBM
Bengaluru
09.2026 - Current
- Developed ETL workflows using Apache Spark and Hadoop frameworks.
- Designed data pipelines to integrate diverse data sources, enhancing data accessibility for stakeholders.
- Developed data pipelines for ingestion and processing of large datasets, ensuring data availability for analytics.
- Collaborated with cross-functional teams to define data requirements, facilitating alignment on project goals.
Lead Data Engineer
Waisl limited
Bengaluru
07.2025 - 09.2026
- - Spearheaded the design and implementation of scalable data pipelines, improving data processing efficiency by 30% and enabling timely, data-driven decision-making
- - Led a cross-functional team of 8 data engineers to optimize ETL workflows, resulting in a 25% reduction in system downtime and enhanced data quality
- - Developed and maintained robust data architecture solutions, supporting daily data ingestion of over 10 million records and ensuring 99.9% system reliability
- Developed data pipelines that enhanced data flow and management efficiency.
- Collaborated with cross-functional teams to gather and document data requirements, ensuring alignment with business goals.
- Established data governance standards that upheld data quality and regulatory compliance.
Senior Consultant
HCL
08.2021 - 06.2025
- Built big data and Spark solutions for the banking domain project supporting Singapore and Vietnam.
- Delivered the second banking services project on schedule for DB UK banking services.
- Managed production service SL3 support and completed codebase rework plus new feature delivery.
- Analyzed client needs to create tailored consulting strategies and recommendations.
- Facilitated workshops to gather requirements and align project goals with stakeholders.
- Built big data and Spark solutions for banking domain project supporting Singapore and Vietnam.
- Delivered second banking services project on schedule for DB UK banking services.
- Managed production service SL3 support and completed codebase rework plus new feature delivery.
Confluent kafka developer
Deloitte Consulting India Private Limited
01.2021 - 05.2021
- Implemented Kafka Confluent solutions, improving healthcare data management processes.
- Developed software solutions using Java and Python for client projects.
- Collaborated with cross-functional teams to gather requirements and define project scope.
- Conducted code reviews to ensure adherence to best practices and quality standards.
Hadoop Developer
Xebia IT Architects
10.2019 - 11.2020
- Banking and Finance
- Developed and maintained Hadoop frameworks, enhancing large-scale data processing capabilities.
- Implemented ETL processes for data integration and transformation, streamlining data workflows.
- Collaborated with data scientists to optimize data storage solutions, improving data accessibility and retrieval.
Sr Technical Engineer
Intellipaat.com
08.2016 - 05.2019
- Leveraged big data technologies, including Hadoop, Spark, and Scala, to extract insights from large datasets.
- Led technical projects to enhance system performance and reliability.
- Mentored junior engineers in troubleshooting and problem-solving techniques.
- Developed technical documentation for system configurations and processes.
Education
B.Tech/B.E. -
Bansal Institute of Science & Technology
01-2014
12th - English
SB College Ara
01-2010
10th - English
High School Karari
01-2008
Skills
- Data pipeline design
- ETL optimization
- Kafka Spark
- Hadoop and HDFS management
- Big data frameworks
- Hive KSQL
- Scala programming
- Python programming
- Machine learning models
- Data visualization tools
- AWS and Azure
- Shell scripting languages
- Cross-functional teamwork
- Agile practices
- Problem solving
- Team leadership
- Data modeling
Timeline
Senior Data Engineer
IBM
09.2026 - Current
Lead Data Engineer
Waisl limited
07.2025 - 09.2026
Senior Consultant
HCL
08.2021 - 06.2025
Confluent kafka developer
Deloitte Consulting India Private Limited
01.2021 - 05.2021
Hadoop Developer
Xebia IT Architects
10.2019 - 11.2020
Sr Technical Engineer
Intellipaat.com
08.2016 - 05.2019
B.Tech/B.E. -
Bansal Institute of Science & Technology
12th - English
SB College Ara
10th - English
High School Karari
Extracurricular Activities
Team Building Activity, Participated in team building programs and training sessions for colleagues after project completion.
Languages
- Hindi
- English
- Hindi
Languages
English, Hindi
Hobbies
Music's and sports on weekend.
Projects
- Matrix migration to RFDS, migration oracle to big data
- Farud Dedication In BANKING, We are working on Banking domain project
- Data Migration From Mainframe System To AWS, Data migration from mainframe to AWS. We have to move data batch processing and Real-time processing both. we have to migrate and process both kind of data with a lot of transformation and reduce the time to process the data for the client.
- Gst Reconciliation And Divident Tax, calculating GST and Cess and Surcharge. we are using pyspark with databricks platform.
- Toyota Finance, This project aim to do data Aggregation and data analytics on top of raw data which comes from various showroom. dataset contains PDF image records about purchase from multiple showrooms of cars.
- B2B App ETL pipeline, 1. I worked on building etl pipeline through nifi to ingest the data from kloudspot api do transformation through groovy script and store data in Postgres's table., 2. i build data ingestion pipeline to ingest data from multiple source connectors through nifi and send data to Kafka topic and store it as raw bronze layer data and same data read in flink job to transformation and kpi dashboard data for computation in flink job and both data store in silver layer topic and over iceberg table. Now again flink job was there to read data from silver topic and do transformation and make it gold data and store in Postgres's and iceberg both table
Extra curricular
- Team building activity
- Participate of team building programs and training session to our colleague after completion of any project in house training.