Shubham Jaiswal
Senior Data Engineer
Summary
Experienced Senior Data Engineer with 5+ years of expertise in designing and optimizing scalable ETL pipelines using Apache Spark (Scala) , Kafka , Oozie , and Hive . Skilled in ensuring data accuracy , resolving complex issues, and leading seamless data migrations . Currently delivering high-performance data solutions at Target , driving efficiency and scalability across critical data processes.
Overview
5
5
years of professional experience
18
18
years of post-secondary education
2
2
Certifications
Work History
Senior Data Engineer
Target
04.2022 - Current
- Designed and implemented scalable ETL workflows using Apache Spark , processing diverse data sources including Kafka JSON , Hive tables , and APIs to support data integration and analysis.
- Built and maintained data validation and control mechanisms , ensuring data accuracy through recovery APIs and automated solutions to resolve discrepancies.
- Developed data quality assurance scripts to validate post-ETL data, automating alerts through email and Slack for anomaly detection and quick resolution.
- Optimized Spark workflows and cluster performance , applying code enhancements and resource management techniques to boost execution speed and efficiency.
- Led data migration projects , ensuring seamless transitions and minimal operational disruptions by implementing strategic migration plans and risk mitigation techniques.
- Troubleshot and resolved user-reported data issues , debugging workflows and providing clear, actionable resolutions to enhance user satisfaction and data integrity.
Senior System Engineer
Infosys (Client: BMG)
11.2019 - 04.2022
- Migrated legacy data processing from Oracle to Apache Spark , significantly improving royalty calculation processes for artists by enhancing scalability and performance.
- Developed and optimized Spark and Scala scripts for sales data processing and royalty calculations , streamlining complex data workflows.
- Managed Hive and PostgreSQL tables, creating and maintaining stored procedures and user-defined functions to ensure data integrity and efficient querying.
- Monitored and troubleshot production environments, resolving performance issues to ensure smooth and efficient royalty runs with minimal downtime.
- Collaborated with support and testing teams to resolve issues promptly, ensuring consistent and timely task completion.
- Managed and tracked development tasks using Jira , ensuring that all tasks were completed within sprint cycles and progress was maintained.
Education
Bachelor of Technology - Computer Science
Inderprastha Engineering College
04.2001 - 01.2019
Skills
Big Data Technologies: Apache Spark, Hive, Oozie, Sqoop, Kafka, HDFS, Data Lakes, ETL Processes
undefinedCertification
AWS Certified Data Engineer – Associate
Timeline
AWS Certified Data Engineer – Associate
11-2024
AWS Certified Cloud Practitioner
04-2024
Senior Data Engineer
Target
04.2022 - Current
Senior System Engineer
Infosys (Client: BMG)
11.2019 - 04.2022
Bachelor of Technology - Computer Science
Inderprastha Engineering College
04.2001 - 01.2019
Similar Profiles
Agustin LopezAgustin Lopez
Team Member at TargetTeam Member at Target
Nazya RoperNazya Roper
Food and Beverage/General Merchandise Associate at TargetFood and Beverage/General Merchandise Associate at Target
Jalen MooreJalen Moore
Remodel Team Member at TargetRemodel Team Member at Target
Cassidy RollinsCassidy Rollins
Barista at TargetBarista at Target
Chih Han YuChih Han Yu
Senior Data Engineer / Data Engineer at MicronSenior Data Engineer / Data Engineer at Micron