Vivek Gupta
Lead Software Engineer
Summary
Lead Software Engineer with 11+ years of experience in designing and delivering scalable big data and cloud-native solutions across AWS and GCP. Proven expertise in building end-to-end Spark pipelines, creating reusable connectors, and integrating Dataproc and EMR for enterprise data workloads. Strong background in API development, team leadership, and performance optimization with a deep focus on reliability and user-centric product design.
Overview
11
11
years of professional experience
4
4
years of post-secondary education
1
1
Certification
2
2
Languages
Work History
Lead Software Engineer
Gathr Data, Inc.
09.2022 - Current
- Designed and implemented a data pipeline for a customer HCS to migrate petabytes of data from MongoDB to Amazon Redshift , enabling efficient large-scale analytics in under 2 months.
- Engineered a highly efficient BigQuery-to-BigQuery pipeline for Boticario Group with custom Upsert and Update logic , overcoming Spark connector limitations and achieving 1.6M record processing in under 4 minutes with optimized, production-grade Spark code.
- Spearheaded end-to-end integration of Dataproc (GCP) and EMR (AWS) within the product to support full Spark pipeline execution and cluster lifecycle management through an intuitive UI, leading a team of 10+ engineers to deliver enterprise-grade, cross-cloud orchestration
- Led a team of software engineers to successfully complete projects within deadlines, ensuring high-quality end products.
Module Lead Software Engineer
Impetus Technologies
06.2019 - 08.2022
- Architected and implemented the Connection Management module, delivering low-latency, scalable REST APIs for creating, updating, and scoping reusable pipeline connections; contributed to backend flow design and ensured efficient access control across user, workspace, and project levels.
- Engineered modular Spark connectors for S3, GCS, and BigQuery with full multi-cloud support , enabling reusable pipeline components across EMR and Dataproc , and streamlining data integration across hybrid cloud environments.
Senior Application Engineer
Oracle India Pvt. Ltd.
11.2016 - 05.2019
- Developed efficient code for high-traffic applications, resulting in reduced server load and faster response times.
- Optimized existing applications by refining algorithms and data structures, leading to increased productivity and user satisfaction.
- Actively participated in code reviews, promoting best practices and improving overall code quality across the team.
- Provided technical guidance to junior engineers, improving overall team efficiency and skill levels.
Programmer Analyst
Cognizant Technologies Solutions
02.2014 - 09.2016
- Facilitated knowledge transfer among team members by creating comprehensive technical documentation for developed solutions.
- Optimized code quality through regular peer reviews, resulting in fewer defects and easier maintenance.
- Mentored junior developers on best practices, helping them grow their skill sets while enhancing overall team capabilities.
Education
B.Tech/B.E. - Computer Science
Rajiv Gandhi Proudyogiki Vishwavidyalaya (RGPV)
10.2009 - 05.2013
Skills
J2EE
undefinedTimeline
Lead Software Engineer
Gathr Data, Inc.
09.2022 - Current
Data Science
06-2019
Module Lead Software Engineer
Impetus Technologies
06.2019 - 08.2022
Senior Application Engineer
Oracle India Pvt. Ltd.
11.2016 - 05.2019
Programmer Analyst
Cognizant Technologies Solutions
02.2014 - 09.2016
B.Tech/B.E. - Computer Science
Rajiv Gandhi Proudyogiki Vishwavidyalaya (RGPV)
10.2009 - 05.2013
Languages
Hindi
English
Accomplishments
- Mentored 5 new employees to bring them up to speed on projects, resulting in quicker overall completion milestones.
- GATE(2013) qualified.
Interests
Visiting historical places
Software
Eclipse, IntelliJ, Git, Maven, Jira
Similar Profiles
Abby LinderAbby Linder
Consumer Engagement Specialist at GATHR GCI OUTDOORSConsumer Engagement Specialist at GATHR GCI OUTDOORS
Abdulla TreekAbdulla Treek
Freelance Data Engineer / Data Analyst at Freelance Data Engineer / Data AnalystFreelance Data Engineer / Data Analyst at Freelance Data Engineer / Data Analyst
Kate HollidayKate Holliday
KEY ACCOUNT MANAGER at THRYV DATA (SENSIS DATA SOLUTIONS)KEY ACCOUNT MANAGER at THRYV DATA (SENSIS DATA SOLUTIONS)
ARNAB KUNDUARNAB KUNDU
Lead Software Engineer/Sr. Software Engineer at Lowes'sLead Software Engineer/Sr. Software Engineer at Lowes's
Ali MuradAli Murad
Lead Software Engineer at I2c Inc.Lead Software Engineer at I2c Inc.