Summary
Overview
Work History
Education
Skills
Websites
Certification
TOOLS
Accomplishments
Timeline
Generic
Yash Gupta

Yash Gupta

Senior Software Engineer
Bengaluru

Summary

Results-driven Data Engineer with 4 years of experience in designing and implementing scalable data pipelines, optimizing ETL workflows, and managing comprehensive data lake solutions. Expertise in transforming raw data into actionable insights that drive informed business decision-making, supported by proficiency in SQL, Python, Databricks, ADF, and Talend. Strong foundation in database management and cloud-based data integration complements the ability to collaborate effectively with cross-functional teams to deliver robust, high-performance data solutions aligned with organizational goals. Committed to maintaining high standards of data quality and reliability while continuously learning and adapting to evolving technologies and industry best practices.

Overview

6
6
years of professional experience
5
5
Certifications

Work History

Senior Software Engineer

HCL Technologies
06.2024 - Current
  • Built and maintained scalable ETL pipelines in Databricks using PySpark, enabling efficient data processing for analytics and reporting needs across multiple business units.
  • Optimized Databricks workflows, reducing job execution time by 20%, and improving reliability across multiple data sources.
  • Designed and orchestrated complex ADF pipelines to automate data movement across on-premise and cloud sources, improving data availability and refresh cycles.
  • Implemented data quality checks using Python and SQL, improving data accuracy and consistency across pipelines.
  • Worked closely with data analysts, business teams, and data scientists to gather requirements and deliver end-to-end data solutions.
  • Integrated Power BI with curated datasets from ADF and Databricks, ensuring real-time and accurate reporting.

Senior Data Engineer

Wissen Infotech
04.2023 - 11.2023
  • Implemented and validated data quality checks post-migration to ensure completeness, accuracy, and consistency with legacy systems.
  • Led the migration of ETL workflows from Talend to Databricks, rewriting data pipelines using PySpark and SQL to improve performance and scalability.
  • Implemented and validated data quality checks post-migration to ensure completeness, accuracy, and consistency with legacy systems.
  • Converted complex PostgreSQL queries and SQL scripts into Databricks-compatible SQL (Delta Lake), ensuring functional equivalence and performance optimization through rigorous unit and integration testing.
  • Converted existing Python scripts to PySpark for scalable data ingestion from multiple sources (SharePoint, flat files, databases), improving performance, and enabling parallel processing in the Databricks environment.
  • Documented end-to-end migration workflows, including pre-migration assessments, code transformations, testing strategies, and deployment processes.

Associate Software Engineer

Tata Consultancy Services
03.2021 - 12.2022


  • Created custom SQL scripts to transform and move data from staging to analytical, and consumption layers, enabling structured data flow and supporting downstream reporting and analytics.
  • Built custom Talend jobs and SQL scripts to automate data ingestion from APIs, databases, and flat files; enabled bi-directional data flows, including database-to-database and database-to-SharePoint list sync.
  • Designed and implemented data quality rules to validate ingested HR data, ensuring completeness, accuracy, and consistency across multiple systems.
  • Develop and maintain ETL workflows using Talend Open Studio.
  • Mentor and train junior data engineers and other team members on best practices and emerging technologies in data engineering.

Forsk Technologies, Jaipur
05.2019 - 06.2019
  • Assisted in design and development of signal light algorithms, contributing to enhancement of traffic flow efficiency
  • Collaborated with engineering team to test and validate signal light prototype, ensuring functionality and reliability

Education

B.Tech - Information Technology

Jaipur Engineering College And Research Center
Jaipur
11-2020

School - XII - Science

Central Academy
Udaipur
04-2016

X - undefined

01.2014

Skills

Databricks

Python

PostgreSQL

undefined

Certification

Databricks Certified Data Engineer Associate

TOOLS

  • Denodo Platform
  • Talend Open Studio
  • DBEAVER
  • Data Bricks
  • Power Apps
  • Rally
  • Microsoft Azure

Accomplishments

  • Databricks Certified Data Analyst Associate
  • Databricks Certified Data Engineer Associate
  • Databricks Certified Data Engineer Professional

Timeline

Senior Software Engineer

HCL Technologies
06.2024 - Current

Senior Data Engineer

Wissen Infotech
04.2023 - 11.2023

Associate Software Engineer

Tata Consultancy Services
03.2021 - 12.2022

Forsk Technologies, Jaipur
05.2019 - 06.2019

X - undefined

B.Tech - Information Technology

Jaipur Engineering College And Research Center

School - XII - Science

Central Academy
Yash GuptaSenior Software Engineer