Yash Gupta
Senior Software Engineer
Bengaluru
Summary
Results-driven Data Engineer with 4 years of experience in designing and implementing scalable data pipelines, optimizing ETL workflows, and managing comprehensive data lake solutions. Expertise in transforming raw data into actionable insights that drive informed business decision-making, supported by proficiency in SQL, Python, Databricks, ADF, and Talend. Strong foundation in database management and cloud-based data integration complements the ability to collaborate effectively with cross-functional teams to deliver robust, high-performance data solutions aligned with organizational goals. Committed to maintaining high standards of data quality and reliability while continuously learning and adapting to evolving technologies and industry best practices.
Overview
6
6
years of professional experience
5
5
Certifications
Work History
Senior Software Engineer
HCL Technologies
06.2024 - Current
- Built and maintained scalable ETL pipelines in Databricks using PySpark, enabling efficient data processing for analytics and reporting needs across multiple business units.
- Optimized Databricks workflows, reducing job execution time by 20%, and improving reliability across multiple data sources.
- Designed and orchestrated complex ADF pipelines to automate data movement across on-premise and cloud sources, improving data availability and refresh cycles.
- Implemented data quality checks using Python and SQL, improving data accuracy and consistency across pipelines.
- Worked closely with data analysts, business teams, and data scientists to gather requirements and deliver end-to-end data solutions.
- Integrated Power BI with curated datasets from ADF and Databricks, ensuring real-time and accurate reporting.
Senior Data Engineer
Wissen Infotech
04.2023 - 11.2023
- Implemented and validated data quality checks post-migration to ensure completeness, accuracy, and consistency with legacy systems.
- Led the migration of ETL workflows from Talend to Databricks, rewriting data pipelines using PySpark and SQL to improve performance and scalability.
- Implemented and validated data quality checks post-migration to ensure completeness, accuracy, and consistency with legacy systems.
- Converted complex PostgreSQL queries and SQL scripts into Databricks-compatible SQL (Delta Lake), ensuring functional equivalence and performance optimization through rigorous unit and integration testing.
- Converted existing Python scripts to PySpark for scalable data ingestion from multiple sources (SharePoint, flat files, databases), improving performance, and enabling parallel processing in the Databricks environment.
- Documented end-to-end migration workflows, including pre-migration assessments, code transformations, testing strategies, and deployment processes.
Associate Software Engineer
Tata Consultancy Services
03.2021 - 12.2022
- Created custom SQL scripts to transform and move data from staging to analytical, and consumption layers, enabling structured data flow and supporting downstream reporting and analytics.
- Built custom Talend jobs and SQL scripts to automate data ingestion from APIs, databases, and flat files; enabled bi-directional data flows, including database-to-database and database-to-SharePoint list sync.
- Designed and implemented data quality rules to validate ingested HR data, ensuring completeness, accuracy, and consistency across multiple systems.
- Develop and maintain ETL workflows using Talend Open Studio.
- Mentor and train junior data engineers and other team members on best practices and emerging technologies in data engineering.
Forsk Technologies, Jaipur
05.2019 - 06.2019
- Assisted in design and development of signal light algorithms, contributing to enhancement of traffic flow efficiency
- Collaborated with engineering team to test and validate signal light prototype, ensuring functionality and reliability
Education
B.Tech - Information Technology
Jaipur Engineering College And Research Center
Jaipur11-2020
School - XII - Science
Central Academy
Udaipur04-2016
X - undefined
01.2014
Skills
Databricks
Python
PostgreSQL
undefinedCertification
Databricks Certified Data Engineer Associate
TOOLS
- Denodo Platform
- Talend Open Studio
- DBEAVER
- Data Bricks
- Power Apps
- Rally
- Microsoft Azure
Accomplishments
- Databricks Certified Data Analyst Associate
- Databricks Certified Data Engineer Associate
- Databricks Certified Data Engineer Professional
Timeline
Senior Software Engineer
HCL Technologies
06.2024 - Current
Senior Data Engineer
Wissen Infotech
04.2023 - 11.2023
Associate Software Engineer
Tata Consultancy Services
03.2021 - 12.2022
Forsk Technologies, Jaipur
05.2019 - 06.2019
X - undefined
B.Tech - Information Technology
Jaipur Engineering College And Research Center
School - XII - Science
Central Academy
Similar Profiles
Srinivasan NandagopalSrinivasan Nandagopal
ETL Lead /Onshore Technical Business Analyst at HCL Technologies India & HCL Technologies NZ LtdETL Lead /Onshore Technical Business Analyst at HCL Technologies India & HCL Technologies NZ Ltd
Pooja SaxenaPooja Saxena
Senior Analyst at HCL Technologies Pvt. Ltd, HCL technologies Pvt LtdSenior Analyst at HCL Technologies Pvt. Ltd, HCL technologies Pvt Ltd
GIRISH KORIGIRISH KORI
Senior Software Test Engineer / Technical Leader at HCL Technologies Ltd. & HCL GMBHSenior Software Test Engineer / Technical Leader at HCL Technologies Ltd. & HCL GMBH
Abhishek KumarAbhishek Kumar
Data Engineer/Technical Lead at HCL Technologies Limited (HCL Tech)Data Engineer/Technical Lead at HCL Technologies Limited (HCL Tech)
Sabir PashaSabir Pasha
Senior Technical Solutions Engineer at BARRACUDA NETWORKSSenior Technical Solutions Engineer at BARRACUDA NETWORKS