Summary
Overview
Work History
Education
Skills
Accomplishments
Timeline
AssistantManager

Aditya Roy Choudhury

Data Engineer
Pune

Summary

Data Engineer with 4.5 years of experience in building high-performance ETL pipelines, cloud-native data solutions, and data lakehouse architectures using Databricks, PySpark, Azure, and AWS. Adept at driving data migration, automation, and analytics enablement for 3 Fortune 500 companies, with a strong focus on scalability, reliability, and business impact.

Overview

5
5
years of professional experience

Work History

Systems Engineer - Data Engineer

Tata Consultancy Service Ltd
10.2023 - Current
  • Modernized ETL pipelines, migrating from AWS Redshift and DBT to Databricks for a U.S.-based food wholesale company, United Natural Foods (UNFI), improved data processing speed by 40% and reduced infrastructure costs by 25%.
  • Implemented Medallion Architecture on Databricks, leveraging Autoloader for raw data ingestion, ensuring 99.9% data integrity through automated conversion and cleaning, optimizing query performance for analytics, developed Delta Live Table (DLT) pipelines, and configured jobs to streamline data-loading workflows.
  • Developed dashboards and reports in Power BI and Databricks, integrating Databricks data for actionable insights, and improved decision-making.
  • Utilized Databricks Bundle and GitLab CI/CD for seamless version control, collaborative development, and faster, more reliable deployments.
  • Resolved data inconsistencies and conducted client demos, ensuring data accuracy, aligning requirements, and driving successful project delivery.
  • Contributed to two additional projects post-migration, focusing on Gold Layer query development, enhancement, optimization, and validation of query issues to ensure consistency and reliability across workflows.

Senior Systems Engineer

Infosys Ltd
10.2022 - 09.2023
  • Worked as a Data Engineer for one of the biggest beverage company clients: Starbucks.
  • Developed, deployed, and maintained data pipelines, ETL processes, and data integration workflows using Azure Databricks, Azure Data Lake Storage, Airflow, GitHub, and Jenkins, reducing data processing time.
  • Skillfully handled diverse data sources and formats, such as CSV, JSON, Parquet, and Delta tables.
  • Automated data pipelines and workflows, validating retail and customer data using SQL and PySpark in Databricks.
  • Conducted an in-depth system analysis, identifying and resolving critical data pipeline issues, which decreased downtime by 30%.
  • Implemented and maintained data analytics protocols, standards, and manuals, including conducting training sessions for 12+ co-workers, improving team proficiency in data engineering tools.

Systems Engineer

Infosys Ltd
03.2021 - 09.2022
  • Successfully managed ETL (Extract, Load, and Transform) operations on the e-commerce data pipeline for a prominent sports company, Adidas, handling vast data volumes.
  • Utilized Databricks, PySpark, SQL, GIT, Bitbucket, and Jenkins in an Agile methodology to ingest data from the Legacy SAP Business Warehouse into the Databricks Lakehouse (ADLS Gen 2).
  • Completed Infosys training on SQL, SAP BW/BI, and data engineering with high-performance grades.
  • Created jobs for scheduling daily data loads to the Lakehouse, and performed data modeling for specific data assets.
  • Enhanced the HR system in SAP BW by adding metrics and attributes to tables, resulting in faster query results, tested through MicroStrategy.

Education

Bachelor's of Technology - Biotechnology

Haldia Institute of Technology
Haldia, India
07-2020

Indian School Certificate (ICSE/ISC) - Science

St. Xavier's School
Sahibganj, India
03-2016

Skills

Python, SQL & Pyspark

undefined

Accomplishments

  • Databricks Certified Data Engineer Associate
  • Microsoft Certified: AZ-900, DP-900, and AI-900
  • AWS Certified Cloud Practitioner
  • 5-Star HackerRank in Python and SQL
  • GATE 2020 Qualified (GEN)

Timeline

Systems Engineer - Data Engineer

Tata Consultancy Service Ltd
10.2023 - Current

Senior Systems Engineer

Infosys Ltd
10.2022 - 09.2023

Systems Engineer

Infosys Ltd
03.2021 - 09.2022

Bachelor's of Technology - Biotechnology

Haldia Institute of Technology

Indian School Certificate (ICSE/ISC) - Science

St. Xavier's School

Similar Profiles

CREATE PROFILE
Aditya Roy ChoudhuryData Engineer