Summary
Overview
Work History
Education
Skills
Certification
Websites
Languages
Timeline
AdministrativeAssistant

Ayush Gupta

Pune

Summary

Experienced Data Engineer with 9+ years in developing high-performance big data solutions for leading financial and telecom organizations. Currently, I am a Senior Data Engineer at Mastercard, focusing on Spark, Scala, Python, SQL & AWS. Expertise in automating ETL processes and transforming outdated systems into efficient data pipelines.

Overview

9
9
years of professional experience
1
1
Certification

Work History

Senior Software Developer

Mastercard
PUNE
01.2023 - Current
  • Designed and developed automated batch and streaming processes for merchant location management using Spark and Kafka.
  • Engineered merchant data standardization pipelines utilizing Apache Spark and machine learning models.
  • Built aggregation logic to produce insightful reports for enhanced merchant profiling and analytics.
  • Migrated legacy cardholder processing jobs to NiFi, AWS, and Spark for improved efficiency.

Senior Software Developer

Credit Suisse
PUNE
12.2021 - 12.2022
  • Developing & Designing Pipeline to automate calculation related to Transfer Pricing and create business report as per banking standard.
  • Working with microservices, spark, scala & impala to develop framework for ingestion, transformation & publishing on Cloudera Hadoop Platform.
  • POC: Developed Kubernetes Operator for Apache Spark from Open-Source Spark operator project.
  • Developed Docker Images for Apache Spark and Spark Operator.
  • Worked with DevOps to create a full CI/CD.

Associate

Deutsche Bank
PUNE
11.2020 - 11.2021
  • Designed Regulatory Reporting Framework that ingests, validates, adjusts, processes and publishes data to downstream systems for reporting purpose.
  • Worked on Designing Pipeline for UK based Regional Regulatory reporting framework.
  • Developing Spark code for Ingestion, processing and publishing layer, to extract data and transforming the same using Scala and generating output file on Hadoop Platform.
  • Worked on Spark Tuning, performance related issues and Memory Management.
  • Enhance Scala code to read DRL to generate SQL queries.
  • Designed and implemented API’s and REST services with SCALA (Spark)
  • Migrated Oracle based processes into Hadoop based process.

Software Developer

AMDOCS
PUNE
01.2019 - 10.2020
  • Designed & Developed Data Validation Framework that help to analyze and validate end to end Data.
  • Worked on Performing extraction, transformation and loading of data to HBase and Hive tables through Spark and Kafka and in Real Time.
  • Developed Data Distribution tool for Client that help them to get all the information related to data using Scala.
  • Developed Spark Scala tool to read Kafka Consumer Topic
  • Wrote Scala, Unix Scripts to grep info from Large Avro data as well as publishing HTML reports to Client.
  • Configured OOZIE workflow using Shell Script.
  • Handling, managing, installing & configuring all type of environment like SIT, FAT, UAT, NFT, BAU, PROD.
  • Generated Graphical Visual Reports Using Python Pandas.
  • Created automation scripts using Spark Scala which reduced real -time data backlog by 50%.
  • Individually handled AT&T, PLDT, AIS Thailand, VFUK, VFIR, VFDE accounts.
  • Improving and enhancing performance into Hadoop ecosystem in process level.
  • Worked on Hadoop Mapr/Cloudera Cluster, HBase , Kafka, MySql DataBase.

Software Engineer

CGI
HYDERABAD
02.2016 - 12.2018
  • Created Hive tables to store the processed results in a tabular format.
  • Loaded unstructured data into Hadoop File System (HDFS).
  • Developed Spark code for loading data from legacy data warehouse and CSV files to Hive staging area as project design demanded.
  • Storing data as partitions on cluster in various file formats like Avro and Parquet.
  • Build and deploy compiled code to IIS servers from TFS build system (CI & CD Implementation).
  • Compile the code from various development phases, package the compiled code and deploy it in Application server.

Education

MTech - Data Science & Engineering

BITS
Pilani
09-2025

Bachelor of Technology - ECE

GNIT
Kolkata
08-2015

Skills

  • Big data technologies: Spark, HDFS, Hive, MapReduce, Oozie, Livy, Flink, Kafka
  • Programming languages: Scala, PL/SQL, SQL, Unix, Core Java, Python
  • Distribution platforms: Cloudera, MapR, Databricks
  • Database management: MongoDB, Oracle, MySQL
  • DevOps tools: Kubernetes, Docker, Bitbucket
  • NoSQL databases: MongoDB, HBase
  • Computing environments: Ubuntu, Linux
  • ETL tools: Nifi
  • Cloud services: AWS (S3, Glue, Lambda, EC2)
  • Caching solutions: Redis

Certification

Certified SAFe Practitioner, Scaled Agile, Inc., 2021, 93

Languages

English
First Language
Hindi
Advanced (C1)
C1

Timeline

Senior Software Developer

Mastercard
01.2023 - Current

Senior Software Developer

Credit Suisse
12.2021 - 12.2022

Associate

Deutsche Bank
11.2020 - 11.2021

Software Developer

AMDOCS
01.2019 - 10.2020

Software Engineer

CGI
02.2016 - 12.2018

MTech - Data Science & Engineering

BITS

Bachelor of Technology - ECE

GNIT

Similar Profiles

CREATE PROFILE
Ayush Gupta