Shivangi Gupta

• Worked on AWS Lambda–based logic for Payment API validation and processing rules.

•Built an end-to-end data pipeline using Databricks and AWS following the Medallion architecture.

•Designing the data flow, ensuring schema quality, optimizing transformations, and enabling API integration for real-time data consumption

• Prepared Spec Files with request/response validation rules, schema definitions, and error scenarios.

• Conducted mapping analysis aligned with Payment Catalog & Mapping Documents.

• Used Claude and Copilot to generate mapping documentation, validation logic ideas, and test case recommendations.

• Developed pipelines using techniques and libraries like BeautifulSoup, Selenium, pandas etc. for website crawling, extracting and processing raw data from public harvesting websites.
• Processing and cleansing structured data with Pyspark and Pandas developed in Databricks.
• Extracting data from various file formats like CSV, and PDF files and transforming into a suitable form for analysis.
• Creating process/utility to validate the data based on the business logic.
• Contribution in code review, Bug Fixing, UAT and maintaining high standard code quality.
• Schedule and automate the jobs to do incremental/Historical loads using Azure Services.
• Created Pipelines in ADF to load the data from multiple sources to Targets.
• Created Mappings in Informatica end to end and load data into DB2.

• Development of code for extracting data from source system transforming data in Spark and loading it to Enterprise DataLake.
• Development of code for data transformation based on the business requirements in data bricks environment using pyspark, sql.
• Developing Jenkins job for automating the build and deployment processes (CI/CD).
• Implementing Github practices/automation for code commit.
• Identifying and developing scripts to do the automation in order to improve the.
• Optimizing the Application while doing the migration.
• Setting up Jenkins for the project for the continuous integration.
• Responsible for code deployment in different environment.
• Creating Jenkins pipeline for packaging, executing automated unit test and pushing packaged code to different environments.

• Application Migration from on prem to AWS cloud.
• Understanding the EDL(Enterprise Data Lake) Platform and ways to utilize it in projects.
• Developing python scripts to perform ETL transformation and provisioning data to enterprise Data lake.
• Understanding the existing ETL system(pentaho) and writing the pyspark transformations.
• Optimizing the Application while doing the migration.

• Understanding ETL tools like Pentaho Data Integration, implementing Transformation logics in Pentaho.
• Created ETL jobs, perform ETL runs and data load to Netezza DW.
• Maintaining Data quality with performing audits on Netezza Tables.