Durga Manoja

FINTELLA MIGRATION:

Led end-to-end upgrade of a mission-critical APTransformation pipeline from EMR 6.7 → 7.3 and Hudi 0.11 → 0.15, driving platform modernization under a strict timeline aligned with upstream VP-level goals resolving cross-version compatibility issues, aligning DAG lineage in Airflow, and ensuring zero downstream disruption.

Responsibilities:

• Solved major compatibility challenges where EMR 7.3 could not read legacy Hudi 0.6 data produced by upstream, by implementing custom migration and backward-compatibility strategies.
• Designed alpha ecosystem mirroring production to identify issues early and implemented incremental-based migration strategy phase by phase to avoid risks associated with Hoodie commit times in production.
• Strategized and coordinated cross-team migration efforts to align downstream systems (on EMR 6.7 & Hudi v4) with the new EMR 7.3 & Hudi v6 platform; collaborated with downstream owners to ensure their upgrade preceded ours to maintain data compatibility and pipeline stability.

CONFLUENCE:

Designed and Developed the integration of VPS (Vendor Payment System) with Confluence Data Lake, architecting a solution to optimize data ingestion for a 4-petabyte enterprise data lake serving 530+ AP/AR datasets. Designed and implemented a centralized streaming architecture utilizing AWS SNS, Kinesis Firehose, and DMS, replacing individual database connections with a scalable event-driven system.

Responsibilities:

• Implementing table-level data filtering, creating reconciliation mechanisms, managing full data loads and CDC processes, and establishing data quality controls.
• Developed CDK infrastructure for automated deployment of Firehose pipelines and SNS subscriptions, while implementing robust error handling and monitoring solutions.
• The project significantly reduced database overhead, improved data reliability, and streamlined the data ingestion process across multiple consumer systems including Veritas and Search Data Store.

JIGSAW:

Jigsaw is designed to decrease the number of disputes, increase the accuracy of vendor payments, and assist vendors in understanding what was and was not paid and why. By creating additional ways to reimburse vendors for overages and, more crucially, giving them visibility into Amazon's payment-related decisions and the reasons behind the treatment they received, it will eventually decrease contacts for short-pays.

Responsibilities:

• Analyzed requirements and conducted an in-depth study of a new stream to be created.
• Supported code/design analysis strategy development and project planning.
• Collaborated with stakeholders to develop various defect detection rules using complex SQL.
• Created Airflow orchestration with federated authentication to automate and productionize data pipelines.
• Created Airflow workflows connecting to various AWS services.
• Improved business logic to achieve optimized results.
• Participated in peer reviews, provided technical guidance to team members, and mentored juniors.
• Maintained and supported production flows.

NEXT:

Selection Guidance –> New Products: Goal of this project is to find business opportunities and help sellers to expand their portfolio for new products in same marketplaces.

Selection Guidance –> Cross listing: Goal of this project is to recommend sellers to sell his fast-moving goods across other marketplaces.

Responsibilities:

• Fetch the source data from various source systems accessing through Lake Formation.
• Design and develop the pipeline with compute engine as Spark with the data lake in Iceberg.
• Collaborate with stakeholders and UI team to deliver accurate data in the requested format.
• Maintain and support the production flows.

DEBT MANAGER (DM9):

Debt Manager is a FICO product which through its recovery tactics and strategies allows BOA associates to efficiently recover charge off and delinquent Loans. The main objective of this project is to receive account information which is in the state of recovery from DM9, process the data and push them into tables for End Users.

Responsibilities:

• Development and testing of an ELT process using HDFS, PySpark, Hive, and Teradata.
• Writing logical test cases for the above-built modules for unit testing.
• Executing test cases against the Test/QA/Production environment and verifying code for defects while providing solutions.
• Automated the tedious task of test case execution and evaluation across various platforms, thereby reducing manual intervention and saving efforts and costs for the company.
• Developed a machine learning solution on loans to identify future defaults from loan data.
• Harnessed operational excellence and continuous improvement with a can-do attitude.

Highmark is a health insurance company and we are involved in storing its claim and member data in data warehouse. We use Extract-Transform-Load (ETL) process where we receive data from source, transform it as required and finally load it into target using Mainframes and Teradata.

Responsibilities:

• Design and develop ETL processes using Mainframe and Teradata.
• Participate in Agile daily Scrum calls and interact directly with clients to understand requirements.
• Code, perform unit testing, and involve in application maintenance and issue resolution.
• Perform code reviews.