Nagula Moksha

UDHOpsCore Support Project

Description of Project:

• Efficiently managed and troubleshooted issues within the AWS infrastructure, utilizing the abilities of the AWS Console to ensure optimal operation.
• Managed system performance and availability through the use of AWS CloudWatch and various monitoring tools.
• Handled user requests and addressed infrastructure issues through resolution of Jira and Service Now tickets.
• Monitor and respond to operational emails to ensure timely resolution of incidents.
• Ensured optimal performance of Airflow DAGs through diligent monitoring and troubleshooting using essential tools like Hubble, Gather, DBeaver, Putty, Load Server, and ABI Server.
• Managed the development and maintenance of operational documentation and runbooks for incident response and system maintenance.

UAInc-Apps-UDH-DR-Mobilize

Description of Project:

• Copied pre-requisites of airline data from primary AWS region to secondary region for services including S3, RDS, Glue, Redshift, Airflow, Gathr, and ICON.
• Safeguard data integrity by implementing a backup strategy for disaster recovery.
• Utilize cloud infrastructure to store and process subsequent sets of data in the secondary region.

Roles and Responsibilities:

• Validated and replaced hardcoded values in glue scripts, secret_manager, yml, and Bteq files for primary and secondary regions. Ensured DR compliance by creating airflow variables instead of hardcoded values. Verified data in secondary Redshift.

UA-RTF-UCLOUD

Description of Project:

• The aim of this project is to migrate United Airlines' data from Teradata to AWS Cloud, with the primary objective of making AWS the primary data storage for United Airlines.
• The process involves extracting data in batches and processing it by creating a Direct Acyclic Graph (DAG) in the Managed Apache Airflow Service. Multiple tasks are associated with the Dag, and each task has its own functionality to process data and validate task instances through glue jobs.
• The data which is extracted from source is loaded into temporary tables though ICON(Ingest, Catalog, Optimize and Notify) process which acts as a gatekeeper for incoming data into data lake and loading to target tables in redshift or RDS Database based on the details mentioned in secret managers.

Roles and Responsibilities :

• Designed tables in RDBMS (e.g., PostgreSQL) such as RDS and processed data, including importing and exporting databases to AWS Redshift from Aurora and Athena tables
• Creating and uploading the YAML files related to a particular feed in AWS S3 and validating the DAG in Managed Apache Airflow Service.
• Handled large datasets of various data types with several kinds of interval columns during the data processing.
• Developed batch loads using PLSql within the Redshift framework and integrated data from various sources (JDBC, RDBMS, text files, DAT, CSV)
• Utilized Putty for data extraction and configured Utility for Test Harness to test source and destination data with several checks including alpha numeric and sum check etc
• Worked with AWS Glue Jobs to monitor Airflow DAG progress and troubleshoot errors
• Designed and implemented an ingestion template to generate SQL and HQL files for loading preliminary ingestion into an Athena table, and then transferring it to Redshift for comparison.
• Utilized S3 and several buckets designed to hold files from incremental to backup to post-consumption (RAW bucket) files
• Analyzed data types compatible with Teradata and addressed AWS Redshift's limitations on supported data types
• Imported data from Teradata into Athena via SFTP configurations designed in DAG files and loaded it into AWS Athena with compatible data types for Redshift/RDS
• Imported data from Athena to Redshift/RDS using Airflow and Glue job configurations set up in DAG files
• Analyzed BTEQ scripts and devised solutions for implementing batch load functions
• Configured Gathr Pipeline to import data from Teradata to RDS tables, including data type conversion and matching with PySpark scripting
• Automated data workflows and scripts using Amazon Managed Airflow to ensure scheduled execution in production
• Understanding in logs monitoring within Airflow and Glue Jobs, utilizing this knowledge for effective debugging and troubleshooting
• Created and updated the confluence page with complete process and validation of loads
• Promoted the loads to higher environments by committing in GIT.