Hemasree Moturi

USAA HDP TO CDP MIGRATION PROJECT

• Developed and optimized Shell and Linux scripts tailored to specific operational requirements.
• Engineered robust PySpark code to replace legacy Pig script logic, enhancing data processing efficiency.
• Executed seamless migration of production data from HDP to CDP, ensuring data integrity and continuity.
• Implemented PySpark solutions for data extraction, transformation, and loading (ETL) into Hadoop ecosystems.
• Designed and managed Hive tables and HQL scripts, facilitating structured data storage and retrieval.
• Authored Python scripts for comprehensive data quality validation across tables and file systems.
• Revised and updated existing Python and Shell scripts to meet evolving business and technical needs.
• Scheduled and monitored automated job workflows using the Control-M job scheduler for optimal resource allocation.

USAA DBT DEVELOPMENT

• Engineered and implemented new dbt models and SQL scripts tailored to specific requirements.
• Developed robust dbt test code aligned with user stories and business needs.
• Optimized existing dbt model performance through strategic code enhancements.
• Managed GitLab CI/CD pipelines for seamless dbt and Python integrations.
• Designed and maintained SCD (Slowly Changing Dimension) Fact and Dimension tables, including views for SCD Type 2.
• Authored Python scripts to ensure high data quality across tables and files.

USAA Project: CDP Hadoop to Snowflake Migration & Incremental Data Processing

Project: CDP Hadoop to Snowflake Migration & Incremental Data Processing

• Migrated historical data from CDP Hadoop Hive tables to Snowflake, converting Hive data to Parquet files format, staging on DM servers, and loading using SnowSQL COPY INTO commands .
• Developed PySpark-based data validation scripts to perform checksum validation, ensuring data consistency between source and Snowflake.
• Designed and implemented incremental ETL workflows for daily data loads from Datastage, MFP, and DL2 layers into Snowflake using DBT.
• Built Type 1 & Type 2 Slowly Changing Dimensions (SCD) in DBT for handling historical data and maintaining accurate dimension tables.
• Developed reusable DBT macros for SCD logic and applied across multiple models to ensure consistency and reusability.
• Processed semi-structured data (JSON) from DL2 layer and transformed into DL3 layer using DBT models for business reporting.
• Created and optimized data models in DBT aligned with business requirements, improving transformation accuracy and maintainability.
• Designed and scheduled dbt jobs using Talend batch, integrated with Control-M for automated orchestration and efficient execution.
• Automated Hadoop file transfer to Box using Python and Shell scripts, ensuring secure and reliable movement of large datasets.

Tech Stack: Hadoop, Hive, Unix(shell Scripting)Python, PySpark, Snowflake, DBT, Control-M, SnowSQL.