Sanjay Bhakta K

Project: UDS/Hadoop Exit
Role: Data Engineer
Description:

• Designed and implemented 20 end-to-end data pipelines for migrating data from Hadoop to AWS ecosystem and Teradata warehouse.
• Automated file ingestion from Dropbox into AWS Glue tables using Spark and Glue ETL workflows.
• Used Java and Spark to develop data transformation and validation logic ensuring data consistency between Hadoop and AWS sources.
• Leveraged Ctrl-M for job scheduling and Jenkins for CI/CD to automate deployment and pipeline orchestration.
• Loaded and provisioned cleansed data into Teradata, then performed detailed data comparison to validate successful migration.
• Generated comparison reports to confirm data integrity and completeness post-migration.
• Collaborated closely with data scientists and stakeholders to deliver reliable, accurate datasets for analytics.

Technologies: AWS Glue, Hadoop, Teradata, Spark, Java, Ctrl-M, Jenkins, Dropbox

Project: MBODS Exit
Role: Data Engineer
Description:

• Built and managed data pipelines for MBODS exit, ingesting data via MFTS routes into Amazon S3 buckets.
• Triggered CDC (Change Data Capture) logic to generate full and delta files, followed by LRF (Load Ready File) creation for downstream use.
• Used Spark to write and execute SQL queries based on business logic for data transformation.
• Utilized a Java-based validation utility to verify record counts, business date accuracy, and sum field checks.
• Loaded validated and processed data into Cassandra for downstream systems.
• Maintained and monitored ~70 Ctrl-M jobs, ensuring reliable and timely execution of all workflows.
• Collaborated with platform, QA, and support teams to resolve job failures, optimize performance, and uphold SLAs.

Technologies: Spark (SQL), AWS S3, MFTS, Java (utility usage), Cassandra, Ctrl-M, Linux, Shell Scripting