AWS stack, Spark, SCALA, Python, HiveQL, Hadoop, Linux, UNIX and Windows
Project 1: Capital Regulatory
Duration: July 2021 to till date
Team Size: 15 Members
Description:
The purpose of this project is to calculate risk parameters and behavior of the contracts to improve the consistency between the margin projections performed by the financial management and ones used for capital planning exercises
Role: Senior Software Engineer
Solution Environment: Cloudera, AWS
Tools: Scala, Hadoop, Spark, Spark SQL, Hadoop, Hive, Impala, Jenkins, Control-M, S3, EMR, Step Functions, Athena, Glue, Terraform.
Responsibilities:
• Requirement analysis of the problem identified.
• Working on production critical incidents to find the source of the code’s issue and swiftly fix it.
• Writing Spark using SCALA code as per the requirements.
• Creating Unit test cases to ensure that all requirements are met and functioning as intended.
• Creation of deployment pipelines for deployment sing Jenkins and GitHub.
• Finding the problematic code by utilizing Sonar Qube to evaluate code quality and coverage.
• Creation of Job and Scheduling and managing the dependency using Control-M.
• Performance Optimization and fine tuning of long running jobs in Apache Spark.
Worked together with the client to comprehend the requirements and record the project in Confluence.
Project 2: Recvue Billing System
Duration: Jan 2020 to July 2021
Team Size: 5 Members
Description:
Recvue Billing System is a Product Calculating Recurring Revenue for various Organization Using Big data (Spark) for Business Logic Implementation and high speed Performance
Role: Data Engineer
Solution Environment: Linux , Windows
Tools: Scala, Hadoop, Spark, Spark SQL, Databricks, AWS
Responsibilities:
• Performance tuning of spark jobs processing millions of data, bulk-testing and sending the Statistics to the client
• Owned code development and enhancement of Compensation Management, Delivery Imports and Price Deliveries Module
• Performance Testing and fine tuning of code in Databricks Cluster
• Administered Sprint planning with Stakeholders in Onsite
• Analyzed work as per ticket assigned in current sprint based on work-priority
Project 3 : Raptor
Duration: Jun 2019 to Dec 2019
Team Size: 5 Members
Description:
Reynolds American (RAI) is on a journey to build out a next generation analytics platform. As part of this journey RAI is looking to replace an existing on-premise data warehouse with a cloud-based data warehousing system. RAI has selected the Amazon Web Services Platform (AWS) to house this data warehouse. RAI plans to leverage the AWS Big Data technology stack (Postgres, Spring Boot, S3, EMR, etc.) to build out this analytics platform.
Role: Data Engineer
Solution Environment: Linux
Tools: Scala, Hadoop, Spark, Spark SQL, Oracle Bigdata Cloud, Databricks
Responsibilities:
• Building Data Pipeline
• Write Data models in Spring Boot
• Convert traditional system transformations into transformations.
• Optimizing Spark jobs.
• Process Automation using Shell Scripting.