RUPESH TURKAR

• Project 1: Data Pipeline Management, Client: Circuittrack, USA
• Technologies used: Spark-SQL, Spark-Core, PySpark, HDFS

Roles & Responsibilities:

• Understand the requirements and clarify the doubts with Business Analyst
• Successfully completed POC on pipeline
• Responsible to merge data coming from different sources and loaded into HDFS
• Supported code/design analysis, strategy development and project
• Used Spark-SQL to process the data and to run on Spark engine
• Combined data from MySQL and file sources & applied various transformations
• Created the PySpark jobs
• Finally stored table in HDFS in partitioned format.

• Project 2: HBO MTIS Operations, Client: Warner Media
• Technology used: AWS, Looker, Snowflake, Apache Airflow, ServiceNow, Jenkins, GitHub

Roles & Responsibilities:

• Worked on managing access for various applications such as Snowflake, Tableau, Looker, Jenkins, Airflow, AWS, Datadog and GitHub
• Worked on AWS Identity & Management (IAM) such as creating, deploying IAM roles, users and policies using Terraform application and periodically rotated AWS CLI key of IAM users
• Provided exceptional support to data engineering and data scientist team by creating the buckets, storing/copying the files and changing the lifecycle policies using AWS S3 through AWS EC2
• Airflow - Closely Monitored the data level operations using Agile methodologies to provide continuous support to attain maximum success rate of Directed Acyclic Graphs (DAGs).
• Terraform - Created IAM users, EC2 instances, ECS clusters and S3 buckets using Terraform and provided extended support in securing key level access to avoid vulnerabilities
• Datadog - Worked on creating monitors and synthesis URL alerts for various applications using Datadog agent and for the AWS services.
• Jenkins – Creating Jenkins pipelines for automating the deployments and generating the user id with pipeline trigger.
• Snowflake - Closely working with Software development teams to create, enable, modify the warehouses, shares and views in Snowflake application. Supported to minimize the compute costs by monitoring queries and resizing warehouses.
• Looker - Closely worked with Business Analytics team by providing role permissions, creating connections, and migrating dashboards for different vendors to Looker application.
• Git - Knowledge of Source Code Management (Version Control System) tool like Git.
• Service now - Having an exposure towards ITIL process and familiar about handling of Incident management process.