Summary
Overview
Work History
Education
Skills
Languages
Personal Information
Profile - Professional Summary
Certification
Timeline
Generic
Sudershan Vaidya

Sudershan Vaidya

Bangalore

Summary

Dynamic Senior Technical Lead with expertise in AWS Glue and Spark, successfully enhanced user cashback experiences at HCL Technology Ltd. Proven ability to build efficient data pipelines and automate processes, while fostering collaboration across teams. Adept at problem-solving and delivering impactful solutions in fast-paced environments.

Overview

7
7
years of professional experience
1
1
Certification

Work History

Senior Technical Lead

HCL Technology Ltd
Bangalore
07.2024 - Current

Company Overview: Common Bank of Australia.

Australia's largest bank offers a wide range of financial services to retail, business, and institutional customers, with a presence in New Zealand, Asia, the United States, and the United Kingdom.

  • Worked on building a pipeline using Spark and Hive to enhance the user cashback experience.
  • Performed A/B testing using a mobile application and encouraged users to use the beta application to fine-tune the performance of the queries when they go live to handle a large number of users.
  • Helped in migrating a few pipelines from RDBMS to a big data platform using Spark.

Associate Consultant

ITI Data
Chennai
03.2022 - 07.2024

Company Overview: Fitch Ratings, Inc.

American credit rating agency and is one of the 'Big Three' credit rating agencies. Its rigorous analysis and deep expertise have resulted in a variety of market-leading tools, methodologies, indices, research, and analytical products, which have helped investors manage risk and fund over a century of growth.

  • Built an efficient data pipeline with the help of AWS Lambda, Step Functions, and EventBridge Rule.
  • Integrated REST API with AWS Lambda.
  • Published the event to the Event bus using Boto3.
  • Performed CDC using PySpark in AWS Glue.
  • Implemented email notifications using SNS, CloudWatch Rule, and Lambda.

Sr. Data Engineer

Emids
11.2020 - 03.2022

Company Overview: Neighborhood Health Plan of Rhode Island (Neighborhood).

A Health Maintenance Organization (HMO) insurance company with a consistent record of earning accreditation by the National Committee for Quality Assurance (NCQA).

  • Interoperability is the ability of electronic health records (EHRs) and other health care data management systems to communicate with one another and to exchange information seamlessly.
  • Created SQL queries to extract data from NHPRI databases (Oracle).
  • Created a script to automate extracted data to AWS S3.
  • Transformed data as per FHIR standard using PySpark in AWS Glue.
  • Implemented email notifications using SNS, CloudWatch Rule, and Lambda.
  • Neighborhood Health Plan of Rhode Island (Neighborhood) is a health maintenance organization (HMO) insurance company with a consistent record of earning accreditation by the National Committee for Quality Assurance (NCQA).
  • Interoperability is the ability of electronic health records (EHRs) and other health care data management systems to communicate with one another and to exchange information seamlessly.

Associate Projects

Cognizant
08.2019 - 08.2020

Company Overview: Huawei.

Leading global provider of information and communications technology (ICT) infrastructure, and smart devices.

  • HUAWEI Video is the default video player in HUAWEI/HONOR smartphones and tablet devices.
  • It also supports online video streaming services in Spain and Italy now.
  • Understanding the requirement.
  • Implementing user stories using Spark with Scala.
  • Performed ETL in Hadoop using Hive for preparing input tables for the requirement.
  • Used analytical, windowing, and aggregate functions to meet requirements.
  • I worked on joins and the performance tuning of Hive scripts.
  • Completing the assigned task in the given timeframe.
  • Developing a data pipeline using Hive and Spark.
  • Performing unit testing.
  • I worked on resolving production issues.
  • Worked on video recommendation based on a few machine learning algorithms (ALS, LSI).
  • HUAWEI is a leading global provider of information and communications technology (ICT) infrastructure and smart devices.
  • HUAWEI Video is the default video player in HUAWEI/HONOR smartphones and tablet devices.
  • It also supports online video streaming services in Spain and Italy now.

Associate Consultant

Capgemini
04.2018 - 07.2019

Company Overview: MediaMarktSaturn Retail Group is Europe's leading consumer electronics retailer and provider of related services.

Europe's leading consumer electronics retailer and provider of related services.

  • Its integrated online and fixed-location offerings reach millions of customers every day.
  • Coordinating with the onsite and offshore teams and understanding the client requirements.
  • Implementing the user stories using Spark with Scala.
  • Implemented Spark using Scala and Spark SQL for faster processing of data.
  • Working on assigned tasks with self-learning within the time frame.
  • MediaMarktSaturn Retail Group is Europe's leading consumer electronics retailer and provider of related services.
  • Its integrated online and fixed-location offerings reach millions of customers every day.

Education

Bachelor of Engineering (B.E) - ECE

VTU
Bidar
05.2013

Skills

  • AWS Glue
  • AWS Lambda
  • AWS S3
  • AWS SNS
  • AWS Step Functions
  • Hadoop
  • Hive
  • Oozie
  • Spark
  • YARN
  • Boto3
  • CloudWatch
  • ETL
  • FHIR
  • HDFS
  • Core Java
  • Linux
  • MapReduce
  • MySQL
  • Oracle
  • Python
  • Scala
  • Sqoop

Languages

English

Personal Information

Date of Birth: 02/01/92

Profile - Professional Summary

Having cumulative experience of 9 Years 4 Months in the areas of Oracle database, Hadoop and relevant Big-Data technologies along with AWS Services. Worked on various AWS services like AWS S3, GLUE, Lambda, SNS, Stepfunction and CloudWatch. Hands on Experience on working with Oracle database. Good knowledge of Hadoop and Spark architecture and various components such as HDFS, Map Reduce, Spark Core and Spark SQL. Hands on experience in working with Hadoop and Spark ecosystems tools like: Hive, Sqoop, Oozie, Spark Core and Spark SQL. Hands on experience in creating Hive tables and using Partition and bucketing techniques to improve hive query performance. Experience using Analytical functions and windowing functions in Hive. Experience in using joins and good understanding of aggregate functions. Experienced on working with Internal and External tables. Hands on experience in using Sqoop to import data from relational databases such as MySql to Hadoop file system. Strong Knowledge on Spark RDD, DataFrames and DataSets and their operations. Good knowledge on architecture and working of YARN, HDFS. Worked on machine learning algorithms ALS and LSI. Used different file formats to improve performance in Hive and Spark SQL.

Certification

  • AWS Certified Data Engineer - Associate

Timeline

Senior Technical Lead

HCL Technology Ltd
07.2024 - Current

Associate Consultant

ITI Data
03.2022 - 07.2024

Sr. Data Engineer

Emids
11.2020 - 03.2022

Associate Projects

Cognizant
08.2019 - 08.2020

Associate Consultant

Capgemini
04.2018 - 07.2019

Bachelor of Engineering (B.E) - ECE

VTU
Sudershan Vaidya