Summary
Overview
Work History
Education
Skills
Certifications Training
Certification
Timeline
Generic

Sanjay Nadarajan

Coimbatore

Summary

Adept Senior Data Engineer with a proven track record at Wavicle Data Solutions, showcasing expertise in PySpark, Python, and AWS. Excelled in data migration and analytics, significantly enhancing efficiency and cost savings. Demonstrates strong problem-solving skills and a commitment to innovation, driving successful new product launches and analytical solutions.

Overview

6
6
years of professional experience
1
1
Certification

Work History

Senior Data Engineer

Wavicle Data Solutions
07.2022 - Current
  • Company Overview: https://carscommerce.com/
  • Role: Senior Data Engineer
  • Domain: Cars
  • Project: Pyspark Forge conversion, API call, Third Party Development, GA4 Data pull from Google Cloud
  • Worked on job creation for GA$ data, which is used for analytical purposes, generated analytics, and sent to the BI team for further processing.
  • Timeline: Conversion of spark EMR jobs to forge ec2 spot instance for memory tuning, cost savings and time efficiency
  • Several issues were fixed on converting these jobs on spark version issue, logic transformation complexity
  • These all fixed in minimal time
  • Development of Api jobs and development on new product launches
  • Environment: AWS, Pyspark, EC2
  • Https://wavicledata.com/

Data Engineer

Wavicle Data Solutions
04.2019 - 06.2022
  • Company Overview: https://wavicledata.com/
  • Role: Data Engineer
  • Project: Data Migration - Talend On-prem to Cloud, Talend to Pyspark conversion
  • Domain: Cars
  • Timeline: Conversion of datastage to talend jobs in cloud using talend big data tool
  • Using Spark environment will execute the jobs and also TMC for console purposes
  • Hive/Redshift will be used for DDL query creation & optimization
  • Created DDLs and views on RedShift for MasterData tables
  • Worked on data registration and data ingestion using API Gateway
  • Experience working with AWS Cloud services like EC2, Amazon S3, AWS IAM, AWS RDS, EMR, Glue, Athena Redshift, Amazon API Gateway
  • Worked on UC4 and Automic jobs to schedule backfill and repartitioned jobs
  • Analyzed datasets using Python/SQL, and Jupiter Notebook
  • Created Talend jobs for one-time load and enriched to move to S3 bucket and published it to TMC
  • Conversion of various talend to PySpark jobs for faster performance
  • Environment: Talend, Spark, EMR
  • Https://wavicledata.com/

Jr. Data Integration Developer

Wavicle Data Solutions
11.2018 - 03.2019
  • Company Overview: https://wavicledata.com/
  • Role: Jr
  • Data Integration Developer
  • Project: Data Migration— C360
  • Domain: C360
  • Timeline: Worked on ETL job creation using Talend based on requirement/mapping document
  • Developed ETL jobs in Talend to cleanse, process, and load data that was similar to provided Talend jobs
  • Managed SQL scripts to execute shell commands for the provided connections and database tables
  • Worked on importing data from MSSQL to HDFS to Redshift with ETL jobs
  • Performed Data Analysis using Amazon RedShift, MSSQL and have good knowledge in HIVE querying
  • Handled migrations to the Production environment and maintained a track of execution times and enhancements required
  • Involved in the implementation testing process and manage Offshore in assisting development and testing on a daily basis
  • Interacted with clients and support teams to keep track of job scheduling and data loads
  • Environment: Talend
  • Https://wavicledata.com/

Education

Master of Business Administration - Information Systems

Bharathiyar University
Coimbatore, Tamil Nadu
01.2021

Bachelor of Technology - Information Technology

Sri Ramakrishna Engineering College
Coimbatore, Tamil Nadu
04.2018

Skills

  • PySpark
  • Python
  • HDFS
  • MR
  • Hive
  • AWS
  • EC2
  • EMR
  • Hue
  • Athena
  • Redshift
  • Teradata
  • Bigquery
  • UC4
  • Atomic
  • Talend
  • Hadoop
  • JIRA
  • Jenkins
  • Snowflake Basics
  • Databricks Basics
  • Airflow

Certifications Training

  • Databricks Certified Associate Developer for Apache Spark 3
  • Talend Data Integration
  • Talend Big Data Basics & Advanced

Certification

  • Databricks Spark Certified Associate
  • Talend Data Integration Certified

Timeline

Senior Data Engineer

Wavicle Data Solutions
07.2022 - Current

Data Engineer

Wavicle Data Solutions
04.2019 - 06.2022

Jr. Data Integration Developer

Wavicle Data Solutions
11.2018 - 03.2019

Master of Business Administration - Information Systems

Bharathiyar University

Bachelor of Technology - Information Technology

Sri Ramakrishna Engineering College

Similar Profiles

CREATE PROFILE
Sanjay Nadarajan