Aboli Patil

EDLI :

• This project designed for stream data processing based on cloud, data loss detection, reconciliation of records, normalization of data
• Technologies used: Pub-sub, BigQuery, GCS, DataProc, StackDriver, Spark Streaming,Spark Batch Jobs, Java
• Developed, implemented, supported and maintained data engineering frameworks, standards and documentation.

Gopay

• This project is designed for moving/copying data from Oracle to Hive and creating snapshot on hive table.
• Responsible for developing and leading framework for hive snapshot in whole pipeline, maintaining audit tables and logs tables for each spark job run.
• Technologies used :Spark batch job, DALM(Airflow scheduler), Hive , Shell script

iHUB

• This system is designed for real time analysis of data, identifying data loss , reconciliation of records, It involves transactional data processing. This involved replaced of HBase with hive, Mentoring juniors and involved in production live environment issues
• Technologies used : Spark Structured Streaming, Apache Kafka, Spark Batch jobs, Shell script, Java

Salesforce

• This project is designed for copying data from salesforce objects to hive tables and creating snapshots on top of hive tables
• Technologies Used : Salesforce API, Python 3.0 , Hive, Spark Batch Job

iHUB

• This system is designed for real time analysis of data, identifying data loss , reconcilliation of records, It involves transactional data processing.
• Technologies used : Spark Structured Streaming, Apache Kafka , Spark batch jobs, Hive,HDFS, HBase, Shell script, Java
• Analyzed complex data and identified anomalies, trends and risks to provide useful insights to improve internal controls.

• CME liquidity app
  This project designed for analysis of trading data which help in analyzing history trades and their effects.
  Prepared Jmeter scripts for API testing and measure response time. Involved in optimizations of partitions to
  improve the performance of queries in spark. Also used eclipse MAT for heap dump analysis.
• Technologies Used : Java, Spark batch, JMeter