Summary
Overview
Work History
Education
Skills
Accomplishments
Timeline
Generic
Abhinav Jha

Abhinav Jha

Data Engineer
Hyderabad

Summary

Responsive expert experienced in monitoring database performance, troubleshooting issues and optimizing database environment. Possesses strong analytical skills, excellent problem-solving abilities, and deep understanding of database technologies and systems. Equally confident working independently and collaboratively as needed and utilizing excellent communication skills.

Overview

3
3
years of professional experience
4
4
years of post-secondary education

Work History

Data Engineer

Amazon
Hyderabad
09.2022 - Current
  • Created a Real Time Data Pipeline in AWS to populate the Quicksight Dashboard with a latency of less than 6 seconds. Used AWS Lambda, SQS, SNS, CloudWatch, Airflow
  • Developed a Complete End To End Configurable Data pipeline using EMR , Spark and Scala which is then used by multiple teams to plug the configs according to the use case.
  • Daily Connect with Customers and Stakeholders to gather the requirements and develop and design.

Big Data Engineer

Datametica Solutions Pvt. Ltd
Pune
01.2022 - 09.2022
  • Constructed a data pipeline to process clickstream records from kafka using spark streaming and store records in a bigquery table (GCP) used for dashboarding. Handles Late Arrival of Data. Implement Data Quality Checks as an intermediary steps. Pipeline is running sucessfully in production till now achieving low latency.
  • Design and implement a data masking framework using python and Google Cloud. This framework is capable of creating dynamic views on top of a table based on column level access and user access. Integrated Okta and Cloud Identity in order to get users access levels.
  • Migrated legacy Hadoop Map Reduce Code for data processing to a Apache Beam Framework in Java with Google Cloud Dataflow And Flink Runner along with handling data quality checks and Auditing. The Performance increase was around 42% along with the cost savings. With the in built auditing in framework, debugging any issues in pipeline really gets so simpler.

Big Data Engineer

Wissen Technology
Bengaluru
10.2021 - 01.2022
  • Integrated Data from various BCDA Api (Authentication using OAuth2.0 protocol) for a medical domain customer , Process it using Apache Beam in python and sent records to Amazon dynamoDB to be used for reporting. Used Python as language, Aws as cloud provider and Apache Beam as data processing framework.
  • Develop a framework to dynamically create dags/tasks in Apache Airflow which was then used for multiple projects in the company and reduce the manual effort by almost by 50%.

Data Engineer

Datametica Solutions Pvt. Ltd
Pune
02.2020 - 10.2021
  • Ingested streaming and transactional data across various primary data sources using Spark, Redshift, S3, and Python.
  • Automated ETL processes across billions of rows of data, which saved 45 hours of manual hours per month
  • Worked in the product team to build a data lineage tool which provides a complete end to end lineage of data showing all the upstream system and downstream system of data, which was considered as one of the best innovation in the company and used along multiple projects in the company for various purposes.

Education

Bachelor of Technology - Computer Science

Maualana Abul Kalam Azad University
Kolkata, India
08.2015 - 07.2019

Skills

    Python , Java, Scala,

undefined

Accomplishments

    Databricks Certified Apache Spark Developer (88 %)

Timeline

Data Engineer

Amazon
09.2022 - Current

Big Data Engineer

Datametica Solutions Pvt. Ltd
01.2022 - 09.2022

Big Data Engineer

Wissen Technology
10.2021 - 01.2022

Data Engineer

Datametica Solutions Pvt. Ltd
02.2020 - 10.2021

Bachelor of Technology - Computer Science

Maualana Abul Kalam Azad University
08.2015 - 07.2019

Similar Profiles

CREATE PROFILE
Abhinav JhaData Engineer