Abhinav Jha
Data Engineer
Hyderabad
Summary
Responsive expert experienced in monitoring database performance, troubleshooting issues and optimizing database environment. Possesses strong analytical skills, excellent problem-solving abilities, and deep understanding of database technologies and systems. Equally confident working independently and collaboratively as needed and utilizing excellent communication skills.
Overview
3
3
years of professional experience
4
4
years of post-secondary education
Work History
Data Engineer
Amazon
Hyderabad
09.2022 - Current
- Created a Real Time Data Pipeline in AWS to populate the Quicksight Dashboard with a latency of less than 6 seconds. Used AWS Lambda, SQS, SNS, CloudWatch, Airflow
- Developed a Complete End To End Configurable Data pipeline using EMR , Spark and Scala which is then used by multiple teams to plug the configs according to the use case.
- Daily Connect with Customers and Stakeholders to gather the requirements and develop and design.
Big Data Engineer
Datametica Solutions Pvt. Ltd
Pune
01.2022 - 09.2022
- Constructed a data pipeline to process clickstream records from kafka using spark streaming and store records in a bigquery table (GCP) used for dashboarding. Handles Late Arrival of Data. Implement Data Quality Checks as an intermediary steps. Pipeline is running sucessfully in production till now achieving low latency.
- Design and implement a data masking framework using python and Google Cloud. This framework is capable of creating dynamic views on top of a table based on column level access and user access. Integrated Okta and Cloud Identity in order to get users access levels.
- Migrated legacy Hadoop Map Reduce Code for data processing to a Apache Beam Framework in Java with Google Cloud Dataflow And Flink Runner along with handling data quality checks and Auditing. The Performance increase was around 42% along with the cost savings. With the in built auditing in framework, debugging any issues in pipeline really gets so simpler.
Big Data Engineer
Wissen Technology
Bengaluru
10.2021 - 01.2022
- Integrated Data from various BCDA Api (Authentication using OAuth2.0 protocol) for a medical domain customer , Process it using Apache Beam in python and sent records to Amazon dynamoDB to be used for reporting. Used Python as language, Aws as cloud provider and Apache Beam as data processing framework.
- Develop a framework to dynamically create dags/tasks in Apache Airflow which was then used for multiple projects in the company and reduce the manual effort by almost by 50%.
Data Engineer
Datametica Solutions Pvt. Ltd
Pune
02.2020 - 10.2021
- Ingested streaming and transactional data across various primary data sources using Spark, Redshift, S3, and Python.
- Automated ETL processes across billions of rows of data, which saved 45 hours of manual hours per month
- Worked in the product team to build a data lineage tool which provides a complete end to end lineage of data showing all the upstream system and downstream system of data, which was considered as one of the best innovation in the company and used along multiple projects in the company for various purposes.
Education
Bachelor of Technology - Computer Science
Maualana Abul Kalam Azad University
Kolkata, India 08.2015 - 07.2019
Skills
Python , Java, Scala,
Apache Beam, Apche Spark
Apache Kafka
Apache Airflow
Google Cloud (Bigquery, Cloud Functions, Dataflow, Pubsub, DataProc, Netwroking, Composer)
Aws (Kinesis, EC2, S3, DynamoDB, Rds,Lambda)
Sql (Mysql) and NoSql (Hbase and Cassandra)
Hadoop Ecosystem (Sqoop, Hive, Map Reduce)
Accomplishments
Databricks Certified Apache Spark Developer (88 %)
Timeline
Data Engineer
Amazon
09.2022 - Current
Big Data Engineer
Datametica Solutions Pvt. Ltd
01.2022 - 09.2022
Big Data Engineer
Wissen Technology
10.2021 - 01.2022
Data Engineer
Datametica Solutions Pvt. Ltd
02.2020 - 10.2021
Bachelor of Technology - Computer Science
Maualana Abul Kalam Azad University
08.2015 - 07.2019