ABHIJEET NAIB

Project: Deltalake Housekeeper

Develop a spark based SAAS application and java client which can integrate with any pipeline , optimize the run time of the spark job and perform delta table maintenance activities as well

Roles and Responsibilities

• 1. Responsible for end to end design development integration and management of the project
• 2. Responsible for optimization of data pipelines running across the team.
• Achievements:
• 1. Implemented smart checks to run optimize only when necessary and reduce run times of jobs
• 2. Implemented mechanism to calculate ideal file size in a table helping file pruning and improving delta log dependency
• 3. Run time of non partitioned tables reduces by 50% and resource usage reduced by 40%.
• 4. Periodic vacuum fetching tables via central catalog helped to reduce storage cost significantly

Project: Schema Manager

Spark and kubernetes based multi-tenant application built to create version controlled, contract obligating and maintenance friendly tables across delta lake , postgres , snowflake.

Roles and Responsibilities:

• 1. Responsible for design, development, availability and resiliency of the application.
• 1. Champion Design and Coding best practices while technically leading teams.
• 2. Work with product managers in developing strategy and road map to provide compelling capabilities that helps them succeed in their business goals.

Achievements:

• 1. Schema manager helped multiple clients create 5000+ tables with less than 1% failure rate.
• 2. Designed common schema declaration language which took away effort of storing schema as per storage engine.
• 3. Single source of truth for schema across deltalake, snowflake, postgres.
• 4. Successfully delivered , project as delivery lead.
• 5. As the tables were registered on central catalog the maintenance cost was reduced significantly.

Project: Egress Share

• This Service provided self serviceable data share solution to clients. Data stored on postgres/deltalake tables was shared as required with the clients on click of a button.
• Achievements:
• 1. Any client was able to integrate their snowflake account with company systems via the application in few minutes.
• 2. A snowflake tables was automatically created parallel to delta or postgres table to reduce any engineering effort.

Project: Low Code Platform

• Spark based interactive ETL framework designed to reduce coding effort in building data pipelines.
• Technology: Pyspark, kubernetes, kotlin, Postgres, snowflake, Sparksql, Deltalake

Roles and Responsibilities:

• 1. Part of the design team
• 2. Responsible for designing custom/sql code block
• 3. Develop modules which are processing framework agnostic and can run with pandas or Pyspark with 0 code change.

Achievements:

• 1. Was able to deliver framework agnostic code in tight deadlines.
• 2. Delivered backend spark code which was called via API's to perform ETL

Project : Nike Communications platform
Platform to campain users on nike owned apps.

Roles and Responsibilities:
1) Built scalable, maintenance-friendly data solutions to provide reliable and consistent data to stakeholders.
2) Build and manage workflows of batch and streaming fashion, consuming and writing to various sources such as kafka , s3,postgres etc.
3) Analyze customer needs and make them into requirements.

Project: Nike Data Quality Framework
Designed and Developed configurable data quality solution working seamlessly with airflow dags without any changes to existing codes, dags. Solution provided functionalities like Alerting, EOD reports, Slack interactives etc.

Project: SLA Monitoring Framework
A framework built on top of airflow metadata to monitor pipelines and raise alarms as necessary. The product is built by using AWS lambda, Postgres and airflow dag. Seamlessly integrates with slack interactives to provide reports on demand.

Achievements:
1) Implemented a streaming framework based on spark and improved developer productivity significantly.
2) Designed process for sanity, monitoring, and consistency of the streaming and batch jobs.
3) DQ solution was adopted for 100+ tables and 60 + dags in the vertical.

Project: Albertsons

Roles and Responsibilities

• Lead a team of developers to design and implement analytical and technical solutions
• Work with cross-functional teams and establish clear communication for
• Digital transformation and cloud migration of Data Warehouse

Achievements:

• Successfully tested multiple architectures Poc's with Databricks Delta and snowflake
• Implemented CDC methods for cloud and digital transformation on snowflake Automated snowflake procedure creation through Jinja Templates.

NB Churn/Customer Attrition:

A product aimed to build an attrition model with and support marketing teams to retain customers on and monthly to quarterly basis.

Habitual AI:
The flagship product of the company aims to improve customer engagement via prescriptive analytics and make them engaged to client services across domains. Works as a single point for all product recommendations.

Technology used: HortonWorks Ambari, Hadoop, Hive, Stream sets,
Spark/Scala, PostgreSQL.

Responsibilities:
1) Contribute to product development by creating data pipelines and implementing models with Spark, Scala.
2) Work closely with internal and external key stakeholders regarding the performance and evaluation of deliveries.

3) Identify correct features to improve model metrics

4) Design pipelines and architecture for automated model training, tuning and data preparation for monthly activities.

5) Evaluate and improve model performance by identifying and adding new features based on the domain knowledge of the banking industry and Exploratory data analysis.

Achievements:

1) Implemented code to generate 400+ features dynamically for the attrition model thereby improving the model accuracy significantly in spark/scala.

2) Managed billions of data points on confined hardware by tuning and improving the code design and following spark tuning guidelines to provide timely delivery to clients.

3) Achieved a recall of 87+ % and a model accuracy of above 90% for banking clients.

4) Designed pipelines across two banks to ensure smooth functioning of the monthly recommendation process.

Project: Barclays Compliance Application Support

• Worked at client location for Barclays Compliance application support project
• The role was to support, automate, troubleshoot, and fix the production data pipelines build from Data Warehouse to 55+ compliance applications
• Provide solutions on the go to meet defined SLAs
• Coordinate with various teams to support the production environment and deploy fixes
• Achieved Cognizant Hall of Fame award to reduce project expenditure through automatons.