Abhishek Kumar

• Developed and Designed data pipelines using Apache Spark
  for performing Audit and Reporting of various accesses
  across group as part of Digital Identity and Access
  Management Solution
• Created generic Data Ingestion Framework using Apache
  Spark that ingests data in various formats into Data Lake
• Engineered various automation to reduce developer efforts
  in integrating new data sources into Platform
• Established process of Production Monitoring by
  implementing automated status reports, notifications, etc
• Investigated and Improved existing ETL Logic and
  Transformations

· Contributed in Generic Ingestion framework having configuration based approach using Spark, Scala and Hadoop

· Worked on the framework development to perform migration of data from legacy to salesforce system

· Contributed in day to day operation Support and defect fixing to enable smooth ingestion

· Developed Framework to check multiple utilities for refinement of data using Spark and Scala

· Developed functionalities like Duplicate Checks, Null Checks, Anomaly Detection, Critical Data Element detection, Records with Issue detection, Regex Pattern Matching etc.

· Developed Prototype for restart utility automation

· Developed functionalities to find columns with sensitive information’s of a table like SSN, Credit Card number, Account Number etc. using Regex matching.

· Developed Framework to process historical data using Apache Spark and Scala

· Developed pipeline using SQOOP and Spark-JDBC to perform File transfer to postgre DB

· Scheduling Jobs Using Autosys tool and JIL language, to run jobs and manage the dependencies on other Jobs at specified time.

· Optimization of Spark Jobs, MapReduce Jobs, Hive/SQOOP Scripts