SHANTANU WADHANKAR

Project :

SAS to Databricks Migration (Finance)

• Designed and implemented an efficient ETL process to migrate financial data from Oracle to Azure Data Lake Storage Gen2 (ADLS gen-2) using Databricks and Azure Data Factory.
• Translated SAS jobs into PySpark to replicate existing functionality and enhance performance using optimization techniques.
• Developed ETL pipelines using Databricks and Azure Data Factory to extract data from Oracle databases to ADLS gen-2.
• Translated SAS job logic into PySpark to maintain and improve data processing workflows.
• Implemented optimization techniques to enhance job performance and efficiency.
• Utilized spark.sql and SQL to convert SAS proc sql statements to PySpark equivalents for seamless migration and improved performance.
• 
  

Combatting Fraud in Healthcare Insurance:

• Addressed the pervasive issue of fraudulent claims in healthcare insurance, which accounts for a significant portion of payments worldwide, posing a multi-billion-dollar challenge. Implemented solutions to enhance fraud detection and prevention.
• Roles & Responsibilities:
• Migrated data from RDBMS sources to AWS and Snowflake using PySpark.
• Developed PySpark jobs for data extraction into S3 and transformation/ingestion into Snowflake.
• Optimized SQL scripts using Spark SQL for improved performance.
• Created SQL scripts for CDC data on transactional tables.
• Wrote Snowflake DDL scripts and queries for data analysis.
• Conducted unit testing for the developed tables.
• Implemented SCD type 1 & 2 logic and joins in PySpark scripts.