Data Back Sync:
- Currently working on Database Back Sync of data from Salesforce (PRM) to DB2(AAS) as part of Migration Project
- Developed Fixed Width File Extract on top of salesforce data which will be consumed by Business
- Implemented data pipelines in Azure Data Factory to automate spark notebook executions and execute the sql scripts into the target Database
Source System to Salesforce integration:
- Worked on End-to-End Data Integration from Source System to Salesforce
- Deployed ADF Pipeline to Upsert(insert/update) source system data into salesforce
- Used Databricks Workflow to automate notebook execution instead of ADF normal process and reduced the execution time around 30%
Data Insights:
- Crafted Extracts on top of Data Lake/Source Files which will be used by the end users
- Applied SCD and CDC functionality as per business requirement using delta lake merge
- Built the pyspark code to read the data from data lake and load into Datamart