Divya Sahu

Client: Retail Industry.

• Built a data validation framework in Databricks using BigQuery as the source, ensuring accuracy during data ingestion and transformation.
• Developed a PII masking framework based on existing Unity Catalog tables, applying role-based access to protect sensitive data.
• Migrated datasets from BigQuery to Databricks, aligning with the client’s cloud modernization goals.
• Handled the deployment of pipelines and frameworks using GitHub and Databricks Asset Bundle.
• Used Databricks Workflows for orchestration, and integrated with Control-M for production-grade job scheduling and monitoring.
• Refactored legacy pipelines to improve performance, reliability, and ease of maintenance.
• Created detailed technical documentation to support development, deployment, and client handover.
• Collaborated closely with the client team for requirement gathering, progress updates, and ongoing support.

Client: Logistics Industry.

• Implementing an ETL solution for the seamless migration and transformation of data into Azure storage.
• Developed ETL pipelines for the smooth transfer of data from sources like Azure PostgreSQL and Azure Data Lake to Azure Data Lake, through Azure Data Factory.
• Maintained task-based execution, and conducted post-deployment monitoring of the pipelines.
• Created documentation outlining business requirements for seamless ETL integrations.

Client: Pharmaceutical Industry.

• Developed launch trackers for the client's products.
• Leveraged Spark SQL in Databricks to produce the necessary metrics for launch trackers.
• Handled multiple business requests.

Client: Retail Industry

• Developed data pipelines with Azure Data Factory, Azure Databricks, and Azure Synapse Analytics, ensuring task-based execution and post-production deployment monitoring.
• Leveraged diverse data sources, including on-prem (Teradata, shared drive), Azure Cosmos DB, Azure Blob, Azure DevOps, and Adobe Analytics application, for integration into Azure Data Explorer.
• Used PySpark within Databricks for a brief period to perform data transformations.
• The pipeline deployment was executed through Azure DevOps. Prepared detailed documentation delineating business requirements for ETL integrations.
• Handled several applications, monitored jobs, and fulfilled customer requirements.