Dedicated developer with experience in achieving tangible results and cross-team collaboration. Proactive and excited to partner with like-minded individuals to achieve goals.
Developed end to end ETL pipelines in Azure Databricks, processing and transforming datasets.
• Created reusable SQL views and stored them in Catalog, Integrating with Power BI for daily refresh
and reporting.
• Designed and implemented Apache Airflow DAGs fortriggering of data pipelines and Databricks jobs
by using service principle and Managed Identities.
• Create mount points to access azure data lake storage(ADLS 2) or azure blob storage account in
azure databricks.
• Create secret scope authentication in azure databricks by using azure key valut.
• Excellent knowledge of ADF building components like Integration Runtime, Linked Services,
Datasets.
• Moving different types of data from on-premises or different sources into two or more
application to load data from source to raw, raw to defined, defined to staging and staging to
dimension layers.
• Setup alerts for failed activities in pipeline and databricks jobs by using azure monitor alerts
and logic apps.
• Sending Email alerts when pipeline fails orany files added, deleted in the Blob storage account
using Azure Logic Apps.
• Familiar with various Azure storage services like Azure Blob storage, Azure Data lake and
Azure data lake Gen storage.
• In Databricks executed projects with incremental loading strategies, including the use of
Autoloader for seamless data ingestion and real-timeupdates.
• Proficient in SQL scripting, debugging and performance tuning that reduced 30% of runtime in
production environments.
• Worked with peers from various functional areas to define IT wide processes like code reviews,
unit test documentation and knowledge sharing sessions.
• Familiar with Agile methodologies, including Scrum and Scaled Agile.
• Collaborated with international clients and multidisciplinary teams, interacting with Dutch
clients frequently.
Azure Synapse Analytics
Azure Monitor
PySpark
Power BI