

Results-oriented Azure Data Engineer with 11–12 years of experience specialising in Azure Data Engineering (ADF, Azure Databricks, Synapse, Azure SQL) built on a strong foundation of SQL Server/T-SQL development. Proficient in Python scripting and PySpark/Databricks for large-scale data transformation, Delta Live Tables, Unity Catalog governance, and Structured Streaming. Hands-on with Databricks Asset Bundles (DAB) for CI/CD-as-code deployment, Photon-optimised workloads, and production pipeline support across Azure and on-premises environments.
Pricing & Inventory Analytics — VINtelligence data ingestion, Databricks-based pricing intelligence, and Synapse reporting for the auto-parts platform.
Python, PySpark & Databricks
• Wrote standalone Python scripts for data validation, secret retrieval via Azure Key Vault SDK, and utility automation tasks outside the Spark context — directly addressing Python-as-scripting requirement.
• Designed a Customer 360 data model on Azure Databricks integrating transactional, clickstream, and support-ticket data into unified Delta tables with SCD Type 2 history for customer profile changes.
• Built a near-real-time personalisation feed using Azure Event Hubs and Spark Structured Streaming, scoring customer sessions for cross-sell recommendations within seconds of an event.
• Used Auto Loader with schema evolution for incremental file ingestion; applied AQE, broadcast joins, and Z-Order/OPTIMIZE on high-traffic Gold tables to maintain dashboard query performance as data volumes grew.
Delta Live Tables & Unity Catalog
• Added DLT expectations to validate order totals, currency codes, and duplicate transaction IDs, cutting weekly reconciliation tickets from ~12 to a handful — concrete DLT data-quality gating.
• Worked with the Data Governance team to define Unity Catalog access policies (schema-level access control, PII column masking) on owned tables and registered pipelines for lineage tracking.
Databricks Asset Bundles (DAB) & CI/CD
• Collaborated with the DevOps team to deploy Databricks notebooks and Delta Live Tables using Databricks Asset Bundles (DAB) on the Azure DevOps release pipeline — Dev → UAT → Prod promotion with parameterised job configs.
• Maintained bundle YAML manifests for cluster policies, job configs, and pipeline definitions, enabling version-controlled, repeatable Databricks deployments.
Cost & Performance Optimisation
• Right-sized Databricks cluster policies (auto-scaling limits, spot-instance mix, job vs. all-purpose clusters) and adopted Photon runtime for SQL-heavy notebooks, meaningfully reducing compute spend.
• Investigated failed Databricks job runs using cluster event logs and Spark UI; resolved SLA-impacting incidents as part of the on-call rotation.
Python & PySpark on Databricks
• Wrote Python utility scripts for file-arrival monitoring, secret management via Azure Key Vault SDK, and automated post-load validation checks outside the Spark execution context.
• Built PySpark pipelines on Azure Databricks to compute dynamic pricing recommendations from VINtelligence transaction and competitor-pricing data, using Delta Lake for versioned pricing history.
• Scheduled nightly Databricks Workflows jobs to refresh inventory-forecasting datasets for the data-science team's Azure ML model.
Azure Data Factory
• Designed parameterised ADF pipelines using Linked Services, Datasets, Copy, Lookup, Conditional Split, Filter, Join, and Union activities.
• Automated ingestion of VINtelligence flat files from FTP into ADLS Gen2 with archival workflows for compliance; replaced legacy SSIS ETL with ADF + Databricks notebooks.
Azure SQL / Synapse & Security
• Developed and optimised T-SQL stored procedures, functions, and views for auto-parts groupings, pricing updates, and inventory lookups; implemented SCD Type 1 & 2 and PolyBase external tables.
• Created a Synapse serverless SQL pool layer over curated Delta tables for ad-hoc analyst queries; implemented row-level security in shared Power BI datasets for regional managers/dealers.
• Used Azure Key Vault for Linked Service secrets and configured RBAC on Azure SQL and Synapse for least-privilege, read-only dealer-facing access.
• Managed production deployments via Git (Azure Repos), Redgate script generation, and CI/CD pipelines; triaged failed ADF runs and SQL Agent failures with hotfix and rollback procedures.
Athena Health — Patient Registration, Appointment, and Billing modules for healthcare data integration.
• Performed data profiling and validation for Athena Practice and Athena Flow healthcare applications.
• Designed SSIS ETL packages for extraction, validation, and loading against Athena modules (Patient Registration, Account, Appointment).
• Implemented ADF pipelines for medical billing data extraction from Azure Blob Storage; managed deployments across test, UAT, and production environments.
• Ingested satellite imagery, cadastral data (unstructured), and land transaction logs into ADLS Gen2 using ADF pipelines; processed TB-scale geospatial datasets using PySpark in Databricks.
• Applied PySpark transformations to cleanse and classify geospatial datasets; enabled detection of illegal land encroachments and land-use optimisation through advanced spatial analytics.
• Implemented SCD Type 2 for historical land ownership and zoning records; implemented CDC for DML audit history; designed SQL stored procedures and migration scripts for core land-management logic.
• Delivered Power BI reports with geospatial visualisations and interactive dashboards for urban-development policy-making.
• Set up Git feature branches, Azure Repos PRs, code reviews, and CI/CD best practices; mentored and task-allocated to junior developers.
Workers Comp & Smart Audit — integrated claims management for workers compensation and auto liability.
• Analysed requirements; created and altered SQL Server objects (tables, stored procedures) with unit testing via TFS source control.
• Designed ETL-SSIS packages for Fee Schedule module (sdup, suds, pswi formats) with incremental loading and performance tuning; developed ADF pipelines with SCD Type 1 & 2.
• Collaborated with business analysts to translate requirements into technical ETL specifications; mentored junior team members.
Samaritan Health Plans — reporting across network hospitals, doctors, and specialists.
• Designed and maintained Crystal Reports with complex multi-database SQL queries; optimised report performance through SQL tuning and best-practice report design.
SDZ — Australian Government governance workflow and data dictionary management.
• Implemented Telerik Reporting with data connectivity, binding, parameterisation, and multi-format rendering (HTML, PDF, Excel, Word, CSV).
• Managed Azure Blob Storage; built ADF pipelines for JSON ingestion and used OPENJSON to convert JSON into structured SQL Server format.
Telecom domain — Indus Towers & ATC passive infrastructure WFM data (BTS, MW, RTT, GBT).
• Developed comprehensive SSIS ETL processes for CSV-based WFM data: Extract, Transform (Derived Column, Conditional Split, Aggregate, Lookup), and Load.
• Wrote complex T-SQL queries and stored procedures; optimised database performance through indexing, bulk inserts, and query tuning; scheduled SSIS packages via SQL Server Agent.