Collaborated with business teams to translate requirements into technical ETL specifications, ensuring accurate data flow from multiple sources to the Azure platform.
Designed, implemented, and optimized end-to-end data pipelines in Azure using Azure Data Factory and Azure Databricks, ensuring efficient data transformation and loading.
Validated transformation logic using Python and Spark SQL, ensuring accuracy and consistency of large datasets in Databricks.
Performed ETL testing and data validation to ensure data integrity across Azure SQL databases and external sources.
Developed performance optimization strategies, using partitioning and indexing techniques, to enhance the efficiency of data ingestion and transformation processes.
Logged, tracked, and resolved defects using JIRA, and performed regression testing to maintain system stability during ETL jobs.
Conducted load and performance testing in Azure Databricks, ensuring scalability and smooth operation under high data loads
Skills
Azure Databricks
Azure Data Factory
Azure SQL Database
Azure Synapse Analytics
SQL Server Integration Services (SSIS)
Azure Databricks Notebooks
SQL Server Management Studio (SSMS)
SQL
Python
ETL Testing
Data Engineering
Data Warehousing
Certification
Accomplished DP -203(Azure Data Engineer Associate) Certification.
Ocean Certified: Azure Developer (Fresher) by Capgemini
Personal Information
Total Experience: 3+ years of experience in designing, building, and managing data pipelines using Azure cloud services. Adept in ETL testing, data validation, and automation with a strong background in SQL and Azure Databricks. Skilled at delivering scalable data solutions and optimizing ETL processes to ensure data accuracy and integrity.