Avinash S
Chennai
Summary
- Azure Data Engineer with 2 years of experience in designing, developing, and deploying scalable ETL/ELT solutions on the Microsoft Azure Cloud platform.
- Developed and maintained robust data pipelines using Azure Data Factory (ADF) to orchestrate seamless data movement between AWS S3, ADLS Gen2, and Azure SQL Database.
- Leveraged PySpark and SQL on Azure Databricks to execute complex data transformations, cleaning, and aggregation, ensuring high-quality data for downstream analytics.
- Orchestrated dependency-driven workflows and scheduled triggers in ADF, ensuring strict adherence to batch processing SLAs and timely data availability.
- Managed Databricks cluster configurations, implementing auto-scaling policies to optimize compute performance and control operational costs.
- Implemented producer/consumer integration patterns to facilitate reliable and efficient data ingestion from diverse sources.
- Demonstrated strong problem-solving skills, effectively collaborating with cross-functional teams to troubleshoot pipeline failures and optimize performance.
- Committed to continuous learning, actively staying updated with emerging Azure services and software engineering best practices.
Overview
2
2
years of professional experience
6
6
Certifications
Work History
Associate Engineer
Virtusa
Chennai
01.2024 - Current
Project: Financial Transaction Processing System (Banking Domain) Technologies: Azure Data Factory (ADF), Azure Databricks (PySpark/SQL), ADLS Gen2, Azure DevOps (CI/CD), Apache Airflow, Jira.
- Engineered end-to-end ETL pipelines using Azure Data Factory to ingest high-volume financial transaction data from external APIs into ADLS Gen2.
- Developed optimized data transformations in Azure Databricks using PySpark and SQL, implementing a Medallion Architecture (Bronze, Silver, Gold layers) to ensure high data quality for reporting.
- Orchestrated and scheduled complex batch processing workflows using Apache Airflow, ensuring timely data availability and adherence to SLAs.
- Implemented CI/CD pipelines via Azure DevOps (Git) to automate deployment across DEV, UAT, and PROD environments, reducing manual deployment errors.
- Collaborated with QA teams using Jira for defect tracking and resolution, ensuring robust performance and data accuracy prior to production release.
Associate Engineer
Virtusa
01.2024 - Current
Project: Client & Audit Data Transformation Platform (Professional Services) Technologies: Azure Data Factory (ADF), Azure Databricks (PySpark/SQL), ADLS Gen2, Azure SQL Database, JSON.
- Engineered a metadata-driven ETL framework using Azure Data Factory and JSON configuration files to dynamically orchestrate data ingestion from SharePoint and SQL sources into ADLS Gen2.
- Developed scalable PySpark transformations within Azure Databricks, processing data through a Medallion Architecture (Raw, Enriched, Curated) to ensure data integrity and audit compliance.
- Orchestrated end-to-end data pipelines, automating the flow from raw ingestion to final loading into Azure SQL Database for downstream analytics.
- Implemented robust error handling and conditional logic within ADF, including automated email notifications for pipeline monitoring and failure recovery.
- Managed the full deployment lifecycle, conducting rigorous end-to-end testing in DEV and UAT environments to ensure seamless production releases.
Education
B.E - Computer Science And Engineering
Sri Venkateshware College of Engineering
Sriperumbudur,Kanchipuram07-2023
Skills
- Apache Spark and PySpark
- Azure Databricks and Data Factory
- SQL Server and MySQL
- Python programming
- Version control with GIT
- Project management with JIRA
Certification
Azure Databricks – Udemy
Acknowledgement
I Hereby declare that the above shared information is all true to my knowledge. Avinash S
Timeline
Associate Engineer
Virtusa
01.2024 - Current
Associate Engineer
Virtusa
01.2024 - Current
B.E - Computer Science And Engineering
Sri Venkateshware College of Engineering