Yashwanth Basavan

Environment: Azure Data Factory, ADLS, Azure Sql, Azure DevOps

• Migrated client data from an on-premises environment to the cloud by transitioning existing SQL Server Integration Services (SSIS) jobs to Azure Data Factory (ADF).
• Designed and developed ETL pipelines using Azure Data Factory (ADF) to move and transform data.
• Created stored procedures for complex data transformations.
• Extracted data from various sources, including APIs and SFTP servers.
• Used Logic Apps to automate email notifications for pipeline status updates.
• Monitored production pipelines and addressed performance issues to ensure seamless operations.

Environment: Azure Data Factory, Data flows, ADLS, PostgreSql, SQL, Azure DevOps

• Designed an ETL pipeline to load JSON files into database tables.
• Implemented Slowly Changing Dimension (SCD) Type-2 to maintain historical data in dimension and fact tables.
• Developed error-handling mechanisms to log and capture errors in parent and child pipelines.
• Automated database schema versioning using Azure DevOps for seamless updates.
• Used ADF Data Flows for advanced data transformations.

POC: Incremental Data Ingestion using Databricks Auto Loader

Environment: Databricks, Source: ADLS Destination: Delta Tables (Bronze → Silver → Gold).

• Used Databricks Auto Loader to ingest raw data files from ADLS into the Bronze Layer for incremental processing.
• Applied data transformations in the Silver Layer, including data cleansing (null handling, deduplication) data type casting and column standardization.
• Aggregated and enriched data in the Gold Layer for analytical reporting.
• Optimized performance using Delta Lake features like Z-Ordering and Vacuum for storage efficiency.

Associate Consultant (Contractor: 10/2021 – 08/2022, Permanent: 08/2022 – 2023)

Environment: Azure Data Factory, Oracle, ADLS, Synapse DB, Postgresql , Azure DevOps

• Designed ETL processes using Azure Data Factory to collect, transform, and process data
• Collaborated with stakeholders for requirements gathering, analysis, design, and development
• Developed data pipelines to transfer large volumes of data (GBs) from Oracle to PostgreSQL
• Implemented data transformations using Data Flows in Azure Data Factory
• Extracted data from Oracle and Blob Storage and loaded it into PostgreSQL and Synapse databases
• Utilized Azure Synapse as an intermediate data staging area
• Leveraged control flow activities like Get Metadata, Lookup, Copy Activity, For Each, If Condition, Filter, and Set Variable for efficient pipeline management
• Used transformation components such as Aggregate, Alter Row, Cast, Conditional Split, Derived Column, Exists, Join, Lookup, New Branch, Select, and Union for data manipulation
• Implemented Slowly Changing Dimension (SCD) Type-1 in Synapse
• Prepared unit test cases and release documentation for QA processes
• Followed Agile methodologies and tracked user stories and tasks using JIRA
• Deployed pipelines to QA environments using Azure DevOps CI/CD pipelines