Sourabh Gupta
Summary
Senior Azure Data Engineer with 9 years of experience in building scalable data pipelines and warehouses using Azure technologies. Expertise in designing comprehensive solutions that integrate various data sources and utilize ETL methods to generate actionable insights. Proficient in collaborating across teams and managing secure data environments with Azure Key Vault, Databricks, and Azure Synapse to enhance analytics for enterprise-level projects.
Overview
9
9
years of professional experience
Work History
Lead Azure Data Engineer
Nagarro
09.2022 - Current
- Led development of large-scale Customer Value Management platform, enhancing customer analytics and business decision-making.
- Architected and delivered over 200 scalable data pipelines using Azure Data Factory, Azure Synapse, and Azure Databricks.
- Designed comprehensive data engineering frameworks for ingestion, transformation, and analytics across Azure Data Lake and Synapse.
- Built dynamic, metadata-driven ETL/ELT pipelines utilizing ADF parameters for improved reusability and maintainability.
- Implemented CI/CD pipelines for ADF and Databricks via Azure DevOps, enabling automated deployments and version control.
- Leveraged Databricks Delta Lake for incremental loads, upserts, schema evolution, and reliable data versioning.
- Established secure secret management with Azure Key Vault and IAM/RBAC across data platforms.
- Designed centralized Power BI dashboards for monitoring data quality, pipeline health, and data freshness.
Senior Azure Data Engineer
Deloitte
09.2021 - 09.2022
- Architected end-to-end Azure data pipelines to integrate dealer performance metrics for advanced analytics.
- Established performance segmentation frameworks, grouping dealers by operational units to enhance benchmarking accuracy.
- Built scalable ingestion and transformation pipelines with Azure Data Factory, Azure Databricks, and Azure Data Lake Storage.
- Implemented metadata-driven transformations and business rules using ADF activities such as Lookup and ForEach.
- Enhanced reporting accuracy by approximately 30% through data quality validation frameworks.
- Set up monitoring, logging, and automated scheduling for ADF pipelines to ensure reliable data processing.
- Developed CI/CD pipelines for ADF utilizing Azure DevOps, facilitating automated deployments and version control.
- Orchestrated Databricks notebooks and workflows for data cleansing, transformation, and feature engineering.
Data Engineer
Coforge
09.2019 - 05.2021
- Implemented data integration solutions utilizing Azure ETL tools and frameworks.
- Created robust ETL processes to extract, transform, and load large datasets into Azure SQL Database and Azure Data Lake Storage.
- Developed and maintained Azure Data Factory pipelines, ensuring seamless data flow and integration.
- Optimized ADF for incremental data loading and change data capture techniques to enhance processing efficiency.
- Configured ADF pipeline execution using parameters and variables for dynamic data handling.
- Utilized Azure Databricks for data exploration, pre-processing, and transformation tasks.
- Integrated Azure Databricks with Azure Data Lake Storage for scalable data processing capabilities.
- Maintained ETL workflows in Databricks, facilitating smooth extraction and loading of diverse data sources.
ETL Developer
Tata Consultancy Services
08.2016 - 09.2019
- Leveraged extensive experience with diverse data sources, including SQL, Oracle, flat files, and Salesforce.
- Developed Informatica Cloud mappings and data synchronization tasks to streamline data processes.
- Published regional data to downstream systems, ensuring accuracy and timeliness in Salesforce integration.
- Executed Integration Hub projects, demonstrating proficiency in topic tables, publication, and subscription components.
- Created optimized ETL code for full loads and incremental runs during daily operations.
- Designed mappings for staging and publish layers, employing SCD type I and SCD type II concepts.
- Performed data extraction, transformation, and loading using advanced techniques like Lookup, Aggregator, and Router.
- Implemented performance tuning strategies in IICS through key range partitioning and pushdown optimization.
Education
Bachelor of Technology -
Institute of Technology & Management
Gurgaon, Haryana, India06.2016
Skills
- Microsoft Azure Synapse
- ETL and data integration
- SQL database management
- Snowflake data warehousing
- Azure Data Lake storage
- Databricks for big data analytics
- Power BI for data visualization
- Azure DevOps methodologies
Timeline
Lead Azure Data Engineer
Nagarro
09.2022 - Current
Senior Azure Data Engineer
Deloitte
09.2021 - 09.2022
Data Engineer
Coforge
09.2019 - 05.2021
ETL Developer
Tata Consultancy Services
08.2016 - 09.2019
Bachelor of Technology -
Institute of Technology & Management