Data Engineer with 4 years of hands-on experience designing, deploying, and implementing end-to-end data solutions that drive business growth and measurable results.
Software Engineer 1, MAQ Software, 07/2021 - 09/2024.|Mumbai, India
Project 1: Cloud Data Migration & ETL Optimization
Objective: Migrate data products from ADLS Gen1 to Gen2, transition U-SQL scripts to PySpark, and optimize data processing workflows.
Responsibilities:
Project 2: On-Prem to Cloud Data Modernization.
Objective: Migrate on-premises SQL Server data warehouse to Azure while enhancing scalability, performance, and reporting capabilities.
Responsibilities:
Software Engineer 2, MAQ Software, 09/2024 - Present|Mumbai, India
Project 3: Enterprise Data Reporting and Automation. Objective: Build an automated reporting system that integrates data from multiple sources to deliver real-time business insights.
Responsibilities:
Key Achievements Across Projects
Data Pipeline Optimization: Designed and implemented solutions using Azure Synapse Analytics, boosting data processing efficiency and reducing processing times for large-scale operations by 25%.
ETL Automation: Automated and streamlined complex ETL workflows using Azure Data Factory, improving data accuracy and reducing manual errors by 50% across multiple data sources.
CI/CD Implementation: Built and maintained CI/CD pipelines in Azure DevOps, simplifying deployment processes, achieving 99.9% system uptime, and speeding up deployment cycles.
Data Visualization: Developed interactive and dynamic Power BI dashboards, enhancing reporting efficiency and enabling stakeholders to access real-time insights for better decision-making.
Data Migration: Successfully led the migration of over 5 terabytes of data, improving processing speed, and ensuring seamless integration with minimal downtime.
Spark Optimization: Optimized Spark and PySpark jobs, reducing processing times by 40% and ensuring the system could scale effectively to handle growing data volumes.
Team Collaboration: Worked in project teams to lead and coordinate database development, determine project scopes, and resolve technical limitations.
Technologies Used Across Roles: Azure Synapse Analytics, Azure Data Factory, Databricks, Power BI, Python, Spark, PySpark, CI/CD, SSIS, Azure Logic Apps, Azure DevOps, Power Apps, and Power Automate.