Senior Data Engineer with 5+ years of expertise in developing scalable Azure-based data solutions, ETL pipelines, and data warehouses. Experienced with a strong track record of improving data reliability, performance, and business outcomes.
Overview
5
5
years of professional experience
Work History
Packaged Application Developer - Senior Data Engineer
Accenture
Bengaluru
10.2024 - Current
Designed and developed scalable ETL/ELT pipelines for ingesting, transforming, and processing large-scale datasets across cloud platforms.
Built and optimized data transformation workflows using Python, SQL, PySpark, and Spark SQL to support analytical and reporting requirements.
Developed Azure Data Factory pipelines, triggers, and orchestration workflows to automate end-to-end data processing.
Engineered robust data integration solutions across multiple data sources, including Azure Cosmos DB, ADLS Gen2, and Azure Synapse Analytics.
Implemented data validation, reconciliation, and quality control frameworks to ensure data accuracy, consistency, and reliability.
Developed Python-based automation scripts to streamline data migration from legacy platforms to Azure Synapse environments.
Optimized data processing performance through query tuning, Spark optimization, and efficient pipeline design.
Performed detailed analysis and debugging of data pipeline failures, identifying root causes and implementing permanent fixes.
Collaborated with business analysts, architects, and development teams to translate business requirements into scalable technical solutions.
Supported deployment, testing, and release activities while ensuring adherence to development best practices and coding standards.
Created monitoring and reporting solutions to track pipeline health, data freshness, and overall platform performance.
Contributed to continuous improvement initiatives by evaluating and implementing new technologies, tools, and development methodologies.
Senior Engineer - Cloud & Infra Management
LTIMindtree
03.2023 - 09.2024
Managed and resolved production incidents by coordinating with cross-functional teams, ensuring timely restoration of services and minimizing business impact.
Led incident triage, root cause analysis, and resolution efforts for critical system and data platform issues, improving system reliability and operational efficiency.
Acted as the primary point of contact during major incidents, providing clear communication, status updates, and post-incident reports to stakeholders and customers.
Enhanced customer experience by delivering rapid incident resolution, maintaining transparent communication, and ensuring minimal disruption to business operations.
Collaborated with engineering, infrastructure, and support teams to drive continuous service improvements and optimize incident response processes.
Senior Analyst
Capgemini
03.2021 - 03.2023
Investigated system issues and implemented resolutions to reduce downtime.
Actively monitored ADF pipelines and data flow and checked if the data is up to date.
Creating triggers for various products in ADF pipelines and scheduling for both master and staging pipelines.
Worked with the data in data warehouse using SSMS Job monitoring tool to make sure all the scheduled job runs effectively.
On-boarding of data from CosmosDB to ADLS Gen 2 during MAP Migration.
Preparing scripts in python to effectively copy data from the legacy environment to migrated environment (Azure synapse workspace).
Performed system analysis, documentation, testing, implementation, and user support for platform transitions.
Validated results and performed quality assurance to assess accuracy of data.
Identified and resolved problems through root cause analysis and research.
Queried databases for information needed for report processing.
Researched and adopted new technologies to add value to existing offerings.
Preparing daily reports of data warehouse runs and running daily queries to check if all the product runs are upto date and generating mails to the team daily.
Education
B.Tech/B.E. - Computers
Calcutta Institute of Engineering and Management
05.2020
12th -
CISCE
03.2016
10th - English
CISCE
03.2014
Skills
Azure Data Factory
Azure Data Lake
Azure Synapse
Azure Data Pipelines
Python
PySpark
Spark SQL
Spark Analytics
SQL
SSMS
ETL/ELT design
Timeline
Packaged Application Developer - Senior Data Engineer