Naveen
Hyderabad
Summary
Results-driven Data Engineering Lead specializing in Azure Data Factory, Fabric, and Databricks. Skilled in architecting and implementing data migration solutions from legacy ETL frameworks to Azure Cloud. Focused on pipeline architecture and problem-solving, contributing to cost reductions and improved analytics capabilities across various business units.
Overview
12
12
years of professional experience
Work History
Data Engineering Lead
Deloitte USI
Hyderabad
05.2024 - Current
- Led design, development, and optimization of 60+ mission-critical ADF pipelines processing 6-10 TB/day from diverse sources (Oracle, GuideWire, APIs, on-prem files,Json,Parquet files) - achieved 99.8% uptime, reduced average end-to-end latency by 55%, and supported enterprise analytics & reporting for 200+ users.
- Architected advanced ADF solutions including mapping data flows for complex transformations, self-healing patterns (retry policies, wait activities, error handling), parameterized pipelines, and dynamic content — improved pipeline reusability and maintainability across multiple business units.
- Spearheaded ETL migration from IBM InfoSphere DataStage to Azure Data Factory - migrated 150+ jobs, translated custom logic to ADF native activities/data flows, eliminated legacy licensing costs, and achieved 40% reduction in monthly ETL compute spend through serverless scaling.
- Implemented ADF monitoring, alerting, and logging using Azure Monitor, Log Analytics, and custom Logic Apps — facilitated proactive issue resolution and reduced mean time to recovery (MTTR) by 70%.
- Integrated ADF with Azure Databricks for heavy Spark-based transformations and Azure Synapse for analytics consumption — optimized partitioning, caching, and autoscaling to cut transformation runtimes from hours to minutes.
- Project : Intelligence 360
- Client : WSIB , Canada.
Senior Azure Data Engineer
SS&C Technologies
Hyderabad
05.2022 - 04.2024
- Led design and delivery of large-scale ADF pipelines (100+ activities) integrating on-prem, SaaS, and cloud sources into Azure Data Lake for internal analytics product SalesConnect.
- Migrated legacy Talend ETL to Azure-native services (ADF + Databricks) — cut monthly compute costs by 35–50% while increasing throughput.
- Developed reusable mapping and control flows in ADF; implemented self-healing patterns with Wait-on-Completion and retry logic, achieving pipeline reliability of over 99.5%.
- Designed cost-optimized storage and compute architecture using ADLS cold/hot/archive tiers, decreasing storage costs by 25%.
- Project : SalesConnect
Senior Azure Data Engineer
HCL Technologies
Hyderabad
09.2021 - 05.2022
- Developed complex transformations using Databricks notebooks (PySpark) and ADF data flows.
- Designed monitoring and alerting solution with Azure Monitor and Logic Apps to proactively manage pipeline failures.
- Developed and maintained ADF-based ingestion framework integrating 10+ data sources including ILC, Taleo, Peoplesoft, and Workday.
- Created a centralized HR datalake to streamline data access and reporting.
- Project : HR Datalake
- Client : ING Bank, Netherlands
Senior Data Engineer
HSBC Software Pvt. Ltd
Hyderabad
11.2019 - 08.2021
- Designed and developed ADF pipelines for migrating on-premise DB2 data warehouse to Azure cloud, facilitating a smoother transition to cloud infrastructure.
- Developed complex transformations using Databricks notebooks (PySpark) and ADF data flows.
- Created monitoring and alerting solution with Azure Monitor and Logic Apps to proactively address pipeline failures, enhancing system reliability.
- Analyzed existing IBM Datastage jobs and developed low-level design documents for Azure migration, ensuring comprehensive documentation for the transition.
- Worked on GCP & Azure POCs to finalize suitable cloud architecture for migration of legacy systems.
- Project : SIM – Smart Information Management
- Client : HSBC Bank, Dubai
Datastage Developer
Optum Global Solutions Pvt. Ltd
Hyderabad
04.2016 - 11.2019
- Designed and developed Datastage jobs to implement complex business logic on common format text files for loading into Oracle database.
- Project : DQM - Data Quality management.
Datastage Developer
Magna Infotech
Hyderabad
06.2014 - 03.2016
- Developed solutions as an IBM Datastage Developer.
Education
B-Tech -
PRRM Engineering College
Hyderabad06-2012
Skills
- Microsoft Fabric
- Azure Data Factory (ADF)
- Azure Databricks
- OneLake / Delta Lake
- Data Modeling
- Pipeline architecture
- CI/CD tools
- PySpark
- Python
- SQL
- Azure SQL DB
- Azure Cosmos DB
- IBM Infosphere Datastage
- IBM Cloud PAK
Role
Lead / Senior Azure Data Engineer
Timeline
Data Engineering Lead
Deloitte USI
05.2024 - Current
Senior Azure Data Engineer
SS&C Technologies
05.2022 - 04.2024
Senior Azure Data Engineer
HCL Technologies
09.2021 - 05.2022
Senior Data Engineer
HSBC Software Pvt. Ltd
11.2019 - 08.2021
Datastage Developer
Optum Global Solutions Pvt. Ltd
04.2016 - 11.2019
Datastage Developer
Magna Infotech
06.2014 - 03.2016
B-Tech -
PRRM Engineering College