Aniruddha Deshmukh
Data Engineer
Summary
Having 7.5+ years of experience as a Data Engineer, including 3+ years in Azure cloud technologies such as Azure Data Factory (ADF), Azure Databricks, PySpark, and Data Flows. The remaining experience is in SAS (Base & Advanced), Informatica, SQL, and Python, with strong skills in data processing, transformation, and workflow automation across both on-premise and cloud environments.
Overview
8
8
years of professional experience
Work History
Sr. Data Engineer
3KT Technologies
08.2024 - Current
- Built scalable data transformation pipelines in Azure Databricks using PySpark and SQL, streamlining complex ETL processes.
- Developed and maintained ADF pipelines to orchestrate end-to-end workflows, ensuring timely data delivery across environments.
- Optimized storage and retrieval by implementing partitioning strategies in ADLS Gen2, improving query performance and cost-efficiency.
- Applied Medallion Architecture (Bronze, Silver, Gold) for layered, modular data modeling in ADLS Gen2.
- Improved pipeline maintainability by introducing dynamic parameterization in ADF, eliminating hardcoded values and enabling reusability.
- Automated time-based and trigger-based ETL scheduling in ADF, supporting near-real-time data ingestion and processing.
- Used Delta Lake to enable ACID-compliant transactions and efficient upserts and deletes in large datasets.
- Configured and managed Linked Services in ADF for secure integration with on-prem and cloud-based data sources.
- Built a Power BI dashboard for Customer Profitability and 360 Customer Insights, enabling data-driven strategies in retail banking.
Sr. Data Engineer
Wipro Technologies
02.2022 - 08.2024
- Developed metadata-driven ETL pipelines in Azure Data Factory (ADF) to ingest data from diverse sources (SQL Server, CSV, JSON APIs) into Azure Data Lake Storage Gen2 (ADLS Gen2).
- Integrated Self-Hosted Integration Runtime (SHIR) in ADF to securely pull on-premise SQL Server data into the Azure ecosystem.
- Implemented Medallion Architecture (Bronze, Silver, Gold) for modular data modeling and scalable lakehouse design in ADLS Gen2.
- Utilized Azure Key Vault to manage and access secrets, connection strings, and SAS tokens securely across ADF pipelines.
- Automated daily and event-based ADF pipelines and monitored them via Azure Monitor, setting up alerts and diagnostics for failures, latency, and retries.
- Engineered high-performance PySpark jobs in Azure Databricks to process and cleanse nested JSON and Parquet datasets, including schema evolution handling.
- Applied Spark caching, broadcast joins, strategies to boost performance and reduce shuffle operations in large-scale data transformations.
- Implemented Delta Lake for time-travel queries, scalable metadata management, and efficient handling of upserts/deletes.
Associate Consultant
Protogene Consulting Pvt Ltd
07.2020 - 02.2022
- Understood existing SAS 9.4 code (Base & Advanced SAS) and SQL queries, and successfully migrated them to Python.
- Validated Python output against original SAS program results to ensure consistency and accuracy.
- Used Informatica Developer 10.5.1 for ETL operations: extracted, transformed, and loaded data from flat files into target databases.
- Extracted data from flat files and relational databases into a staging area.
- Applied various Informatica transformations such as Filter, Expression, Lookup, Joiner, Router, Sorter, Union, and Sequence Generator for data processing.
SAS Consultant
Nupeak IT Solution
01.2019 - 07.2020
- Responsible for manipulating, transferring and managing data in SAS.
- Using Proc http fetching JSON data from API.
- Involved in writing code using SAS and SAS/Macros to extract clean and experienced with Creating Dashboard, multiple Reports and Data query On SAS VA.
Data Analyst
Qualsquad Infotech Pvt Ltd
11.2017 - 12.2018
- Skills used – Base SAS, SQL, MS-Excel, MS-office
Education
MCA -
Jaipur National University
04.2001 -
BCA -
College Of Computer Science, Pune
04.2001 -
Skills
- Azure Data Factory (ADF)
- Azure Data Lake Storage (ADLS)
- Databricks (PySpark)
- Azure SQL DB
- Data Pipeline Design
- ETL Development (Data Integration, Transformation)
- Python Pandas, SQL
- Data Warehousing
- Delta Lake
- Power BI
- Azure DevOps
- Base SAS
- Advanced SAS
Timeline
Sr. Data Engineer
3KT Technologies
08.2024 - Current
Sr. Data Engineer
Wipro Technologies
02.2022 - 08.2024
Associate Consultant
Protogene Consulting Pvt Ltd
07.2020 - 02.2022
SAS Consultant
Nupeak IT Solution
01.2019 - 07.2020
Data Analyst
Qualsquad Infotech Pvt Ltd
11.2017 - 12.2018
MCA -
Jaipur National University
04.2001 -
BCA -
College Of Computer Science, Pune
04.2001 -
Similar Profiles
Chong Soo KenChong Soo Ken
Cloud Finance Speciallist (Cloud BG CFO) at Sparkoo Technologies (Malaysia) Sdn. Bhd (Cloud Subsidiary for Huawei Technologies (Malaysia) Sdn Bhd)Cloud Finance Speciallist (Cloud BG CFO) at Sparkoo Technologies (Malaysia) Sdn. Bhd (Cloud Subsidiary for Huawei Technologies (Malaysia) Sdn Bhd)
SRIRAM RSRIRAM R
Software Quality Analyst at GAVS Technologies (formerly Neurealm Technologies Pvt Ltd)Software Quality Analyst at GAVS Technologies (formerly Neurealm Technologies Pvt Ltd)
Venkata Nagaraju InakolluVenkata Nagaraju Inakollu
Technical Lead(System Design) at Relysys Technologies Pvt Ltd (now Invendis Technologies pvt ltd)Technical Lead(System Design) at Relysys Technologies Pvt Ltd (now Invendis Technologies pvt ltd)
Balachakradhar RocksagayarajBalachakradhar Rocksagayaraj
Sr. Data Engineer at GAVS TechnologiesSr. Data Engineer at GAVS Technologies
OMKAR R JAGTAPOMKAR R JAGTAP
Sr. Data Engineer at BarclaysSr. Data Engineer at Barclays