ASHWIN MADHAVAN
Bangalore
Summary
Accomplished Senior Data Engineer with expertise in Azure Databricks and Apache Spark, delivering over 30 production-grade data solutions at Sopra Steria. Proven ability to optimize data pipelines and enhance performance, while effectively collaborating with stakeholders to drive impactful results. Strong analytical skills combined with a commitment to data governance and quality.
Overview
5
5
years of professional experience
1
1
Certification
Work History
Module Lead / Senior Data Engineer
Sopra Steria
Bangalore
04.2024 - Current
- Delivered 30+ production-grade data solutions in partnership with business stakeholders, aligning data capabilities with business needs.
- Implemented incremental ingestion pipelines using CDC patterns, enhancing data update reliability.
- Designed scalable data pipelines within Palantir Foundry, enabling ingestion and transformation of enterprise datasets.
- Optimized distributed Spark transformations improving overall data processing performance.
- Managed 70+ datasets with version control, lineage tracking, and governance policies, ensuring data integrity and compliance.
- Built reusable transformation logic using Code Workbooks, improving developer productivity and pipeline maintainability.
Senior Data Engineer
Hexaware Technologies
01.2023 - 04.2024
- Implemented Lakehouse architecture using Delta Lake and ADLS Gen2 enabling ACID transactions and schema evolution.
- Built large-scale ETL pipelines using Azure Databricks and PySpark to enable processing of 1TB+ of daily data for analytics.
- Developed ingestion pipelines with Azure Data Factory and Databricks Workflows to streamline data accessibility for analytics.
- Optimized Spark jobs and tuned partitions to enhance data ingestion performance, supporting timely data availability.
- Designed scalable data models supporting analytics across multiple business teams.
Big Data Developer
Tata Consultancy Services
06.2021 - 12.2022
- Built Spark-based ETL pipelines processing millions of records daily.
- Optimized Spark jobs and Hive queries, significantly boosting pipeline performance.
- Migrated legacy relational data to Hadoop ecosystem with Sqoop and Hive, enhancing data accessibility.
- Implemented data validation and transformation workflows, ensuring high data quality for analytics.
- Managed British Telecom project
Education
Master of Computer Science -
Alagappa University
ChenaiBachelor of Computer Applications -
Ramakrishna Mission Vivekananda College
Skills
- Data Engineering Platforms
- Azure Databricks
- Apache Spark
- Delta Lake
- Azure Data Factory
- Azure Data Lake Gen2
- Azure SQL
- Cloud
- Microsoft Azure
- Architecture & Design
- Lakehouse Architecture
- Medallion Architecture
- CDC / Incremental Data Pipelines
- Data Modeling (Star Schema, SCD Types)
- Distributed Data Processing
- Data pipeline design
- Security & Governance
- Data Governance & Lineage
- Azure RBAC
- Azure Key Vault
- Programming
- Python
- PySpark
- SQL
- Big Data Ecosystem
- Hadoop
- Kafka
Certification
• Microsoft Azure Fundamentals (AZ-900)
• Microsoft Azure Data Fundamentals (DP-900)
• Databricks Certified Associate Developer
• Microsoft Azure Data Fundamentals (DP-900)
• Databricks Certified Associate Developer
Projects
- Azure Lakehouse Platform, Ernst & Young, Architected enterprise Azure Lakehouse platform using Databricks, ADF, and ADLS Gen2., Built automated ETL pipelines orchestrated through Azure Data Factory pipelines., Implemented Delta Lake storage layer supporting ACID transactions and schema evolution., Applied Unity Catalog governance policies to manage data access and lineage.
- Data Platform Development, Airbus (Palantir Foundry), Developed ingestion pipelines integrating enterprise datasets into Palantir Foundry., Built transformation logic using Slate applications and Foundry code workbooks., Implemented scalable data pipelines supporting analytics use cases across multiple teams., Collaborated with business stakeholders to translate requirements into production-grade pipelines.
Timeline
Module Lead / Senior Data Engineer
Sopra Steria
04.2024 - Current
Senior Data Engineer
Hexaware Technologies
01.2023 - 04.2024
Big Data Developer
Tata Consultancy Services
06.2021 - 12.2022
Master of Computer Science -
Alagappa University
Bachelor of Computer Applications -
Ramakrishna Mission Vivekananda College