Akshay Shankar
Bengaluru
Summary
Results-driven Lead Data Engineer with 11 years of experience optimizing enterprise data platforms in diverse sectors. Expertise in Azure Databricks, Apache Spark, and data architecture, with a strong focus on enhancing platform performance and reliability. Led initiatives that reduced storage costs and pipeline runtimes while automating reporting processes. Proficient in data governance and stakeholder management.
Overview
4
4
years of post-secondary education
11
11
years of professional experience
Work History
Lead Data Engineer
PEI Group
Bengaluru
08.2026 - Current
- Streamlined website plan performance assessment by implementing an automated data pipeline in Databricks, replacing manual reporting.
- Created reusable data foundation for reporting and analysis by integrating Google BigQuery data into Databricks Raw and Refined layers, ensuring consistent access to website performance data.
Senior Data Engineer
GlobalLogic
Bengaluru
10.2024 - 08.2026
- Reduced pipeline runtime by 40% through PySpark, Spark SQL, and Delta Lake optimization, improving compute efficiency for production workloads.
- Drove compute optimization assessments across production Databricks workloads, identifying over-provisioned clusters and recommending configuration and code changes to improve resource efficiency.
- Strengthened platform governance by implementing Unity Catalog for fine-grained access control, centralized governance, and end-to-end data lineage.
- Conducted compute optimization assessments on production Databricks workloads, identified over-provisioned clusters, and recommended configuration and code changes that enhanced resource efficiency.
- Designed and implemented an end-to-end Medallion Architecture spanning Landing, Raw, Refined, and Curated layers, improving data reliability, maintainability, and downstream consumption.
- Built scalable ingestion pipelines for data from SFTP, Amazon S3, Oracle, and SQL Server into Azure Data Lake Storage using Azure Data Factory and Azure Databricks.
- Enhanced production reliability through standardization of schema enforcement, incremental processing, and MERGE-based patterns across managed Delta workloads.
- Implemented Unity Catalog to establish fine-grained access control, centralized governance, and ensure end-to-end data lineage.
Senior Data Engineer
EY
Bengaluru
06.2019 - 10.2024
- Led a five-member data engineering team delivering an Azure Databricks-based regulatory-reporting solution, aligning technical delivery with business, compliance, and reporting stakeholders and reducing manual effort by up to 80%.
- Designed and delivered end-to-end reconciliation pipelines between IBM Db2 and Salesforce using PySpark, generating exception reports that enhanced data accuracy and streamlined issue resolution.
- Migrated legacy data workloads to Azure Databricks, enhancing processing scalability and operational stability for enterprise reporting workflows.
- Migrated legacy data workloads to Azure Databricks improving processing scalability, performance, and operational stability across enterprise reporting workflows.
- Designed and delivered end-to-end reconciliation pipelines between IBM Db2 and Salesforce using PySpark, generating exception reports that improved data accuracy, auditability, and issue resolution.
- Modernized legacy mainframe workflow using Python, NumPy, and Pandas, minimizing dependency on aging infrastructure and lowering maintenance overhead.
Senior Business Analyst
EXL
Gurgaon
10.2015 - 06.2019
- Engineered Hadoop-based data workflows for large-scale transaction and settlement datasets using HDFS, Hive, and Python, facilitating fraud-risk analysis and enhancing reporting capabilities for a major financial-services client.
- Translated complex fraud and transaction data into actionable insights through Tableau dashboards and analytical reports used by business stakeholders to monitor performance and identify reduction opportunities.
- Designed and maintained curated Hive tables for fraud and settlement data, applying transformations and partitioning strategies to improve query efficiency and Tableau reporting performance.
- Translated complex fraud and transaction data into actionable insights via Tableau dashboards and analytical reports, empowering business stakeholders to monitor performance and uncover fraud reduction opportunities.
Education
B.E - Electrical & Electronics Engineering
Visvesvaraya Technological University
Bangalore 09.2011 - 06.2015
Skills
- Big data technologies
- Data architecture processing
- Data pipeline orchestration
- Data governance quality
- Performance cost optimization
- Databases / Data Warehouses: Oracle, SQL Server, MySQL
- Programming Languages: Python, SQL
- Cloud orchestration
- Version Control and CI/CD: Git, GitHub, Azure DevOps, CI/CD
- Visualization: Tableau
- Project Framework: Agile
Timeline
Lead Data Engineer
PEI Group
08.2026 - Current
Senior Data Engineer
GlobalLogic
10.2024 - 08.2026
Senior Data Engineer
EY
06.2019 - 10.2024
Senior Business Analyst
EXL
10.2015 - 06.2019
B.E - Electrical & Electronics Engineering
Visvesvaraya Technological University
09.2011 - 06.2015