Priyesh
Data Engineering & Architecture Leader
Gurgaon,HR
Summary
Data Engineering leader with 14+ years of experience, specializing in big data platforms, real-time data pipelines, and cloud-native architectures across fintech, telecom, and digital media. Expertise in designing and scaling enterprise-grade Data Lakes, Warehouses, and Mesh solutions on both on-prem and AWS (EMR, Glue, Redshift, Lake Formation). Architected and delivered multiple high-throughput, production-grade pipelines powering mission-critical analytics and regulatory reporting. Deep knowledge of Spark, Kafka, Airflow, and data governance frameworks including CDC, lineage tracking, and modular orchestration. Early background in mobile and web development adds full-stack systems insight. Proven ability to lead cross-functional teams (20+), drive technical strategy, and align data platforms with business outcomes.
Overview
16
16
years of professional experience
Work History
AVP – Data Engineering & Architecture
Airtel Payments Bank
06.2022 - Current
- Led the migration of the entire on-prem data infrastructure to AWS cloud, transforming legacy systems into scalable, cloud-native services.
- Architected and deployed an enterprise-wide Data Mesh platform, enabling domain-oriented ownership and self-serve capabilities across verticals like UPI, Retailer, Merchant, and Finance.
- Designed and operationalized a centralized Data Warehouse and a governed Lakehouse architecture using AWS S3, Glue, Redshift, and Lake Formation.
- Enabled compliance and audit-readiness through metadata governance, lineage tracking, and data quality frameworks using Apache Atlas, Lake Formation, and Great Expectations.
- Established cross-functional collaboration practices, empowering data consumers with secure, scalable, and analytics-ready datasets.
- Led a team of 20+ data engineers, driving project delivery, mentoring, and promoting cloud-first engineering excellence.
SDE IV – Data Engineering
Airtel Payments Bank
09.2019 - 05.2022
- Worked primarily in an on-prem environment, designing and maintaining large-scale data pipelines for operational and analytical use cases.
- Created a robust Hybrid Data Lake Architecture using Apache Spark, Hive, Airflow, Cassandra, and Elasticsearch.
- Built 1TB+/day ingestion pipelines and orchestrated them with Apache Airflow.
- Designed CDC workflows from Oracle Flexcube using GoldenGate → Kafka → Spark → Redshift.
- Developed a scalable, multi-source Data Hub with searchable indexes using Elasticsearch and integrated metadata management.
- Laying the groundwork for AWS migration by standardizing data structures and pipeline modularization.
- Reduced SLA breaches by 30% through modular orchestration using Apache Airflow.
Senior Software Consultant – Full Stack & Data Engineering
AI Matt College Marketplace Pvt. Ltd.
04.2018 - 09.2019
- Developed scalable full-stack ed-tech platforms using Django, PostgreSQL, and MongoDB.
- Integrated analytics layer using Hadoop, Spark, and Kafka for student data insights.
- Delivered secure and scalable data workflows across systems.
Tech Lead – Big Data Platforms
HT Media Ltd.
08.2016 - 03.2018
- Led big data architecture efforts using Spark, Kafka, Hadoop, and Hive.
- Designed scalable pipelines for editorial and advertising analytics.
- Managed development of digital portals (e.g., HTCampus.com) using Python, Redis, MongoDB, MySQL, and XAMPP stack.
- Optimized real-time content delivery, caching, and backend APIs.
Senior Software Engineer
ReTech Labs, Inc.
02.2016 - 05.2016
- Developed microservices for a retail management platform using Django, Celery, Redis, MongoDB, and RESTful APIs.
- Improved API performance with background processing and caching.
Senior Software Engineer & Consultant
Teramatrix Technologies Pvt. Ltd.
08.2009 - 02.2016
- Joined Teramatrix Technologies full-time as a Senior Software Engineer in Dec 2013.
- From Aug 2009 to Nov 2013, worked as a freelance consultant for Teramatrix Technologies and other clients, delivering 35+ full-stack solutions using Django, Android/iOS, REST APIs, MongoDB, MySQL, and AWS EC2/S3.
- Focused on backend systems, dashboard development, API integrations, and cloud deployment across education, healthcare, and retail domains.
- Led the development of telecom network monitoring and IoT systems using Python, Django, MongoDB, MySQL, and Redis.
- Delivered real-time dashboards and alerting platforms, and participated in performance optimization and system architecture design.
- Implemented Redis caching for real-time performance and alerting.
Education
Bachelor of Technology (B.Tech) - Computer Science
Kurukshetra University
01.2009
Skills
- Cloud Platforms: AWS (S3, Glue, Athena, Redshift, EMR, MSK, Lambda, OpenSearch, Lake Formation), GCP
- Big Data: Apache Spark (batch & streaming), Hadoop, Kafka, Hive, Delta Lake, HDFS
- ETL & Orchestration: Apache Airflow, MWAA, Step Functions, AWS Glue Jobs
- Governance & Quality: Apache Atlas, Lake Formation, Great Expectations, DataHub
- Databases: Oracle, PostgreSQL, Redshift, Cassandra, MongoDB, MySQL
- Programming: Python (Expert), SQL, Java, Scala
- DevOps: Terraform, Kubernetes, Docker
- Security: IAM, KMS, CloudTrail, Macie, S3 Policies
Timeline
AVP – Data Engineering & Architecture
Airtel Payments Bank
06.2022 - Current
SDE IV – Data Engineering
Airtel Payments Bank
09.2019 - 05.2022
Senior Software Consultant – Full Stack & Data Engineering
AI Matt College Marketplace Pvt. Ltd.
04.2018 - 09.2019
Tech Lead – Big Data Platforms
HT Media Ltd.
08.2016 - 03.2018
Senior Software Engineer
ReTech Labs, Inc.
02.2016 - 05.2016
Senior Software Engineer & Consultant
Teramatrix Technologies Pvt. Ltd.
08.2009 - 02.2016
Bachelor of Technology (B.Tech) - Computer Science
Kurukshetra University