Summary
Overview
Work History
Education
Skills
Timeline
Generic
Ankit Pandey

Ankit Pandey

Pune

Summary

Data Engineer with over 11 years of experience in developing scalable, cloud-native data systems on AWS and GCP. Expertise in modern data platforms including DBT, Snowflake, Airflow, and Terraform, complemented by strong programming skills in Python, Scala, and Apache Spark. Focused on delivering efficient data pipelines and reliable data products that enhance analytics and support decision-making processes.

Overview

11
11
years of professional experience

Work History

Senior Data Engineer

Equal Experts
Pune
07.2023 - Current
  • Building a modern, cloud-native data platform on AWS for one of the largest sports leagues in the U.S., integrating diverse data sources into a centralized data layer, and delivering curated data products for Marketing, Analytics, and other downstream teams.
  • Developed scalable ingestion pipelines using AWS Glue, Lambda, SQS, and Kinesis, supporting both batch and streaming data.
  • Designed and implemented the transformation layer using DBT on Redshift, orchestrated via Airflow, to serve trusted datasets for audience segmentation, campaign performance, and fan engagement analytics.
  • Contributed to establishing data quality standards and conventions within the team, promoting test coverage, documentation, and governance across DBT models.
  • Collaborated with cross-functional teams to translate business needs into reliable, production-grade data assets, ensuring quality, performance, and governance through CI/CD, and Terraform automation.

Lead Data Engineer

Clairvoyant EXL
Pune
05.2021 - 07.2023
  • Create optimal data pipeline architecture and systems by assembling large, complex data sets that meet functional and business requirements on AWS Cloud using EMR, EC2, Glue, Athena, Kinesis Firehose, Step Functions, Lambda, CloudWatch, etc.
  • Ingest data into the data lake, and develop custom frameworks for ELT, ETL, and services for operating on that data with the use of PySpark and Scala Spark. Pandas, SQL, etc.
  • Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, redesigning infrastructure for greater scalability, etc.
  • Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using Python, Scala, Spark, AWS, Terraform, CI/CD, etc.

Technology Analyst

Edgeverve Systems
Dubai, Pune
05.2014 - 05.2021
  • Part of the digital transformation and modernization of one of the largest financial institutions in the Middle East for building a scalable core data platform.
  • Built generic frameworks for data ingestion, data transformation using Python and Scala by implementing the TDD approach.
  • Worked on the enhancement of data quality and existing ETL flows for legacy frameworks.
  • Write complex Spark jobs for processing huge volumes of data.
  • Customize traditional data management practices of data quality and data governance for the adoption of AWS Cloud.

Education

Bachelor of Engineering -

Madhav Institute of Technology And Science
Gwalior, MP
04-2014

Skills

  • AWS
  • GCP
  • Python
  • Scala
  • DBT
  • Snowflake
  • Terraform
  • Airflow
  • SQL
  • CICD

Timeline

Senior Data Engineer

Equal Experts
07.2023 - Current

Lead Data Engineer

Clairvoyant EXL
05.2021 - 07.2023

Technology Analyst

Edgeverve Systems
05.2014 - 05.2021

Bachelor of Engineering -

Madhav Institute of Technology And Science

Similar Profiles

CREATE PROFILE
Ankit Pandey