Kalyan Singh
Pune,Maharashtra
Summary
Data Engineer with over 5 years of experience in designing and optimizing scalable data pipelines in AWS and Azure environments. Proficient in PySpark, AWS Glue, Airflow, and SQL, with strong expertise in Snowflake, Starburst, and Azure Synapse. Delivered impactful data solutions and automated workflows, contributing to multiple award-winning projects.
Overview
5
5
years of professional experience
2
2
Certification
Work History
Data Engineer
TIAA Global Capabilities
Pune, Maharashtra
12.2023 - Current
- Built and managed data pipelines, leveraging AWS services (S3, Glue, Airflow, Lambda, SNS, SQS, DynamoDB, CloudWatch).
- Designed and implemented Airflow DAGs for orchestrating Glue jobs, improving automation, and reliability.
- Engineered complex DAG chains for seamless Snowflake to Starburst transfers via Glue Iceberg tables.
- Automated infrastructure provisioning and CI/CD workflows using Terraform, CloudBees, and Scalr.
- Collaborated in Agile teams using Jira for planning and delivery.
- Working on a utility to calculate the optimized Glue Job worker count for a particular data feed using historical run metrics.
Consultant & Associate Consultant
Capgemini India
Hyderabad
09.2021 - 11.2023
- Built scalable ETL pipelines using Spark and AWS technologies for consumer health analytics.
- Developed configuration-driven stored procedures for merging data streams with business-specific logic in Snowflake.
- Led the development of a critical analytics interface, optimizing Glue-Spark jobs for 10x performance gains.
- Migrated pipelines from Cloudera to AWS, modernizing frameworks, and improving robustness.
- Created reusable validation modules for over 200 interfaces; won the Capgemini Best Team Award.
Senior Analyst
Capgemini India
Bangalore
07.2018 - 08.2019
- Implemented data models in Azure Data Lake and SQL Data Warehouse for FMCG supply chain analytics.
- Managed Azure pipelines and Databricks workflows for critical modules.
- Collaborated with stakeholders to gather requirements and deliver robust solutions.
Education
B.Tech - Mechanical Engineering
NIT Raipur
06.2018
Skills
- Python
- SQL
- PySpark
- AWS Glue
- Airflow
- Starburst
- DynamoDB
- AWS Lambda
- AWS Step Functions
- AWS S3
- Athena
- SNS
- SQS
- CloudWatch
- Snowflake
- Azure Databricks
- Terraform
- CloudBees
- Scalr
- PyCharm
- GitLab
- SSMS
- MS Excel
Certification
- AWS Certified Data Engineer - Associate
- AWS Certified Cloud Practitioner
- Linux command line (Udemy)
- Snowflake Badge 1: Data Warehousing Workshop
- Snowflake Badge 3: Sharing, Marketplace, and Exchanges
Accomplishments
- Silver Award - Team Performance, 2025
- XtraMile Award for performance in CH Datalake Project, 2021-2022
- Best client value creator and best team award, 2021-2022
- Rising Star Award, 2018-2019
- 4th position, ISNEE QUAD-TORC, 2016, team captain, NIT Raipur
- 2nd position, VIGYAAN National Science Exhibition, 2016, NIT Raipur
Personal Information
Title: Data Engineer
Work Availability
monday
tuesday
wednesday
thursday
friday
saturday
sunday
morning
afternoon
evening
swipe to browse
Timeline
Data Engineer
TIAA Global Capabilities
12.2023 - Current
Consultant & Associate Consultant
Capgemini India
09.2021 - 11.2023
Senior Analyst
Capgemini India
07.2018 - 08.2019
B.Tech - Mechanical Engineering
NIT Raipur