VIJAY KRISHNAN K
Chennai
Summary
Experienced Data Engineer with over 9 years in developing scalable data platforms and distributed pipelines within AWS and cloud-native environments. Specializes in PySpark, Apache Spark, Snowflake, and Databricks for efficient ETL/ELT processes and cloud migration. Focused on designing data architectures and enhancing analytics workloads to drive data reliability and insights.
Overview
1
1
Language
9
9
years of professional experience
Work History
Technical Lead
Sopra Steria | GEODIS
04.2026 - 10.2026
- Managed batch data workflows with pyspark for delta lake to apache iceberg migration, leveraging unity catalog for centralized governance, schema management, and secure access to enterprise data.
- Executed end-to-end data pipelines with astronomer airflow, automating databricks job execution while implementing monitoring, retries, and dependency handling.
- Built pyspark etl pipelines with databricks jobs to load data from databricks volumes mapped to azure blob storage into unity catalog-managed apache iceberg tables.
Technical Lead
Sopra Steria | NatWest
12.2023 - 03.2026
- Built near real-time ingestion pipelines from AWS S3 and Kafka into Snowflake to support faster fraud detection.
- Optimized Snowflake queries and clustering strategies, increasing analytics performance by ~35%.
- Maintained Apache Airflow workflows, enhancing pipeline reliability and reducing failures by ~30%.
- Developed secure cross-domain data sharing solutions in Snowflake, facilitating collaboration across multiple business units.
- Supported NatWest assignment by building high-throughput ETL pipelines that processed millions of fraud records daily with PySpark and Snowpark.
Module Lead
Sopra Steria | EDM
11.2021 - 06.2023
- Led team of 4-5 QA specialists and developers to successfully deliver FY23 priorities on time.
- Built AWS Glue ETL pipelines to process large-scale financial datasets for the Sopra Steria Ethical Debt Management product with high data accuracy.
- Automated batch ingestion workflows, significantly reducing manual effort and enhancing overall pipeline efficiency.
- Implemented data validation frameworks to improve data quality and reduce inconsistencies by ~20%.
- Developed relational data models and SQL procedures that directly supported Tableau dashboards for key business stakeholders.
Software Developer
Tata Consultancy Services | Client: Nielsen
08.2017 - 10.2021
- Created distributed Spark pipelines using Java, Spark, and SQL to process large-scale media datasets.
- Designed and developed STB application by rewriting the existing system into a Spark-based platform using Java/Spark and Scala/Spark.
- Built complex calculations in Java/Spark to meet or exceed SLAs already set by legacy systems.
- Migrated legacy analytics platforms to AWS, enhancing scalability and reducing infrastructure costs.
- Assisted with migration of enterprise data from Netezza to AWS cloud ecosystem, enabling scalable data processing.
- Automated end-to-end workflows with Apache Airflow to ensure reliable, scheduled data processing.
Education
Bachelor of Engineering (B.E) -
SRM Easwari Engineering College
ChennaiSkills
- Big data engineering
- Cloud computing
- Data streaming
- Database technologies
- Data quality frameworks
- Java and Python
- Technical proficiency
- Integrated development environments
- Version control systems
- Process automation
- Data pipeline development
- Migration development
Personal Information
Title: Senior Data Engineer
AWARDS
- Team Excellence Award – NatWest: Recognized by the client for exceptional contribution, timely project delivery, and consistently high-quality deliverables.
- Spot Team Award – EDM Project: Recognized for outstanding contribution and effective collaboration in successfully delivering key EDM project initiatives.
- TCS Star of the Month: Awarded for strong technical contribution, ownership, and consistent delivery excellence.
- TCS On-the-Spot Award: Recognized for exceptional performance, timely execution, and valuable contributions to project success.
- Client Management Appreciation Award: Received multiple appreciations from senior client management for technical excellence, ownership, and successful delivery of critical initiatives.
- Client Recognition – NatWest: Recognized by client management for delivering high-quality solutions and demonstrating strong commitment to project objectives.
Languages
English
Proficient (C2)
C2
Timeline
Technical Lead
Sopra Steria | GEODIS
04.2026 - 10.2026
Technical Lead
Sopra Steria | NatWest
12.2023 - 03.2026
Module Lead
Sopra Steria | EDM
11.2021 - 06.2023
Software Developer
Tata Consultancy Services | Client: Nielsen
08.2017 - 10.2021
Bachelor of Engineering (B.E) -
SRM Easwari Engineering College