Summary
Overview
Work History
Education
Skills
Certification
Timeline
Generic

MANISH GUPTA

Bengaluru

Summary

Principal Engineer with over 13 years of comprehensive experience, including 3 years dedicated to Core Data Engineering and a robust 10-year foundation in Software Engineering and Machine Learning. Self taught, up skilled & proved in building scalable data pipelines, optimizing ETL workflows, and designing cloud-based big data solutions. Real hands-on in Data Modeling, Columnar Storage, Apache Spark, Kafka, Airflow, Snowflake, BigQuery, AWS Glue, and real-time streaming. Passionate about driving innovation, data governance, automation, and distributed computing.

Overview

15
15
years of professional experience
1
1
Certification

Work History

Principal Engineer

Square Shift, India
04.2022 - Current

Project: Learning Portal (Client: CSOD)

  • Business. Impact

1. Led & developed core Data ETL from application to

Datalake for looker based dashboards & to power UI based reports by hosting rollup data to elastic search.

2 . Major Business tasks accomplished was revamp ETL based on multi-tenant to single-tenant DB architecture using AWS Glue, DBT, and BigQuery, Developed an event-driven pipeline for real-time leaderboard score computation with a 1-hour NRT, Delivered Asynchronous & distributed data deletion framework based on Python SDK for decommissioned business orgs across AWS & GCP, automating 100% of compliance to comply GDPR.

3. Minor enhancements delivering multiple customer requested features wrt establishing new ETL from scratch to power AI tools or consumed data in downstream tools like workato .

  • Tool Automation & Performance Impact

1. Real-time monitoring framework for CDC connector health , automate ETL data synchronization validation, storing insights in BigQuery for daily Tableau reporting.

2. Apache Airflow orchestration with dynamic job scheduling, reducing pipeline delays, cutting processing time by 50%, accelerating analytics for business users.

3. Optimized resources (BigQuery, Cloud Run, Cloud Functions) post analysis from insights in execution & memory pressure , requests throtling

4. Performed Elastic search optimization by migrating excess shards data over 500 GB to multiple shards using cluster reindexing and reduce bottleneck for data ingestion.

5. Created Export , Import jobs to migrate data from Production to lower environment for data evaluation.

  • Leadership & Mentoring

1. Led architecture discussions with senior management, Business analysts , provided technical guidance to teams, followed developed early & fail fast approach , scheduled pilots demo early to get business suggestions and make robust delivery.

2. Led a 4-member team, mentoring juniors, help to fast track & bring on the project speed .

Project lead / Senior Applications Engineer

Oracle India
04.2014 - 03.2022

Project: CRM Sales Fusion Application

Product Development

  • Led backend development for CRM SaaS applications, systems architectures, development of UI to backend Model in JAVA (UI, RESTful) from designing to code to automation.
  • Designed REST APIs, micro-services, and high-performance data models
  • Followed Model and Test driven development AMDD/TDD.
  • Worked with 50+ enterprise clients in troubleshooting and resolving production issues.
  • Brought reduction over 25% in key Performance clicks in the application by analyzing JFR, SQLs, Eclipse Memory Analyzer and optimizing the code

Senior Systems Engineer

Infosys
03.2011 - 03.2014
  • Worked as a developer for a RBS Relationship Manager platform (banking domain), delivered bank application model service and UI in java for 2 years.
  • Retail domain and worked on developing a social collaboration suite for Retail giant Darden for 1 year.
  • Trained in Infosys Mysore Campus in Computer fundamentals, algorithm design, data structures with
  • Specialization in Java/J2EE with CGPA 4.81/5 in 6 month

Education

PG - Data Analytics

IIIT Bengaluru
08.2017

B.Tech - Information technology

College of Engineering Roorkee
08.2010

Skills

  • Big Data & Streaming: Apache Spark, Kafka, Flink, Hadoop
  • Databases: Snowflake, BigQuery, MySQL, PostgreSQL, Elasticsearch, Deltalake
  • Cloud Platforms: GCP (BigQuery, Pub/Sub, Cloud Run, GCS), AWS (Glue, Redshift, Lambda)
  • Programming: Python, SQL, Java
  • ETL & Orchestration: Apache Airflow, dbt, AWS Glue
  • DevOps & CI/CD: Docker, Kubernetes, Jenkins, Terraform

Certification

  • Oracle Java Certified Professional
  • Google cloud certified Data Professional & Architect

Timeline

Principal Engineer

Square Shift, India
04.2022 - Current

Project lead / Senior Applications Engineer

Oracle India
04.2014 - 03.2022

Senior Systems Engineer

Infosys
03.2011 - 03.2014

B.Tech - Information technology

College of Engineering Roorkee

PG - Data Analytics

IIIT Bengaluru
MANISH GUPTA