Summary
Overview
Work History
Education
Skills
Certification
Timeline
Generic

Rohan P

Data Engineer
Bengaluru

Summary

Data Engineer with 6 years of experience in financial services and investment banking, specializing in Spark, Pyspark, Snowflake, AWS, and modern data platforms. Proven expertise in building scalable ETL/ELT pipelines, real-time streaming systems, and reusable frameworks that accelerate analytics and machine learning adoption. Skilled in data quality engineering, performance optimization, and orchestration (Airflow, Autosys, Kafka) with strong domain experience in fraud detection, trading, risk, compliance, and customer analytics.

Overview

6
6
years of professional experience
1
1
Certification

Work History

Data Engineer

Commonwealth Bank of Australia
07.2022 - Current
  • Spark Development Enablement Framework
  • Built a framework for rapid Spark pipeline development on Cloudera Hadoop with Hive, supporting 100+ pipelines handling terabytes of structured and semi-structured data.
  • Provided reusable modules, schema evolution handling, and partitioning strategies, improving developer productivity.
  • Reduced pipeline development time by ~60%, accelerating delivery of ML-ready and analytics datasets for fraud detection, customer insights, and operational reporting.
  • On-Prem to Snowflake Data Replication
  • Developed Ab Initio pipelines on AWS Lambda to replicate datasets from on-prem Hive to Snowflake, orchestrating 60+ daily jobs migrating 200–300 GB/day.
  • Created an automated data testing framework for schema validation, record-level checks, and reconciliation, ensuring high trust and reliability.
  • Delivered validated datasets powering regulatory reporting, fraud analytics, and ML pipelines, reducing manual reconciliation effort.
  • Snowflake Transformation & Outbound Framework
  • Developed a config-driven transformation framework in Snowflake with stored procedures using Python and Shell scripting.
  • Processed ML model results for fraud detection, offer eligibility, customer scam susceptibility, and investment analytics, applying calibration rules and egressing curated data to AWS S3.
  • Orchestrated 40+ workflows in Airflow and implemented CI/CD pipelines with Terraform and GitHub Actions for environment consistency.
  • Self-Serve Feature Engineering Framework (DBT + Snowflake)
  • Built a self-service framework with DBT on Snowflake, enabling data scientists to independently design and manage feature transformations.
  • Designed the framework to be compute agnostic (Snowflake + Redshift), providing flexibility across business units.
  • Reduced feature delivery cycles by ~70%, accelerating ML feature creation for fraud detection, personalization, and risk analytics.

Senior Engineer – Data Engineering

Mindtree
07.2019 - 07.2022
  • Market EDM – Golden Source Data Pipelines
  • Built ETL pipelines in Market EDM consolidating golden copy data for market prices, positions, equities, and hedge funds from multiple upstream sources.
  • Developed PySpark/Hadoop workflows to process large-scale financial datasets, building a data warehouse for trading insights, analytics, and compliance reporting.
  • Implemented real-time Kafka streaming pipelines for ingesting market feeds, ensuring low-latency data delivery to trading desks, risk, and compliance teams.
  • Automated batch workflows using Autosys and applied Spark performance optimizations (partitioning, caching, cluster tuning) to improve throughput.
  • Created unit testing and data validation frameworks for schema checks, reconciliation, and data quality assurance, improving trust in downstream systems.

Education

Bachelor of Engineering - Computer Science

Nagarjuna College of Engineering & Technology
01.2019

Skills

Programming & Scripting: Python, Scala, SQL, Shell, PySpark

Big Data & Processing: Spark, Hadoop, Hive, Kafka

Cloud & Storage: AWS (S3, Lambda, EMR, Glue, Athena, Redshift), Snowflake

Data Warehousing: Snowflake, MySQL

undefined

Certification

AWS Certified Data Engineer – Associate (Completed)

Timeline

Data Engineer

Commonwealth Bank of Australia
07.2022 - Current

Senior Engineer – Data Engineering

Mindtree
07.2019 - 07.2022

Bachelor of Engineering - Computer Science

Nagarjuna College of Engineering & Technology

Similar Profiles

CREATE PROFILE
Rohan PData Engineer