Rajesh Patil
GCP Lead Data Engineer/Senior Data Engineer/ Data Architect
Pune
Summary
Seasoned and results-driven Data Architect / Senior Data Engineer with deep expertise in building cloud-native data platforms, modernizing legacy systems, and designing scalable ETL/ELT pipelines. Proven, hands-on experience across Google Cloud Platform (BigQuery, Dataflow, Cloud Composer, Airflow) and advanced data warehousing technologies such as Snowflake . Successfully led multiple enterprise-scale migration programs, including Teradata and SQL Server to BigQuery , delivering high-performance, cost-optimized, and fully automated data processing ecosystems.
Strong background in Markit EDM for mastering reference, pricing, and security data, implementing data quality rules, golden-copy logic, and integration with downstream analytics and reporting systems. Adept at architecting end-to-end data solutions, optimizing performance through partitioning, clustering, orchestration, and pipeline tuning while ensuring robust governance, reliability, and audibility.
Recognized as an analytical, client-focused professional with experience collaborating directly with stakeholders onsite in the UK and USA . Known for developing scalable data frameworks, enhancing data capabilities, and enabling business intelligence outcomes through strong data modeling, data integration, metadata management, and architecture design . Skilled at leading cross-functional teams, driving cloud transformation initiatives, and adapting quickly to dynamic business and technology landscapes.
Overview
16
16
years of professional experience
3
3
Certifications
Work History
Data Architect/Lead Data Engineer
Atos
10.2020 - Current
- Led Teradata and SQL Server to BigQuery migration initiatives, modernizing legacy data warehouse ecosystems into scalable, cloud-native architectures on Google Cloud Platform (GCP) .
- Spearheaded complete migration lifecycle— assessment, schema conversion, SQL rewrite, ETL/ELT modernization, performance tuning, validation, and cutover —ensuring zero business downtime and seamless transition.
- Designed and implemented high-volume ETL/ELT pipelines using Dataflow, BigQuery, and Python, enabling reliable movement and transformation of structured, semi-structured, and unstructured datasets.
- Developed scalable Cloud Composer (Apache Airflow) workflows for orchestration of ingestion, validation, transformation, and load jobs, reducing manual effort and achieving fully automated, event-driven pipelines .
- Built fault-tolerant, resilient pipelines with advanced error handling, retries, alerting, and logging, ensuring reliability compliant with enterprise data governance and SLAs.
- Engineered optimized batch and streaming pipelines using Google Dataflow (Apache Beam), leveraging autoscaling, parallelization, and worker tuning to improve throughput and reduce processing latency.
- Created custom Dataflow transforms and reusable components to support complex business rules, ensuring modular, maintainable, and scalable data pipelines.
- Migrated and optimized complex Teradata BTEQ, TPT, T-SQL, stored procedures, macros, and analytical queries into BigQuery Standard SQL , improving performance and simplifying logic.
- Implemented BigQuery partitioning, clustering, column-level tuning, query optimization, table decorators , and materialized views to reduce costs and enable sub-second analytics on massive datasets.
- Designed modernized semantic layers, star/snowflake models, and data marts , enabling high-performance BI & analytics for business stakeholders.
- Enhanced reporting performance by optimizing execution plans, restructuring transformations, and eliminating technical debt accumulated in legacy platforms.
- Collaborated closely with data analysts, data scientists, product owners, platform engineers, and cloud architects to define migration roadmaps, SLAs, and target state architectures.
- Led multi-disciplinary engineering teams through phased migration execution , task prioritization, workload planning, dependency mapping, and risk mitigation.
- Conducted stakeholder workshops on migration scope, data quality, regression validation, and change management to ensure business alignment.
- Designed and executed comprehensive data validation & reconciliation frameworks , including record counts, checksums, referential integrity, schema validation, and BI regression testing to ensure 100% data fidelity .
- Ensured post-migration business continuity by validating dashboards, KPIs, metrics, and source-to-target mappings across functional areas.
- Established coding standards, peer-review guidelines, and CI/CD practices for Dataflow, SQL, and Airflow deployments; mentored junior engineers in best practices and cloud-native engineering.
- Conducted knowledge-sharing and training sessions on GCP services, data warehousing patterns, migration best practices, Beam pipelines, and BigQuery optimizations , enhancing team competency and platform maturity.
Senior Data Engineer – UK Onsite Assignment
Syntel Europe Ltd - Acquired by Atos
03.2014 - 09.2020
- Designed, implemented, and managed enterprise-grade data pipelines across front, middle, and back-office functions, leveraging Apache Airflow for orchestration and Snowflake as the centralized cloud data warehouse.
- Utilized Markit EDM to master and consolidate reference, pricing, and security master data from multiple market data vendors (Bloomberg, Cit., Refinitiv), implementing golden copy logic , validation workflows, survivorship rules, and automated exception management.
- Architected a scalable and cost-optimized Snowflake data warehouse , applying best practices such as micro-partitioning, clustering keys, pruning, result caching, zero-copy cloning, and time travel to enhance performance and audibility.
- Developed automated, parameter-driven Airflow DAGs to ingest positions, trades, benchmarks, factors, and market data, incorporating SLA monitoring, alerting, retry logic, dynamic task mapping, and observability dashboards.
- Partnered with front-office quanta, risk teams, performance analysts, and data governance stewards to deliver governed, high-quality datasets aligned with data mesh principles , regulatory frameworks (MiFID II, SFTR, ESG, GDPR), and enterprise data standards.
- Played a key role in migrating legacy on-prem warehouse systems (Oracle/Teradata/SQL Server) to cloud-native Snowflake , improving data freshness from T+1 to near real-time/intraday , reducing platform cost by 30% , and accelerating reporting SLA by 40% .
- Reengineered legacy ETL workflows (Python, SQL, Informatica/Talend scripts) to eliminate performance bottlenecks, resulting in improved pipeline efficiency, reduced runtime, and enhanced system stability.
- Ensured end-to-end data quality through automated validation, reconciliation checks, schema enforcement, metadata-driven controls, and continuous monitoring, significantly minimizing inconsistencies and operational risk.
- Implemented robust data lineage, audit logging, and governance frameworks to support compliance, traceability, and transparency across critical investment, risk, and trading datasets.
Automation Consultant
Syntel India - Acquired by Atos
01.2010 - 01.2013
- Built an automated regression testing framework for a UI-based payment platform for American Express, ensuring high-quality deliverables through effective test coverage.
- Collaborated with cross-functional teams to successfully deliver comprehensive solutions for clients.
Education
Bachelor of Engineering - E & TC
University of Pune
Pune, India 04.2001 -
Skills
Experienced in developing solutions on Google Cloud Platform
Certification
GCP Data Engineer
Timeline
Data Architect/Lead Data Engineer
Atos
10.2020 - Current
Senior Data Engineer – UK Onsite Assignment
Syntel Europe Ltd - Acquired by Atos
03.2014 - 09.2020
Automation Consultant
Syntel India - Acquired by Atos
01.2010 - 01.2013
Bachelor of Engineering - E & TC
University of Pune
04.2001 -