Rajesh Patil
GCP Lead Data Engineer/Senior Data Engineer/ Data Architect
Pune
Summary
Seasoned and results-driven Data Architect / Senior Data Engineer with deep expertise in building cloud-native data platforms, modernizing legacy systems, and designing scalable ETL/ELT pipelines. Proven, hands-on experience across Google Cloud Platform (BigQuery, Dataflow, Cloud Composer, Airflow) and advanced data warehousing technologies such as Snowflake . Successfully led multiple enterprise-scale migration programs, including Teradata and SQL Server to BigQuery , delivering high-performance, cost-optimized, and fully automated data processing ecosystems.
Strong background in Markit EDM for mastering reference, pricing, and security data, implementing data quality rules, golden-copy logic, and integration with downstream analytics and reporting systems. Adept at architecting end-to-end data solutions, optimizing performance through partitioning, clustering, orchestration, and pipeline tuning while ensuring robust governance, reliability, and audibility.
Recognized as an analytical, client-focused professional with experience collaborating directly with stakeholders onsite in the UK and USA . Known for developing scalable data frameworks, enhancing data capabilities, and enabling business intelligence outcomes through strong data modeling, data integration, metadata management, and architecture design . Skilled at leading cross-functional teams, driving cloud transformation initiatives, and adapting quickly to dynamic business and technology landscapes.
Overview
16
16
years of professional experience
3
3
Certifications
Work History
Data Architect/Lead Data Engineer
Atos
10.2020 - Current
- Led Teradata and SQL Server to BigQuery migration initiatives, modernizing legacy data warehouse ecosystems into scalable, cloud-native architectures on Google Cloud Platform (GCP) .
- Spearheaded complete migration lifecycle— assessment, schema conversion, SQL rewrite, ETL/ELT modernization, performance tuning, validation, and cutover —ensuring zero business downtime and seamless transition.
- Designed and implemented high-volume ETL/ELT pipelines using Dataflow, BigQuery, and Python, enabling reliable movement and transformation of structured, semi-structured, and unstructured datasets.
- Developed scalable Cloud Composer (Apache Airflow) workflows for orchestration of ingestion, validation, transformation, and load jobs, reducing manual effort and achieving fully automated, event-driven pipelines .
- Built fault-tolerant, resilient pipelines with advanced error handling, retries, alerting, and logging, ensuring reliability compliant with enterprise data governance and SLAs.
- Engineered optimized batch and streaming pipelines using Google Dataflow (Apache Beam), leveraging autoscaling, parallelization, and worker tuning to improve throughput and reduce processing latency.
- Created custom Dataflow transforms and reusable components to support complex business rules, ensuring modular, maintainable, and scalable data pipelines.
- Migrated and optimized complex Teradata BTEQ, TPT, T-SQL, stored procedures, macros, and analytical queries into BigQuery Standard SQL , improving performance and simplifying logic.
- Implemented BigQuery partitioning, clustering, column-level tuning, query optimization, table decorators , and materialized views to reduce costs and enable sub-second analytics on massive datasets.
- Designed modernized semantic layers, star/snowflake models, and data marts , enabling high-performance BI & analytics for business stakeholders.
- Enhanced reporting performance by optimizing execution plans, restructuring transformations, and eliminating technical debt accumulated in legacy platforms.
- Collaborated closely with data analysts, data scientists, product owners, platform engineers, and cloud architects to define migration roadmaps, SLAs, and target state architectures.
- Led multi-disciplinary engineering teams through phased migration execution , task prioritization, workload planning, dependency mapping, and risk mitigation.
- Conducted stakeholder workshops on migration scope, data quality, regression validation, and change management to ensure business alignment.
- Designed and executed comprehensive data validation & reconciliation frameworks , including record counts, checksums, referential integrity, schema validation, and BI regression testing to ensure 100% data fidelity .
- Ensured post-migration business continuity by validating dashboards, KPIs, metrics, and source-to-target mappings across functional areas.
- Established coding standards, peer-review guidelines, and CI/CD practices for Dataflow, SQL, and Airflow deployments; mentored junior engineers in best practices and cloud-native engineering.
- Conducted knowledge-sharing and training sessions on GCP services, data warehousing patterns, migration best practices, Beam pipelines, and BigQuery optimizations , enhancing team competency and platform maturity.
Senior Data Engineer – UK Onsite Assignment
Syntel Europe Ltd - Acquired by Atos
03.2014 - 09.2020
- Designed, implemented, and managed enterprise-grade data pipelines across front, middle, and back-office functions, leveraging Apache Airflow for orchestration and Snowflake as the centralized cloud data warehouse.
- Utilized Markit EDM to master and consolidate reference, pricing, and security master data from multiple market data vendors (Bloomberg, Cit., Refinitiv), implementing golden copy logic , validation workflows, survivorship rules, and automated exception management.
- Architected a scalable and cost-optimized Snowflake data warehouse , applying best practices such as micro-partitioning, clustering keys, pruning, result caching, zero-copy cloning, and time travel to enhance performance and audibility.
- Developed automated, parameter-driven Airflow DAGs to ingest positions, trades, benchmarks, factors, and market data, incorporating SLA monitoring, alerting, retry logic, dynamic task mapping, and observability dashboards.
- Partnered with front-office quanta, risk teams, performance analysts, and data governance stewards to deliver governed, high-quality datasets aligned with data mesh principles , regulatory frameworks (MiFID II, SFTR, ESG, GDPR), and enterprise data standards.
- Played a key role in migrating legacy on-prem warehouse systems (Oracle/Teradata/SQL Server) to cloud-native Snowflake , improving data freshness from T+1 to near real-time/intraday , reducing platform cost by 30% , and accelerating reporting SLA by 40% .
- Reengineered legacy ETL workflows (Python, SQL, Informatica/Talend scripts) to eliminate performance bottlenecks, resulting in improved pipeline efficiency, reduced runtime, and enhanced system stability.
- Ensured end-to-end data quality through automated validation, reconciliation checks, schema enforcement, metadata-driven controls, and continuous monitoring, significantly minimizing inconsistencies and operational risk.
- Implemented robust data lineage, audit logging, and governance frameworks to support compliance, traceability, and transparency across critical investment, risk, and trading datasets.
Automation Consultant
Syntel India - Acquired by Atos
01.2010 - 01.2013
- Built an automated regression testing framework for a UI-based payment platform for American Express, ensuring high-quality deliverables through effective test coverage.
- Collaborated with cross-functional teams to successfully deliver comprehensive solutions for clients.
Education
Bachelor of Engineering - E & TC
University of Pune
Pune, India 04.2001 -
Skills
Experienced in developing solutions on Google Cloud Platform
Google BigQuery expertise
Experienced with Cloud Dataflow
Team leadership skills
Effective stakeholder communication
Proficient in Python,SQL
Experience with Apache Airflow
Big data processing
Data security
Data governance
Collaborative team development
Cross-functional teamwork
Certification
GCP Data Engineer
Timeline
Data Architect/Lead Data Engineer
Atos
10.2020 - Current
Senior Data Engineer – UK Onsite Assignment
Syntel Europe Ltd - Acquired by Atos
03.2014 - 09.2020
Automation Consultant
Syntel India - Acquired by Atos
01.2010 - 01.2013
Bachelor of Engineering - E & TC
University of Pune
04.2001 -