Priyaranjan Dubey
Gurgaon
Summary
Over 13 years of experience in data engineering with a focus on developing scalable cloud data platforms utilizing Snowflake and modern ELT tools across various sectors including insurance, healthcare, pharmaceuticals, and finance. Expertise in designing comprehensive data pipelines with Informatica Intelligent Cloud Services, dbt, and AWS architectures. Proven ability in data modeling, performance optimization, and building robust ETL/ELT frameworks, including SCD2-based dimensional models and cloud migration strategies. Demonstrated leadership in mentoring teams and delivering impactful data solutions for analytics and regulatory reporting, with strong skills in Python, SQL, and big data technologies such as Hadoop and Spark.
Overview
14
14
years of professional experience
1
1
Certification
Work History
Lead Data Engineer
EXL (Clairvoyant)
05.2023 - Current
- Designed end-to-end Snowflake data warehouse solution consolidating policy and claims data from multiple source systems.
- Designed and implemented a Snowflake-based regulatory data warehouse platform to support MCAS (Market Conduct Annual Statements) reporting across multiple business entities.
- Led ingestion of client data into Snowflake and transformation within Medallion architecture, aimed at providing comprehensive view of Unum’s clients.
- Designed and developed Medallion architecture (Bronze–Silver–Gold layers) for customer data in Snowflake.
- Implemented incremental processing using Snowflake Streams & Tasks, enabling near real-time data updates.
- Architected IICS based ingestion frameworks for Salesforce data into Snowflake.
- Built IICS-based ingestion framework for incremental data loads from various systems into Snowflake.
- Implemented semi-structured data ingestion (JSON) using Snowflake Streams, Tasks, and Views.
- Designed and developed data pipelines with IBM CDC, Kafka, and Kafka Snowflake sync connector to facilitate real-time raw data ingestion from DB2 to Snowflake.
- Designed Snowsql scripts for transformation of raw data to enterprise data warehouse tables in Snowflake.
- Designed and developed Snowsql scripts and Snowpark procedures to process the raw data to Silver layer and further aggregated data to Gold layer.
- Designed SCD Type 2 dimensional model to maintain full historical compliance data for regulatory reporting.
- Optimized historical data processing by consolidating multiple SCD2 datasets.
- Built metadata-driven data validation and audit framework in Python, ensuring reusable and scalable data quality checks across pipelines.
- Led design discussions with stakeholders and mentored junior engineers.
- Developed reusable audit scripts to track data completeness.
- Developed recursive IICS taskflows to handle large Salesforce objects in smaller batches.
- Facilitated issue resolution and conducted knowledge transfer sessions for the run team to ensure effective pipeline maintenance.
Lead Data Engineer
Presidio Cloud Solutions
08.2021 - 05.2023
- Led development of cloud-based analytics platform on Snowflake to enable global supply chain visibility and reporting.
- Designed Snowflake-based reporting architecture consuming data from Presto and multiple upstream systems.
- Developed complex Snowflake Views and Stored Procedures for global supply chain analytics (GMS & SIMS).
- Built orchestration logic using Snowflake tasks and procedures for dependent data pipelines.
- Implemented incremental load strategies using IICS with parameter-driven workflows.
- Designed DBT based modular architecture for data warehouse and devised AWS pipelines to ingest cloudwatch logs into the datawarehouse.
- Migrated legacy SQL transformations to DBT-based modular architecture.
- Built AWS-based ingestion pipelines using Lambda, Glue (PySpark), and S3.
- Facilitated scalable ingestion of CloudWatch logs into PostgreSQL, enhancing downstream analytics capabilities.
- Developed Postgres functions to transform dumped data into staging tables, streamlining data processing.
- Created HTML-based automated notification framework for pipeline monitoring and reporting.
- Refined data models to enhance query performance for Tableau dashboards, improving visualization efficiency.
- Mentored team members and led design discussions with business stakeholders.
Senior Software Engineer
Assimilate Solutions
03.2020 - 08.2021
- Led enterprise-wide data migration from legacy MS SQL systems to Snowflake cloud platform, ensuring seamless transition and data integrity.
- Migrated legacy SQL Server data warehouse to Snowflake using Data Vault 2.0 architecture.
- Built ingestion pipelines from APIs, flat files, and databases using IICS and Snowflake.
- Designed task-based orchestration pipelines in Snowflake for automated data processing.
- Streamlined bulk data loading using Snowflake stages and COPY commands, enhancing data processing capabilities.
- Developed stored procedures in Snowflake for efficient loading of type 1 (latest records) and type 2 (historical records) tables.
- Developed Python utilities for preprocessing input data and automation.
Consultant
Capgemini India.
07.2018 - 03.2020
- Engineered REST API integrations with Informatica IICS (REST V2 connectors) to streamline data flow between systems.
- Designed and delivered API-based integration platform between Veeva CRM and YOOBIC systems.
- Designed IICS mappings error handling and retry mechanisms between Salesforce-based systems and external APIs.
- Created Postgres-based staging and transformation layer for efficient integration processing.
- Implemented image data handling using base64 decoding and secure storage pipelines.
- Enhanced enterprise data warehouse ETL pipelines supporting Warner Bros. marketing analytics platform.
- Redesigned ETL framework by implementing control table–driven incremental loading, replacing variable-based logic for improved scalability.
- Built data reconciliation and validation mechanisms to ensure accuracy and completeness of daily data mart loads.
- Developed automated monitoring and alerting framework (UNIX + HTML email notifications) for proactive failure detection.
- Partnered with onsite teams for requirement analysis and enhancements, ensuring successful end-to-end deployment of production-ready ETL solutions.
Senior Software Engineer
Mphasis Ltd.
09.2012 - 07.2018
- Developed ETL pipelines for large-scale insurance claims and membership data systems.
- Optimized SQL and Informatica workflows for performance tuning and batch processing efficiency.
- Implemented SCD Type 1 and Type 2 frameworks for historical data tracking.
- Supported production deployments and resolved critical performance bottlenecks.
- Worked on enterprise-scale Informatica PowerCenter-based data warehouse systems.
- Built UNIX-based automation scripts for workflow orchestration.
Education
Bachelor of Engineering -
Sri Siddhartha Institute of Technology (VTU)
TumkurSkills
- Snowflake
- AWS services
- ETL/ELT Architecture
- Data Warehousing
- Data Vault 20
- SCD Type 1/2
- SQL (Advanced)
- SnowSQL
- PL/SQL
- Python
- Shell Scripting
- IICS
- DBT
- Cloud Databases
- PostgreSQL
- MySQL
- Workflow Automation
- UNIX/Linux
- REST APIs
- Tableau support
Certification
- SnowPro Advanced: Data Engineer (January 2025)
- SnowPro Core (November 2020, renewed January 2023)
- Informatica Cloud Data and Application Integration, Specialist (August 2020)
- Oracle Database SQL Expert (December 2016)
- LPIC-1/Linux+/SUSE CLA certified (January 2014)
Languages
- Hindi
- English
Personal Information
- Passport Number: W9969411
- Date of Birth: 13-March-1987
- Nationality: Indian
Timeline
Lead Data Engineer
EXL (Clairvoyant)
05.2023 - Current
Lead Data Engineer
Presidio Cloud Solutions
08.2021 - 05.2023
Senior Software Engineer
Assimilate Solutions
03.2020 - 08.2021
Consultant
Capgemini India.
07.2018 - 03.2020
Senior Software Engineer
Mphasis Ltd.
09.2012 - 07.2018
Bachelor of Engineering -
Sri Siddhartha Institute of Technology (VTU)