Open To Work
Priyanka Gupta
Lead Data Engineer
Noida
Summary
Dynamic Lead Data Engineer at Brillio with expertise in cloud infrastructure migration and data pipeline management. Proven track record in optimizing ETL processes and enhancing data quality. Adept at strategic development and communication, mentoring junior engineers, and driving successful data integration projects. Passionate about leveraging data for impactful business insights.
Overview
1
1
Certification
12
12
years of professional experience
Work History
Lead Data Engineer
Brillio
03.2022 - Current
- Developed data ingestion processes from various sources into centralized ODS schema.
- Designed table structures in Snowflake to support new data model for efficient transformation.
- Analyzed ETL infrastructure of large retail firm (over 5k pipelines) for migration from legacy DataStage architecture to cloud-native data platforms.
- Assessed ETL solutions for migrating infrastructure to cloud-native environment, including Azure Data Factory, Informatica IICS, Talend, and Databricks/DBT.
- Contributed to design discussions, enhancing scalability and efficiency of data warehouse structures.
- Developed Snow-SQL queries replicating extraction logic from YAML configurations for ODS.
- Analyzed source tables from multiple WMS systems to extract accurate data for ODS loading.
- Created YAML configurations for standardized data extraction across multiple WMS systems.
- Collaborated with ADQ team to perform data validation checks and resolve discrepancies.
- Mentored junior engineers on data modeling and ETL processes.
- Collaborated with the Learning team to curate a comprehensive Datastage Training program for freshers, ensuring that the content was aligned with the current industry standards and practices.
Technical Advisory Sr Analyst
IBM Consulting
10.2021 - 03.2022
- Engineered scalable database structures for credit card data ingestion into data warehouses.
- Developed data transformation logic for cleansing, validation, and imputation processes to enhance data quality and usability.
- Collected business requirements to produce comprehensive high-level and low-level design documents that guided project development.
- Analyzed client data to identify trends and insights for strategic recommendations.
- Managed database backup procedures to maintain data reliability and availability, ensuring seamless access for stakeholders.
Application Development Senior Analyst
Accenture PLC
05.2019 - 10.2021
- Led application development projects using Agile methodologies, enhancing team collaboration and project delivery.
- Collaborated with cross-functional teams to gather and document software requirements, ensuring alignment with business needs.
- Architected ETL processes utilizing IBM DataStage, SAP BODS, and Alteryx for various BI initiatives.
- Automated ETL job transfers across environments using shell and batch scripting, streamlining data processing workflows.
- Optimized overall ETL processes by monitoring performance metrics to decrease processing times.
- Crafted source-to-target mappings that extracted data from Teradata, SQL Server, Snowflake, and flat files.
- Designed database models to enhance organization and manipulation of stored data.
Associate Manager IT-Business Intelligence
Apollo Munich Health Insurance
11.2018 - 05.2019
- Directed upgrade of entire ETL infrastructure from IBM DataStage v8.x to v11.7, improving performance.
- Designed ETL processes to populate data warehouse and marts, supporting multiple departments' data accessibility.
- Established dedicated production and non-production environments for efficient ETL management.
- Constructed data mart tailored for Android app, facilitating marketing initiatives.
- Evaluated user report data needs, providing timely information through SQL queries from data marts.
Senior Integration Analyst
NTT Data Services
04.2017 - 11.2018
- Transformed BA requirements into SQL, producing ETL jobs for data extraction from Oracle and Teradata.
- Developed ETL processes using IBM DataStage, integrating active and passive stages in server and parallel jobs to streamline data flow.
- Coordinated development, deployment, and operational activities, improving overall project efficiency and communication.
- Created files for vendor analysis through thorough data extraction, supporting informed decision-making.
Systems Engineer
Tata Consultancy Services
09.2014 - 04.2017
- Developed ETL processes utilizing active and passive stages in server and parallel jobs within IBM DataStage, optimizing data flow and processing efficiency.
- Migrated ETL processes from Ascential DataStage to IBM InfoSphere DataStage v11.3, ensuring seamless transition and alignment with user needs.
- Designed and developed DataStage jobs and sequences aligning with business user requirements.
- Translated business analyst requirements into SQL, creating ETL jobs for data extraction from Oracle and Teradata.
- Cleansed and transformed data according to established business logic, enhancing data quality and reliability for downstream processes.
- Executed unit testing and system integration testing based on defined test cases.
- Converted batch scripts into UNIX scripts for enhanced performance.
- Wrote scripts to automate file transfers and facilitate file manipulation tasks.
Education
Bachelor of Technology - Information Technology
IEC College of Engineering And Technology
Greater Noida06-2014
Skills
- Data integration and validation
- Cloud infrastructure migration
- Database architecture and warehousing
- Communication proficiency
- Strategic development
- Performance optimization
- Data modeling and pipeline architecture
- Data pipeline management
Tech Stack
Python, Shell Scripting, Azure Data Factory, Azure Databricks, Snowflake, IBM DataStage, Informatica, SAP BODS, Talend, Alteryx, Oracle, MySQL, SQL Server, Teradata, Tableau, Power BI, Data Modeling, ETL/ELT, Data Warehousing, Performance Optimization
Certification
- Microsoft Certified: Azure Data Fundamentals (DP-900), 2021
- Microsoft Certified: Azure Data Engineer Associate (DP-203), 2021
- Alteryx Designer Core Certified, 2020
- Google Cloud Digital Leader, 2025
Work Preference
Job Search Status
Open to workWork Type
Full TimeLocation Preference
RemoteSalary Range
₹3600000/yr - ₹4000000/yrKey Project Highlights
- Developed automated table count validation framework, improving QA data validation accuracy.
- Guided development of AI-powered ETL migration tool converting DataStage jobs (DSX) to SQL.
- Led ETL modernization initiatives, reducing dependency on legacy systems.
Timeline
Lead Data Engineer
Brillio
03.2022 - Current
Technical Advisory Sr Analyst
IBM Consulting
10.2021 - 03.2022
Application Development Senior Analyst
Accenture PLC
05.2019 - 10.2021
Associate Manager IT-Business Intelligence
Apollo Munich Health Insurance
11.2018 - 05.2019
Senior Integration Analyst
NTT Data Services
04.2017 - 11.2018
Systems Engineer
Tata Consultancy Services
09.2014 - 04.2017
Bachelor of Technology - Information Technology
IEC College of Engineering And Technology