Prashant Gupta
Cloud and Data Engineering
Summary
Experienced data engineering leader specializing in designing and implementing scalable data solutions. Proficient in data architecture, ETL processes, and cloud platforms to streamline data management and analytics. Emphasis on collaboration and achieving impactful results, with a talent for adapting to changing project requirements. Recognized for strategic thinking and reliability in high-pressure situations.
Overview
15
15
years of professional experience
2009
2009
years of post-secondary education
3
3
Certifications
2
2
Languages
Work History
Cloud Data Engineering
Intel India
07.2023 - Current
- Led the design and architecture of complex data pipelines, facilitating robust and scalable solutions for diverse sources using Azure cloud service (ADF, Databricks, ADLS,Spark..etc). Specialized in building complex data processing systems that enhance data accessibility and analytical capabilities.
- Acted as a thought leader in data architecture, contributing to the conceptualization and development of scalable ETL/ELT frameworks.
- Spearheaded the design and development of cloud-native applications using a robust Data engineering technology stack, ensuring seamless end-to-end data flow and processing across multiple Data Lake platform.
- Managed and cultivated relationships with key stakeholders across Data, Operations, and Risk Tech domains to advocate and ensure the adoption of Enterprise Architecture frameworks.
- Defined strategic roadmaps for performance tuning of complex Data applications, improving processing efficiency and scalability across various stages of development.
- Consulted and executed data migration projects, transferring large datasets from Hadoop cluster and SAP-HANA/BW systems to Azure data lake and Snowflake cloud data warehouses, ensuring data quality and integrity.
- Provided leadership and technical direction to teams consisting of full-time employees (FTEs), strategic partners, and software vendors, enhancing their capabilities in data architecture and engineering with a focus on Intel standards.
- Developed and conducted training sessions to elevate team competencies in core data technologies and methodologies.
- Mentored and advised scrum teams to meet product expectations, provided technical guidance and performance feedback, and supported career development to foster team growth and individual skill enhancement.
Senior Data Manager / Technical Architect
Capgemini financial services
04.2021 - 07.2023
- Company Overview: London (UK)
- Assisting the customer with the creation of a technical design document from an architecture blueprint for their data platform vision and worked with client to assess, design and implemented Delta-Lake house architecture using Azure cloud services.
- Lead the development of lake-house architecture and business intelligence solutions to support the organization's reporting, AI/ML and analytics needs.
- Holds deep hands-on experience with EDW/DWH Migration Experience to Cloud, platform migration, Hadoop cluster to cloud and building modern data lakes and DataOps.
- Experienced in building and maintaining reliable ETL/ELT on data platforms as well as working with varied forms of data infrastructure.
- Proficient in developing ETL pipelines using Azure cloud services (Data Factory, Synapse, Delta Lake, azure Data Lake, Azure Stream Analytics, Event hub, SQL Server), Databricks / Apache Spark, and other data processing frameworks.
- Resolving technical issues and advising on best practices for Data engineering, big-data, Hadoop and Cloud environments.
- Proficient in developing ETL pipelines using GCP cloud services (Big Query, Data Fusion, Data Flow, Data-Lake) and Container based and open-source technologies with data processing frameworks.
- Optimize and tunning of distributed data processing application and retrieval performance by designing efficient data storage and indexing/partition/bucketing/z-order/distribution strategies.
- Designed and developed reusable meta-data based ETL framework/solution that involves data ingestion, enrichment and transformation which helped to reduce development effort by 60-70%.
- Experience to build and lead a highly performing data engineering team through recruitment and coaching, leading through setting standards and leading by example.
- London (UK)
Lead Data Engineer
Publicis. Sapient
11.2016 - 03.2021
- Company Overview: London (UK)
- Worked with client to assess, design and implement leading practice operating models for brand new risk and control framework and its align banks internal governance to risk management and control activities to assure and support business decision making and performance using Google cloud services (BigQuery, PubSub, Dataflow, Containers, DBT, Argo, microservices).
- Designed and developed REST endpoints to fetch data from Cassandra. Identified REST endpoints consumes 80% mainframe capacity, created spring boot micro service on Casandra to serve the data to its B2B and B2C customers.
- Understanding of current on-premises Hadoop data lake and prepared Azure cloud migration strategy and approach documents, integration approach and hardware capacity planning along with HDInsight deployment for different environment.
- Worked on fast-paced agile development environment which involved design, development and implementing Kafka streaming application for ETL process with apache spark-based reconciliation application.
- Migrated of Hadoop on-premise historical data and app solution to Azure Data Lake as well replacing the existing ingest system to Azure using Azure service (DataLake, CosmosDB, HDinsight,Data Factory,Hadop,Hive,Pig,Ambari,Spark,on-prem Hadoop clusters).
- Application security testing support with non-functional testing strategy (Load,Stress,OAT,Scalability,Peak,Bulk) with RTL (root to live) support.
- Designed and developed reusable components which will be enhanced and leveraged across big data testing using framework for various purpose on BDD or TDD approach for data ingestion, data curation and data consumption for batch or real time process.
- London (UK)
Technical Lead
Manthan Systems
11.2015 - 03.2016
- Part of Marketplace analytics team, responsible for all set of analytics like seller performance, daily transactional data sales analysis, promotion, seller engagement, Daily reports related to marketplace traffic, sales, items.
- Maintained and supported existing Enterprise data warehousing Design and develop summary data models, Transformations and loading strategies for business data, in other way developing new ETL solution using Hadoop for reporting purpose.
Data Engineer
Century link Technologies
11.2013 - 11.2015
- Designed and developed solutions using big data / Hadoop based technologies
Software Engineer
Infosys Limited
03.2010 - 10.2013
- Banking Product development and customization.
Education
Bachelor of Technology - Electrical & Electronics Engineering (EEE)
Dr. A.P.J. Abdul Kalam Technical University
Lucknow, INSkills
Data architecture design
ETL framework development
Data pipeline design
Cloud data migration
Data lake implementation
Azure-ADF,Databricks,Synapse
Unity-Catalogue (UC)
GCP-Big Query,Pub Sub
Data Flow,Data-Fusion
Data build tool (DBT)
Hadoop,MapReduce,Hive
Impala,Phoenix,Pig,Sqoop
Flume,Kafka,Spark,
Presto,Jethro
Cloudera-CDH, HDP, MapR, Qubole
Cask-CDAP
Autosys,airflow,controlM
Java/J2EE, Scala, Python, Bash
SqlServer,Oracle, Redshift,
MongoDB, Hbase, Cassandra,DynmoDB
Jenkins,Terraform,ArgoCD,
GitHub,Azure Devops, Docker,Kubernetes
PowerBI,Tableau
Certification
Microsoft Cloud Certification, Azure Solution Architect (AZ-305), Azure Data Engineer (DP-203), Azure administrator (AZ-104), Azure and Data fundamentals (AZ-900, DP-900)
Accomplishments
- Achieved recognition for best performer.
- Improved efficiency of data-on-cloud solution on Azure by 70% using various optimization technique and automation and building reusable components and frameworks.
- Attended various data conference/meetups/webinars conducted by Google, Facebook, Databricks, Microsoft, Babylon Health, Confluent, MongoDB.
Personal Information
Nationality: Indian
Blog / Publication
- Apache Spark based Reconciliation and Quality assurance framework
https://medium.com/engineered-publicis-sapient/9e442f954ce
- Backup your Google-Big Query Dataset
https://prashant-gupta1616.medium.com/991448c132f5
Timeline
Cloud Data Engineering
Intel India
07.2023 - Current
Senior Data Manager / Technical Architect
Capgemini financial services
04.2021 - 07.2023
Lead Data Engineer
Publicis. Sapient
11.2016 - 03.2021
Technical Lead
Manthan Systems
11.2015 - 03.2016
Data Engineer
Century link Technologies
11.2013 - 11.2015
Software Engineer
Infosys Limited
03.2010 - 10.2013
Bachelor of Technology - Electrical & Electronics Engineering (EEE)
Dr. A.P.J. Abdul Kalam Technical University