Summary
Overview
Work History
Education
Skills
Websites
Certification
Timeline
Generic

Nirmal Bisht

Data Engineer
Delhi,DELHI

Summary

Data Engineer with 3+ years of experience designing and implementing scalable data pipelines, ETL/ELT workflows, and modern data warehouse solutions. Proven track record of delivering high-performance big data systems for Fortune 100 pharmaceutical clients using technologies like Python, PySpark, SQL, Spark, Snowflake, and Databricks. Adept at optimizing data workflows and enabling data-driven decision-making at scale.

Overview

3
3
years of professional experience
3
3
years of post-secondary education
1
1
Certification

Work History

Data Engineer

ZS Associates
Gurgaon
09.2022 - Current
  • Developed and enhanced a scalable US MDM solution for a pharmaceutical client by integrating multiple data sources and orchestrating workflows using AWS S3, SQS, EMR, and Azkaban.
  • Optimized Spark-based data pipelines on AWS EMR, significantly improving performance and ensuring efficient processing of high-volume healthcare data.
  • Developed a Robust DQM Framework: Designed and implemented a Data Quality Management (DQM) framework using Snowflake stored procedures, ensuring smooth execution of 10+ data pipelines daily and enabling proactive identification of data quality issues.
  • Led the end-to-end development of the SDOH Data Lake: Used Talend for data integration and automation of data extraction processes from public portals using selenium and BS4, ensuring seamless storage in S3 and integration with downstream systems.
  • Automated Data Processing: Developed Python scripts to streamline data extraction, store data in S3, and create Staging tables in Snowflake, achieving a 50% reduction in extraction time.
  • Automated Data Store Creation: Developed an automated solution to generate Data Store Layer tables from a CSV configuration file and populate them with data from Staging tables, reducing manual effort and enhancing process efficiency.
  • Data Integration with Salesforce Marketing Cloud: Led the integration of data from Snowflake Tables into Salesforce Marketing Cloud via APIs, reviewing API documentation and collaborating with the SFMC team to ensure seamless data transfer and process efficiency.
  • Data Processing & Storage: Worked on Amazon EMR and EC2 for data processing, and S3 for staging layer data storage, integral to data pipelines in the early phase of my career.
  • Data Processing: Built an MDM solution using Hadoop and Hive; leveraged YARN for resource management and optimized Hive queries for efficient data warehousing.
  • Databricks Notebook Development for Data Ingestion & Transformation: Designed and implemented multiple Databricks notebooks to ingest CSV and JSON files from S3 into Snowflake tables. Applied PySpark for data transformations, optimizing the process for accurate and efficient data loading.
  • Conducted Comprehensive Testing: Performed extensive testing on the DQM framework to validate its functionality, ensuring consistent data quality and smooth operation across all pipelines.

Education

Bachelor of Technology - Computer Science Engineering

Maharaja Surajmal Institute Of Technology
Delhi
05.2019 - 07.2022

Skills

Python

Apache Spark

SQL

Hive

AWS

Snowflake

Data Modelling

Problem Solving

Data warehousing

Websites

Certification

Apache Spark Certified Developer Associate

Timeline

Data Engineer

ZS Associates
09.2022 - Current

Bachelor of Technology - Computer Science Engineering

Maharaja Surajmal Institute Of Technology
05.2019 - 07.2022

Similar Profiles

CREATE PROFILE
Nirmal BishtData Engineer