Nirmal Bisht
Data Engineer
Delhi,DELHI
Summary
Data Engineer with 3+ years of experience designing and implementing scalable data pipelines, ETL/ELT workflows, and modern data warehouse solutions. Proven track record of delivering high-performance big data systems for Fortune 100 pharmaceutical clients using technologies like Python, PySpark, SQL, Spark, Snowflake, and Databricks. Adept at optimizing data workflows and enabling data-driven decision-making at scale.
Overview
3
3
years of professional experience
3
3
years of post-secondary education
1
1
Certification
Work History
Data Engineer
ZS Associates
Gurgaon
09.2022 - Current
- Developed and enhanced a scalable US MDM solution for a pharmaceutical client by integrating multiple data sources and orchestrating workflows using AWS S3, SQS, EMR, and Azkaban.
- Optimized Spark-based data pipelines on AWS EMR, significantly improving performance and ensuring efficient processing of high-volume healthcare data.
- Developed a Robust DQM Framework: Designed and implemented a Data Quality Management (DQM) framework using Snowflake stored procedures, ensuring smooth execution of 10+ data pipelines daily and enabling proactive identification of data quality issues.
- Led the end-to-end development of the SDOH Data Lake: Used Talend for data integration and automation of data extraction processes from public portals using selenium and BS4, ensuring seamless storage in S3 and integration with downstream systems.
- Automated Data Processing: Developed Python scripts to streamline data extraction, store data in S3, and create Staging tables in Snowflake, achieving a 50% reduction in extraction time.
- Automated Data Store Creation: Developed an automated solution to generate Data Store Layer tables from a CSV configuration file and populate them with data from Staging tables, reducing manual effort and enhancing process efficiency.
- Data Integration with Salesforce Marketing Cloud: Led the integration of data from Snowflake Tables into Salesforce Marketing Cloud via APIs, reviewing API documentation and collaborating with the SFMC team to ensure seamless data transfer and process efficiency.
- Data Processing & Storage: Worked on Amazon EMR and EC2 for data processing, and S3 for staging layer data storage, integral to data pipelines in the early phase of my career.
- Data Processing: Built an MDM solution using Hadoop and Hive; leveraged YARN for resource management and optimized Hive queries for efficient data warehousing.
- Databricks Notebook Development for Data Ingestion & Transformation: Designed and implemented multiple Databricks notebooks to ingest CSV and JSON files from S3 into Snowflake tables. Applied PySpark for data transformations, optimizing the process for accurate and efficient data loading.
- Conducted Comprehensive Testing: Performed extensive testing on the DQM framework to validate its functionality, ensuring consistent data quality and smooth operation across all pipelines.
Education
Bachelor of Technology - Computer Science Engineering
Maharaja Surajmal Institute Of Technology
Delhi 05.2019 - 07.2022
Skills
Python
Apache Spark
SQL
Hive
AWS
Snowflake
Data Modelling
Problem Solving
Data warehousing
Websites
Certification
Apache Spark Certified Developer Associate
Timeline
Data Engineer
ZS Associates
09.2022 - Current
Bachelor of Technology - Computer Science Engineering
Maharaja Surajmal Institute Of Technology
05.2019 - 07.2022
Similar Profiles
Debasish GuptaDebasish Gupta
Associate Principal (Partner) at ZS Associates (ZS)Associate Principal (Partner) at ZS Associates (ZS)
Vinita Ahuja BajajVinita Ahuja Bajaj
Senior Human Resources Associate - Staffing at ZS Associates (ZS)Senior Human Resources Associate - Staffing at ZS Associates (ZS)
Kaden HoweKaden Howe
Operations Assistant at ZS AssociatesOperations Assistant at ZS Associates
Chinmai SharmaChinmai Sharma
Decision Analytics Associate at ZS AssociatesDecision Analytics Associate at ZS Associates
SHRESTH ISHSHRESTH ISH
Advisory Intern at Deloitte USIAdvisory Intern at Deloitte USI