Harnoor Singh

Experience:

• Worked in Retail and CPG domain to generate recommendations for products under different KPIs and B2B, B2C business models.
• Worked with data science teams to develop and architect big data pipelines that solve key business problems.
• Designed and developed batch and streaming ETL pipelines using cloud storage, Pub/Sub, cloud functions, Dataproc, Pyspark and BigQuery.
• Built scalable data solutions for large enterprise data leveraging structured and semi-structured data from multiple sources.
• Designed and developed data pipelines using Apache Airflow.
• Implemented Continuous Integration and Continuous Deployment (CI/CD) using Octopus and gitlab CI/CD.

Experience :

• Developed solutions that enabled data scientists to exploit data, obtain insight, and augment data analysis techniques.
• Created a denormalized BigQuery Schema for analytical and reporting requirements.
• Loaded historical data to Cloud Storage using Hadoop utilities and loaded to BigQuery using BQ tools.

Experience:

• Worked as a Big Data Engineer in the Data Lake Team for an automobile client where the client wanted to store, process & manage the huge amount of daily data collected from various sources.
• Developed Pyspark and Spark-SQL applications to extract, transform, and aggregate data from various file formats, uncovering customer usage patterns.
• Experienced in performance tuning and optimization of Spark applications for setting the right batch interval time, correct level of parallelism, and query optimizer.
• Proficient in data modelling and engineering tools like Hadoop, Spark, and Kafka.
• Incorporated data quality into day-to-day work and delivery process.
• Collaborated with cross-functional teams to create complex data processing pipelines to solve client's challenges.
• Built multiple scalable data pipelines with end-to-end ETL and ELT processing for Data ingestion and transformation.
• Developed multiple python libraries for interaction with different Cloud services.

Key Achievements:

• Developed a data pipeline using Data Lake that led to a client revenue increase of 19%.
• Solved an ETL issue while following Pyspark and Sql best practices that resulted in an insight that increased client’s customer base by 20%