Sandeep Chauhan

1. Client: 2. Client: 3. Client: 4. Client: 5. Client:

Project: Services, Interaction, and Subscriptions Data Models
Role: Big Data Engineer (2024 - Present)

About Chamberlain INC:
Chamberlain is a B2B company providing IoT-based services to clients across diverse industries such as deliveries (Amazon, Walmart, etc.), automotive (Tesla, Hyundai, Honda, etc.), and smart home solutions. The IoT devices collect vast amounts of data, which are analyzed to generate actionable business insights.

Roles and Responsibilities:

• Data Modeling:Implement data models for the Bronze, Silver, Gold, and Platinum layers based on data architect requirements.
  Develop metrics derived from existing data to support business intelligence needs.
• DBT Query Development:Write and maintain DBT (Data Build Tool) queries to extract required data from various sources.
  Develop and test DBT models to ensure robust implementation of data models.
• ETL Orchestration:Build and manage Azure Data Factory (ADF) pipelines for orchestrating ETL workflows efficiently.
• Data Transformation and Analysis:Create Databricks scripts using PySpark for analyzing and transforming data stored in the Bronze layer.
  

Technologies Used:
Databricks, Data Build Tool (DBT), Azure Data Factory, Azure DevOps, PySpark

Project: Providence
Role: Flink Developer (2023 - Present)

Project Overview:
The Providence project focuses on collecting and analyzing data from Warner Bros. Discovery’s applications used by clients. The data captures user interactions with the apps, along with any errors encountered during normal operations. This data helps improve app performance and enhances the user experience.

Roles and Responsibilities:

• Streaming Analytics Pipeline Development:Built a streaming analytics pipeline from scratch to monitor the real-time performance of Warner Bros. Discovery client applications.
  Tracked critical real-time metrics such as the health of active sessions, error-free session rates, crash-free session rates, and error types.
  Analyzed the collected data to address issues in new app releases, ensuring an optimal client experience.
• Event Filtering Pipeline:Developed a streaming pipeline to filter malformed events in real-time, aiding developers in improving client logger SDK event generation.
• Alerting and Monitoring:Configured alerts using Prometheus and PagerDuty for monitoring and notifying about the Flink application’s health and performance.
• Automation:Automated Flink application deployments using Terraform and GitHub Actions for streamlined and efficient deployment processes.
• Load Testing Microservice:Designed and implemented a microservice for load testing the Flink application, leveraging Kubernetes jobs to simulate millions of events per minute in a production-like environment.
• Data Partitioning Automation:Developed stored procedures to automate the attachment and detachment of PostgreSQL partitions, enabling seamless data migration between hot, warm, and cold storage tiers for Providence’s data.
• Pipeline Optimization:Provided ongoing support and implemented enhancements to improve the robustness and efficiency of the Flink pipeline.
  

Technologies Used:
Apache Flink, Kubernetes, Terraform, GitHub Actions, JFrog, Maven, AWS Managed Flink, AWS Timestream, AWS RDS, Grafana (for visualization), Prometheus (for monitoring), Kafka (for streaming).

Programming Language:
Java

Deployment Details:
The Flink application is deployed using AWS's managed service, AWS Managed Flink, ensuring scalability and reliability.

Project: OpenSearch Analytics
Role: Data Engineer (2022 - 2023)

Project Overview:
This project focused on identifying and addressing suboptimal OpenSearch query practices that impacted performance and response times. Additionally, it analyzed heavily used indexes to optimize scaling of OpenSearch clusters based on usage patterns.

Roles and Responsibilities:

• Designed and developed a data pipeline to analyze OpenSearch’s slow index and search query logs.
• Visualized analysis results using Grafana for better insights into query performance issues.
• Utilized AWS Lambda for data extraction, transformation, and storage in Athena.
• Automated metadata updates with AWS Glue Crawler.
• Configured S3 object creation notifications with AWS SQS to trigger Lambda functions.
• Deployed the pipeline using AWS CloudFormation and Jenkins for reliable and repeatable deployments.

Technologies Used:
AWS Lambda, AWS Athena, AWS SQS, AWS S3, AWS Glue Crawler, CloudFormation, Jenkins, Grafana

Programming Language:
Python

Project: Observability
Role: Data Engineer (2022 - 2023)

Project Overview:
This project aimed to enhance monitoring, incident reporting, and metering of Discovery’s infrastructure and services. By tagging resources (e.g., code repositories, AWS services, alerts) with operational metadata, the project facilitated quick resolution of failures and accurate cost metering for individual teams.

Roles and Responsibilities:

• Built data pipelines to collect and maintain operational metadata for infrastructure resources on a daily basis in a PostgreSQL database.
• Tagged resources to enable faster failure resolution and provide cost visibility per team.
• Leveraged PySpark for data processing and GitHub Actions for CI/CD automation.
• Integrated Grafana dashboards for visualizing metadata and operational metrics.

Technologies Used:
PySpark, GitHub Actions, PostgreSQL, Grafana, AWS Athena

Programming Languages:
Python, Java

Project: Stream Metrics as a Service
Role: Data Engineer (2022)

Project Overview:
The project involved real-time analysis of data stream events and visualization of metrics on Grafana dashboards. These analytics supported critical business decisions for Discovery.

Roles and Responsibilities:

• Designed and implemented a real-time data pipeline from scratch using Apache Flink.
• Conducted event stream analysis and provided real-time insights visualized on Grafana dashboards.
• Deployed the Flink application using AWS Kinesis Data Analytics service for scalable and managed streaming analytics.

Technologies Used:
Apache Flink, AWS Kinesis Data Analytics, Grafana

Programming Language:
Java

Project:

Role: Data Engineer (Jan 2020- Dec 2021)

Project Overview:
The project focused on the Extraction, Loading, and Transformation (ELT) of data from diverse sources, including Oracle databases, flat files, JSON, CSV, Avro, and Parquet files, into a Data Warehouse. The processed data was then populated into Hive tables for generating reports and supporting business intelligence activities.

Roles and Responsibilities:

• Data Extraction and Ingestion: Extracted data from Oracle databases and various file formats (CSV, Avro, Parquet, etc.) provided by the data ingestion team.
  Ingested the extracted data into EMR clusters using Sqoop and AWS S3.
• Data Transformation and Loading:Transformed data in the staging layer using PySpark and loaded it into external Hive tables stored on AWS S3.
• Workflow Design and Development:Designed and implemented production workflows using Apache Airflow with technologies such as Python, Unix, Hive, PySpark, and Sqoop.
• AWS Automation and API Development:Developed Python APIs for AWS authentication (Cerebrus and Boto), EMR cluster management (spin-up and termination), and AWS S3 storage operations.
• Requirement Gathering and Technical Solutions:Collaborated with clients to gather application requirements and provided tailored technical solutions.
• Production Support:Provided production support for Nike Digital Sales and Analytics, including resolving critical P1 incidents and conducting root cause analysis for failures.
  Addressed job failures in production and QA environments, ensuring operational issues were resolved promptly.
  

Technologies Used:
Spark, Hive, Python, Unix, Sqoop, MySQL

Tools:
GitHub, PyCharm, Eclipse, MobaXterm, PuTTY

Project:

Role: Data Engineer (August 2019 to Jan 2020)

Project Overview:
The project involved analyzing IPL datasets to extract valuable insights and generate reports on player and team performance. Key analyses included identifying the best batsman, bowler, and fielder for a season and predicting future events.

Roles and Responsibilities:

• Data Extraction and Transformation:Extracted, transformed, and loaded data from IPL datasets using source queries and Apache Pig.
  Conducted analyses to determine overall best players based on historical performance.
• Reporting and Dashboard Creation:Developed detailed reports on season and team performance, highlighting key players.
  Created interactive dashboards using Power BI to visualize insights effectively.
  

Technologies Used:
Power BI, Apache Pig

Tools:
Power BI, PuTTY