Rishi Dhariwal

• Client: Skechers
• Elvis Data Framework: Engineered a fault-tolerant data ecosystem using Databricks (Delta format), Confluent Kafka, and Qlik, adhering to enterprise data engineering best practices. Centralized reporting data into a Delta Lake, enhancing accessibility and reliability for downstream analytics.
• File Ingestion Framework: Designed and deployed an end-to-end data ingestion pipeline with PySpark, Shell Scripting, Databricks, and Airflow to load structured files from multiple SFTP sources into Delta Lake, creating a unified source of truth for business analytics.
• Talend to Databricks Migration: Led the automation of Talend job migration to Databricks, leveraging intelligent mapping and scalable design patterns to enable faster processing and performance gains across pipelines.
• Cost Optimization Strategy: Reduced S3 storage costs by 70% for Delta tables via bucketing strategies and implementation of efficient vacuum techniques across Bronze and Silver layers, improving both storage efficiency and query performance.
• Snowflake Migration Initiative: Headed a phased migration of Fact and Dimension tables from Snowflake to Databricks, translating complex SQL logic into scalable Databricks code. Delivered $196K+ in estimated annual savings, while modernizing the BI platform for performance and cost efficiency.
• Company Code-Level Parallelization: Re-architected wholesale data pipelines to process data independently by Company Code (CC). Developed modular DAGs in Airflow to eliminate cross-dependencies, cutting multi-hour delays and streamlining data movement across Bronze, Snowflake, and MSTR layers.

Client: Bank of America

• Migrated VERTICA/TERADATA workloads to Hadoop using Spark, Hive, Shell Scripting.
• Built Spark-SQL scripts for data transformations; applied optimization techniques.
• Developed CI/CD pipeline using Jenkins; integrated Autosys for job scheduling.
• Processed unstructured application logs to extract DDL/DML using Spark.
• Created shell utilities for log processing; automated workflows end-to-end.

Client: PayPal

• Subledger Platform Recon: Built automated ETL pipelines and aggregation logic in Spark.
• Generic Data Quality Framework: Designed parallel processing rules, handled batch/streaming data, and enabled multi-format data profiling.
• Common Exception Framework: Developed end-to-end Spark Java logging system, integrated with CI/CD, and loaded Hive data into HBase.

Client: Morgan Stanley

• Developed automated revocation system using Spark and Scala for UIOLI analytics.
• Built parallel JSON parsers and optimized data validation and small file compaction utilities.
• Designed scheduling and monitoring scripts using Autosys.