Rohith Nukala

Client : Bank of New York Mellon

Project: Finance Business Intelligence (Sentinel Data & Unified Product Master Data).

• Designed and developed scalable ETL workflows to ingest data from MS SQL Server into Snowflake, processing ~15–20 GB of financial data daily.
• Automated daily scheduling with previous business day logic to delete and reload incremental data, eliminating duplication and achieving 100% data accuracy.
• Optimized load performance using efficient transformation logic, reducing data refresh time by ~30%.
• Created and maintained Snowflake DDL scripts (tables, schemas, stages) and implemented role-based access control (RBAC) to ensure secure data access.
• Collaborated with DevOps teams to deploy workflows through CI/CD pipelines, decreasing manual deployment effort by ~40% and enabling zero-downtime production releases.
• Developed secure ETL workflows to extract data from UPM REST APIs using PEM-based authentication, handling high-volume product master data.
• Engineered PySpark transformation pipelines to cleanse, standardize, and optimise large datasets (~200M+ records), enhancing processing efficiency by ~35%.
• Implemented data staging in HDFS before loading curated datasets into Snowflake, ensuring reliable and fault-tolerant ingestion.
• Applied Spark optimization techniques (partitioning, caching, broadcast joins) to enhance performance and reduce job execution time.
• Designed production-grade Snowflake DDLs and enforced RBAC policies in the production environment for secure data governance.
• Automated end-to-end deployment using CI/CD pipelines, improving release reliability and reducing deployment time significantly.
• Tech Stack : Python, Pyspark, HDFS, Snowflake Data warehouse.

Project Description: Connected Components (Clustering of telecom affinity data on the geospatial region to get the data segregated cluster to work on e Tilt-optimization).

• Developed Spark scripts with PySpark shell commands, processing 1.2TB daily data with 40% faster execution time.
• Created a conda environment on a Linux server and uploaded it to the development server.
• Tech Stack: Pyspark, HDFS, Git, Kepler-maps, etc.

Project Description: E-Tilt Optimization

• Processed 1 TB of daily telecom data from third-party APIs into Kafka using PySpark streaming, enabling real-time analytics and insights.
• Implemented Kafka-based data pipeline, handling 50,000 messages per second with 99.9% system reliability.
• Optimized Spark job performance by calculating optimal executor count, memory allocation, and core distribution, reducing processing time by 35%.
• Fine-tuned Spark application parameters through analysis of data partitioning strategies and shuffle operations, optimising resource utilization by 40%.
• Tech Stack: Pyspark, Kafka, HDFS, Git, Kepler-maps, etc.

Project Description: Jio Financial Services

• Ingested Customer 360 data from Kafka, leveraging PySpark Structured Streaming for real-time data processing.
• Followed the Medallion Architecture with Delta Lake for structured storage, enabling efficient data segregation across Bronze, Silver, and Gold layers.
• Managed Delta Tables for versioning, rollback, and time travel, enhancing data governance while Airflow orchestrated end-to-end data pipelines.
• Tech Stack: PySpark, Kafka, Delta Lake, Databricks, Airflow