Akhila Martia Mariadas

Key Responsibilities:

• Developed an evaluation framework for various model use cases within organization, including chatbots, copilot, golden datasets, and more.
• GCP Vertex AI Pipelines to automate daily evaluation jobs.
• Implemented metrics for Coherence, Relevance, Entailment, Prompt-based LLM scores, Model Metadata, Topic Drift, BERTScore (precision, recall, F1-score), BLEU, METEOR, and ROUGE.
• Created a dashboard to visualize model drift/decay over time.
• Used batch prediction jobs for LLMs and horizontally scaled processing based on record volume, optimizing batch size and machine selection.
• Designed and executed parallel processing of multiple metric jobs to reduce overall evaluation time.

Key Achievements:

• Successfully processed 100k records within 2-3 hours, demonstrating high efficiency.
• Delivered a robust framework that significantly enhanced the client's ability to monitor and maintain model performance over time.

Key Responsibilities:

• Developed a comprehensive Python application for benchmarking Nvidia server performance to check which is the best serving solution/LLM model for Intent/Emotion classification give agent-customer conversation for realtime predictions.
• Integrated various serving solutions for benchmarking, including Deep Java Library, Text Generation Inference, vLLM, Triton TensorRT LLM, DeepSpeed Mii, and FastChat.
• Employed quantization techniques including GPTQ, AWQ, and FP16 to optimize model performance.
• Conducted detailed batch and load testing on deployed models to evaluate and enhance performance.

Key Achievements:

• Achieved response times of 0.6-0.7 seconds per request, ensuring high performance under load to enhance real-time performance, leading to improved operational efficiency.
• Maximized server utilization for KV caching and number of max sequences, fully exploiting server capabilities.
• Created comparative plots to analyze performance differences between various serving solutions.

Key Responsibilities:

• Developed a Copilot-like application for the client on an on-premises distributed environment with V100 GPUs. and expose API to be used by Plugins - VScode, STS and Eclipse..
• Implemented distributed fine-tuning across a setup of 3 nodes with 4 V100 GPUs each, using DeepSpeed and PEFT LoRA adapter techniques.
• Deployed the fine-tuned models on a single-node 4 GPU server utilizing DeepSpeed's dynamic batching with Deep Java Library (DJL).

Key Achievements:

• Improved overall productivity by 15% among developers, enhancing the client's development efficiency.
• Enabled seamless toggling between pretrained and fine-tuned models, optimizing task-specific performance.
• Achieved high throughput by accommodating multiple batches with a tensor parallel degree of 4, handling large input (1024 tokens) and output (1024 tokens) sequences effectively with dynamic batching.