Biswapratap Chatterjee

Task 1: Custom Combined Loss function in Faster RCNN model of Detectron2

The task was to add a new head to the Faster RCNN model of Detectron2 to implement a custom combined loss function so that a custom structural constraint can be applied on the model.

Project Highlights:

• A new class named CustomROIHeads was defined. It was derived from the StandardROIHeads class. This is the head where all the losses viz; Mask, Bounding Box and Keypoints are calculated.
• The goal was to add another custom combined loss function which optimizes a constraint i.e. all the Keypoints must lie inside the Bounding Boxes.
• In the training cycle along with the loss optimization, inferencing was also added for the Bounding Boxes and the Keypoints.
• Then the new loss was calculated and optimized back into the model which in turn learned this new structural constraint.

Technologies used

Python, Pytorch, Tensorboard, FasterRCNN, Detectron2.

Task 2: Calculating Keypoints from Bounding Boxes and Segmentation Masks in Detectron2

This task involved the calculation of Keypoints from Segmentation Masks and Bounding Box information.

Project Highlights:

• The Segmentation Mask's Polygon's points were reduced by using approxPolyDP approximation.
• The eucledian distance from each corner of the Bounding Box with its nearby Segmentation Mask polygon is calculated.
• The nearest point is selected as the Keypoint.
• This algorithm generated Keypoints with more than 90% accuracy.

Technologies used

Python, Pytorch, Tensorboard, MaskedRCNN, Detectron2.

Project 1: Generating Test cases from PDF, Excel and JSON data

This project was about automatically generating Test cases from PDF data using large language models. The problem was solved using Google's AI Gemini-flash-2.0 LLMmodel. The solution involved usage of Prompt Engineering, RAG architecture and Langchain pipelines.

Project Highlights:

• Read and collate PDF contents and then split them by the RecursiveCharacterTextSplitter.
• Cloud LLM was prohibited, so Langchain library was modified to accommodate Gemini- Flash2.0 model.
• Text chunks were vectorized by Gemini-Flash2.0’s text-
• embedding-004 and stored in Chroma vector DB.
• The test cases were to be presented as CSV data with multiple fields. So, for each field a RAG template was created.
• A RAG pipeline was created to hierarchical drill down into the PDF contents to extract the concerned field so as to generate the test cases in the CSV.

Technologies used

Large Language Model (Gemini-flash2.0), RAG, Langchain, Python.

Project 2: Identifying and measuring the Degraded Road Markings

Since annotating degraded markings is impossible (because of the random distribution of the degrades and the lack of information of the annotators about every degeneration) found a novel solution which can solve both the problems. The solution was successfully adopted in the product.

Project Highlights:

• Degraded road markings are impossible to guess due to high variability. So masking annotations are of very low quality and misleading.
• Found an innovative way where POSITIVE annotations were injected with random errors and treated as degradations (NEGATIVE annotations)
• Split the masked annotations into 5x5 pixel boxes.
• Trained a binary classifier to classify these boxes as degradations or not.
• Predicted new images as a group of 5x5 boxes.
• Calculated the degradation percentage using the boxes.

Technologies used:

YOLOv8, COCO, Bounding-Box, Masking, Contouring, Polygon Determination, IOU, Mask Area, Custom Loss Function, Custom Image Transformations, Pytorch, LangChain, Onnx, Detectron2, FiftyOne (Voxel51), Unsupervised Clustering, Auto Labeling, Transfer Labeling.

Project 3: Intravascular Ultrasound Image Analysis

Developed a simple Intravascular Ultrasound Image Analyzer Model to predict plaque regions within the blood vessels. Although the training was supervised and respectively simpler but the real challenge was in collecting labeled IVUS images as these are medical documents and mustn't be shared publicly. Moreover, it's illegal to buy the images from any source as well. So used a Triple GAN model to simultaneously generate and classify synthetic IVUS images for classification.

Project Highlights:

• Collected 20 DICOM sets of IVUS data. Each DICOM had a set of 40-50 IVUS frames.
• So we had 100 IVUS frames. Split opened the polar frame image into radial frame.
• Again split the radial image into sets of 5XN strips of number 50. So now we have about 5000 rectangular strips.
• Auto annotated each strip as “has plaque” or not.
• Used these 5000 strips to generate more synthetic strips along with their classification using TripleGAN.
• So now we have enough training data to predict if a strip has plaque or not.
• Converted the radial strips back into polar IVUS image and complete the prediction.

Technologies used:

Radial to Polar Image Conversion, Logical Image Splitting to create Strips, Triple GAN model training, Synthetic Strip Generation and Classification

Project 4: C/C++ Static Code Analyzer

Developed an AIML based static code analyzer for C/C++ projects. This project served as an simple but powerful alternative for existing code analyzers. The purpose of the analyzer was to predict vulnerable regions of code so that we can stress the test cases around those areas.

Project Highlights:

• Trained a NER transformer to identify C/C++ literals.
• Parsed the code and assigned NER values to each literal.
• Created a node data structure consisting of function name and NER type.
• Created a graph data structure connecting all callable functions together.
• Each reachable Node in the graph was a path that was programatically feasible to be called.
• Listed down all such paths.
• Used one hot encoding to vectorize each path.
• Pushed each path as a sequence to an Attention Neural Network.
• The annotated output i.e. vulnerable or not was obtained from developers, JIRA etc.
• On inference every new path will be scrutinized for vulnerability.

Technologies used:

Graphs and Trees, Recursion, Seq-2-Seq AttentionEncoder Decoder Model, Reinforcement Learning.

Project 5: Horizontal and Vertical Vector Search

A challenging but very common use case for search. The situation is when we have tabular data with variations in both the columns and rows. Variations can range from Text Variations, Number Variation, Date Variation, Keyword Variations etc. Currently the state of the art tech to solve such problems is using LLMs (Paid, Unpaid, Downloadable, CloudBased) that depends on the number of parameters and its variations. Solved the problem with a unique approach using Vector Databases and Ranking Based Hybrid Search. The entire solution was multi-threaded (Python Lock was used) where one thread was dedicated in accepting, processing and updating the VectorDB and the other thread is dedicated to the CRUD operations triggered from the user actions via a Menu. Python Signals were used for this.

Project Highlights:

Used Qdrant’s Hybrid search feature.
Crafted complicated queries by adding query points.
Allocated rank values to increase or decrease query points.

Technologies used:

Qdrant, ChromaDB, FTS, Ranking, Hybrid Search, Custom Embedding Logic, Modifying Standard Search Algorithm of Qdrant, Multi-threading, Python Locks, Python Signals

I unfortunately met an accident and had to take a break of 2 years from regular work. I spent these 2 years in personal development healthwise and skillwise. Now I perfectly healthy and working at my 100% capacity.

Project 1: Generic PDF Parser

Extremely interesting and one of my favorite self-made creation. Its not hidden from any NLP engineer how difficult but important it is to be accurately extract information from PDFs. Normal PDF parsing - Block of Text or even Text inside Images are pretty simple but what about extracting tabular information from a PDF into an Excel file? Although there are tools which claim that they can do it but none of them work in all cases, especially when the table's layout and orientation is very different from usual plain tables. Developed the most efficient table data extractor from any PDF with any orientation or layout. The interesting part in this work was its entirely developed using traditional algorithms and not AI.

Project Highlights:

• Divided the pages into rows and columns.
• Arranged all nodes into top left corner of the bounding boxes.
• Recursively took decisions on column and row detection.
• On discovery auto arranged them into a table.
• Monitored for node collisions and applied mitigation plans.
• Compressed sparsed table into compact form.
• Always recursively checking if the layout sanity is maintained or not.
• Wrote the final dataframe into a CSV.

Technologies used:

Core Python, Numpy, Dataframes, Recursion, Graphs and Trees, PyMuPDF, PyTesseract.

Project 2: Siamese Model for Image Identification

Had to identify automobile parts from manuals. The challenge was that the same part had multiple 3D views from multiple angles in many places of the manual. Some similar, some different. There was no information about the geometrical numbers of each subparts as well. And there was no training data to train a classifier which could identify apart irrespective of it view. So, instead of supervised learning, chose contrastive learning to at least predict if two parts are same or not, albeit without predicting the class of the part. Then cluster similar parts and compare with ground truth to manually label the cluster.

Technologies used:

DeTr, Siamese Contrastive Learning.

Project 3: Document Intelligence Contract Review

This product was primarily used for reviewing of contracts by EY auditors. The Product provided contextual AI hints about topics inside a contract, thus, drastically reducing the time and effort of the auditors. Used NLP based techniques such as - Word embedding, NER (Named Entity Recognition), OCR (Optical Character Recognition) - ABBYY and Poppler etc, Online Learning.

Used libraries like - numpy, scipy, sklearn, keras, tensorflow, opencv, pandas etc. Was also involved in the deployment activities which included technologies like - Azure ML Studio, Docker, RabbitMQ, Flask, Kubernetes, CI-CD pipelines
etc.

• Developed full stack C libraries for SATA, SCSI, RAID and NVMe products.
• Development and maintenance of the software management stack for high end network controllers from Broadcom. Acted as a prime engineer in designing and implementing the entire stack. I have complete knowledge on the stack starting from the CLI, CIM till the OS library
• Sustenance of the well-known product – Dothill Storage Array. The storage array was used to address the mid-range of the array market with various features involving SATA and SAS drives. The project required me to work on the sustenance roadmap of the Dothill storage array. I worked in the night US shift to address high priority customer issues by analyzing Firmware logs and providing recovery or code fix action plans. I was involved in multiple modules of the product from – Platform, Backend, RAID, Cache, Paged Storage, Snapshot and Tiring.
• Optimaliseer - A reinforcement learning agent aiding Western Digital Drive firmware to enhance performance. Auto tuning of firmware configurable parameters to achieve enhanced performance. Achieved 12% performance improvement as compared to human (expert) tuning.