Arunbalaji Selvaraj

As the Lead AI Engineer, I am responsible for architecting and delivering intelligent solutions that enhance platform scalability, performance, and automation. Spearheaded initiatives in AI and search infrastructure, while leading end-to-end implementation of advanced language models and retrieval systems. Collaborated cross-functionally to translate business goals into innovative, production-grade AI capabilities.

Key Accomplishments:

• Search Infrastructure Optimization:
  Led the migration from Elasticsearch to self-hosted OpenSearch, achieving significant cost savings and boosting system performance and query efficiency.
• GenAI Microservice Architecture:
  Designed and deployed a standalone GenAI service that enables chat-based interactions, prompt-driven document extraction, and vector-based document relevancy search, forming the core of intelligent content understanding across the platform.
• LangChain Integration for Modular LLM Workflows:
  Incorporated LangChain to enable customizable, chain-based LLM workflows for varied business use cases such as document summarization, query-based retrieval, and multi-turn reasoning.
• GenAI-Powered Intent-Based Search Engine:
  Developed an AI-augmented search engine by integrating an agent-based layer with OpenSearch. The agent interprets user intent from natural language queries and dynamically constructs optimized OpenSearch queries, delivering highly relevant results and a natural, conversational search experience.
• Prefect-Based LLM Agent Orchestration:
  Integrated Prefect , a Python-native workflow orchestration framework, to streamline agent-driven task flows, enabling robust and scalable LLM-based automation across the platform.
• Advanced OpenSearch Data Modeling and Query Design:
  Conducted in-depth research and implemented optimized OpenSearch data strategies tailored to use-case demands. Designed infrastructure around time-based indexing, parent-child document relationships, and deeply nested document structures. Engineered dynamic query builders and complex aggregations for high-performance dashboard analytics, balancing cost efficiency, and speed at scale.

As an AI engineer, I contributed to building scalable, AI-powered document processing, and workflow systems. Focused on integrating intelligent search, dynamic UI components, system orchestration, and post-processing features to improve the platform's efficiency, scalability, and user experience. Played a vital role in enhancing document discoverability, data interaction, and automation readiness.

Key Achievements:

• Elasticsearch Integration for Scalable Document Management: Integrated Elasticsearch into the platform to store documents and their extracted data. This enabled lightning-fast, full-text search across millions of documents, and allowed users to filter results based on multiple factors. Elasticsearch improved not only speed, but also reliability, enabling flexible and complex queries that could handle a massive volume of data without sacrificing performance.
• Dynamic Document UI for Improved Data Management: Developed a dynamic UI for displaying and managing documents, allowing users to configure columns and customize data views based on extracted information. This dynamic interface made it easier to sort, filter, and analyze documents in real time, offering a more intuitive and flexible user experience for managing large datasets.
• Dynamic Analysis Dashboard for Aggregated Insights: Designed and implemented a Dynamic Analysis Dashboard that provides comprehensive insights into key document extraction metrics, such as ML prediction quality, and the document state lifecycle (e.g., under review, completed, accepted, rejected). The dashboard aggregates data across millions of documents, offering scalable performance for large datasets. Users can apply dynamic filters to customize their views, drill down into specific extraction patterns, and analyze trends at scale. This tool significantly improved operational efficiency by offering real-time visibility into the quality and status of document processing.
• Optimized Document Chunking Mechanism: Introduced a chunking mechanism as a core architectural change to optimize the document ingestion flow. This allowed the system to process large documents—such as those exceeding 2,000 pages—without requiring large, expensive machines. The chunking system parallelized the processing of document sections, enhancing both scalability and stability. It also reduced resource consumption, allowing the platform to scale without compromising performance.
• System Logging Migration from MongoDB to Elasticsearch: Spearheaded the migration of the platform's logging system from MongoDB to Elasticsearch, removing the overhead of maintaining an additional database, and providing a more streamlined solution. This change not only reduced system costs, but also enhanced query flexibility, offering powerful features like full-text search and aggregation, which improved the overall system's performance and data retrieval capabilities.
• Comprehensive Document Lifecycle & Workflow Management: Developed advanced document lifecycle management features, such as customizable workflows that track a document's progress through various stages (e.g., under review, accepted, rejected). This allowed users to easily monitor and manage the status of each document in real time, improving visibility and control over document workflows, and helping users efficiently manage document processing at scale.
• Derivation Feature for Enhanced Data Post-Processing: Implemented a derivation feature that enables users to post-process data extracted from documents. This feature opened the platform to external integrations, allowing documents to be processed by external systems to achieve desired outputs. It improved the flexibility of the platform and expanded its potential for integration with third-party applications, further enhancing its utility.
• Customizable Data Feed Actions for Real-Time Triggering: Introduced a data feed feature that allows users to trigger specific actions, such as sending emails, executing webhooks, making API requests, or activating custom RPA bots. These actions could be based on user interactions (e.g., document annotation, viewing, stage changes), or system events (e.g., document ingestion). This increased the platform's interactivity and responsiveness, allowing users to automate workflows and create tailored actions based on document status, or user behavior.
• Custom Export Functionality Based on User Templates: Developed a custom export feature, enabling users to export extracted document data in various formats (e.g., XLSX, DOCX, PDF, JSON) based on dynamic templates. Users could create their own templates, customizing the data output to suit their specific needs. This feature improved the platform's flexibility by allowing users to easily extract and share data in formats that best fit their use cases.
• Integrated Drive Storage Solution Using Elasticsearch and Azure: Designed and implemented a custom drive storage solution that combines the capabilities of Elasticsearch with Azure storage to create a highly scalable and efficient document storage system. This solution ensured fast access to documents while leveraging cloud infrastructure for reliable, cost-effective storage.
• Audit logs for enhanced system monitoring: introduced audit logs to track user actions and system changes. These logs are categorized into User , Subscription , and Project-specific actions, providing granular visibility into how the system is being used. The audit trail improved accountability, security, and compliance, ensuring that all user activities could be tracked and reviewed for potential issues or improvements.
• Platform Optimization and Stability Enhancements: Focused on improving platform stability and robustness by optimizing key workflows, implementing bug fixes, and streamlining system processes. These improvements helped ensure smoother performance, reduced downtime, and provided a more reliable user experience, even under high-load conditions.

In my role as a Junior AI Engineer, I was responsible for maintaining and improving the platform's overall performance, ensuring system stability, and implementing new features to enhance the user experience. I worked closely with senior engineers and other cross-functional teams to resolve technical issues, address bugs, and contribute to the ongoing development of the platform.

Key Achievements:

• Optimized System Error Logging: Implemented a MongoDB-based logging system, improving the platform's ability to monitor and audit system errors, significantly enhancing debugging efficiency, and reducing resolution times for technical issues.
• Enhanced User Management: Developed and integrated a user management system, enabling platform administrators to create and manage user accounts efficiently, providing better access control, and improving the overall administrative experience.
• Dynamic Query Interface: Created a user-friendly interface for querying and filtering system logs, allowing for more flexible and detailed data analysis, which led to faster issue identification and resolution.
• Increased Platform Stability: Identified and resolved critical system bugs, resulting in increased platform uptime and smoother user interactions, thereby enhancing the reliability of the platform.