NVIDIA ChatRTX User Guide

 

Introduction
ChatRTX is a demo app that lets you personalize a GPT large language model (LLM) connected to your own content—docs, notes, photos, or other data. Leveraging retrieval-augmented generation (RAG), TensorRT-LLM, and RTX acceleration, you can query a custom chatbot to quickly get contextually relevant answers. And because it all runs locally on your Windows RTX PC or workstation, you’ll get fast and secure results.

 

ChatRTX supports various file formats, including text, pdf, doc/docx (with LLMs), and jpeg, gif, and png (with CLIP). Simply point the application at the folder containing your files and it'll load them into the library in a matter of seconds. The ChatRTX tech demo is built from the TensorRT-LLM RAG developer reference project available from github. Developers can use that reference to develop and deploy their own RAG-based applications for RTX, accelerated by TensorRT-LLM.

Prerequisites

 

 

Note: The accuracy and relevance of the responses is determined by specificity of the question being asked, the accuracy of the AI model that is used and the accuracy of the dataset. 

 

The application uses a technique called Retrieval Augmented Generation (RAG) to look up the local files you point it to and use that information to provide context when it submits your question to the LLM.  Disabling RAG will result in the LLM generating responses purely based on the data with which it was originally trained.  In order to see how the LLM would respond without RAG, you can disable RAG by selecting “AI Model Default” from the pulldown menu on the right (see image below)

 

Using the CLIP Vision and Language Model
In addition to the pre-installed Mistral LLM model, you can download and install the CLIP vision and language model from the ‘Add new models’ option. After the model is installed you can point the app to your folder of jpeg images and chat with your images. These pictures don’t have to be tagged. You can ask questions such as “Show me images that have cats in them”,  “Show me  pictures taken outdoors”,  “Show me images that have flowers” and other such questions. The accuracy of responses to your questions is determined by the CLIP model training and accuracy.  

 

Using voice to input your questions
This version of ChatRTX has also integrated the Whisper model that does Audio to Text translation. To use this feature, make sure that the microphone on your system is enabled and click on the ‘mic’ icon and ask your question. When you are done asking your question, click on the ‘stop’ icon to stop the recording.  The application will recognize  and output your question into the chat window. You can then click on ‘Send’ to present the text to the LLM for a response. The Whisper model supports multiple languages such as French, Spanish, Mandarin and others. 

 

Guidelines on query results
The data that ChatRTX loads into the vector library is broken into chunks (you can think of it like a paragraph in a document), which are selected, based on their relevance, to formulate a response to a query. This method of storing the data makes ChatRTX good for queries that request information that is covered in few chunks across the dataset, but not good for queries that involve reasoning about the entire dataset at once. For example, asking for some facts covered in a couple of documents is likely to give better results than asking for a summary of a document or set of documents. 

 

As with most AI use cases, the response quality tends to improve with more data. Pointing ChatRTX at more content about a specific subject will tend to result in better responses. 

 

 

Known Issues and Limitations
The following known issues exist in the current build