Oogabooga and other local LLM's

[alsoknownas-stefan]

Hi there,

I am using the Dialogue System Addon for OpenAI. It has been working great for me. I want to venture towards local LLM's though. I got Oogabooga (LLM platform) running locally. Koboldccp is also an option for a local LLM platorm. See for how I installed it. And they run pretty good, no issues.

In the manual of Dialogue System Addon for OpenAI, it say this:
OpenAI API Base URL & Other Services
The Dialogue System Addon for OpenAI connects to the OpenAI API at
https://api.openai.com/v1/ by default. If you want to use a different service, such as Microsoft
Azure, that complies to the same API, you can set OpenAI.BaseURL. The URL should end with a
forward slash ( / ).

According to the devs of Oogabooga, their API is meant to be a drop-in replacement to the OpenAI API, including Chat and Completions endpoints.
https://github.com/oobabooga/text-gener ... OpenAI-API

Koboldccp also claims to use OpenAI formatting:


I am pretty sure I configured Oogabooga right and it seems to be running nicely. I got the --api and --listen flags turned on (openai flag is turned on automatically on the sessions tab and --openai does not have to be added if you launch Oogabooga through a .bat) on the sessions tab in the webinterface (and rebooted Oogabooga after that, the webinterface gives you a button to do that; You must also select and load a model on the models tab). Sillytavern, a popular interface, seems to play nicely with Oogabooga on /127.0.0.1:5000/ (which seems to be the default for Oogabooga's OpenAI-compatible API URL) and I can chat with my characters just fine.

I want to play the Runtime Demo scene of Dialogue System Addon for OpenAI and push the button Text Input Conversation. In the welcome window of your addon in Unity, I add an OpenAI key. I add the OpenAI key in Runtime AI Conversation settings Script on the Basic Standard Dialogue UI gameobject (because otherwise, I would get an error after pushing the button Text Input Conversation that I need to add an API key there). This works, I press the button Text Input Conversation and have a nice dialogue with Private Hart.

Now, I want to connect to my local LLM. I tick override Base URL in the welcome window.

I enter http://127.0.0.1:5000/ , start the scene and push the Text Input Conversation button. Unity gives me the following trace (sometimes, it gives me another error first that I need to add the OpenAI key in Runtime AI Conversation settings Script on the Basic Standard Dialogue UI gameobject; I think enabling the Override Base url tick erases the API key in the Runtime AI Conversation settings Script; so I add the key to the script component and then push the Text Input Conversation Button and then get the following error):

HTTP/1.1 404 Not Found
{"detail":"Not Found"}
UnityEngine.Debug:Log (object)
PixelCrushers.DialogueSystem.OpenAIAddon.OpenAI/<>c__DisplayClass38_0:<SubmitChatAsync>b__0 (UnityEngine.AsyncOperation) (at Assets/Plugins/Pixel Crushers/Dialogue System Addon for OpenAI/Scripts/OpenAI/OpenAI.cs:258)
UnityEngine.AsyncOperation:InvokeCompletionEvent ()

which is odd, since Sillytavern can connect just fine on this url.

I try http://127.0.0.1:7910/, which is the web interface, which gives me the same trace

I try http://0.0.0.0:5000/ or http://0.0.0.0:7910/, which gives me the following trace:

InvalidOperationException: Insecure connection not allowed
UnityEngine.Networking.UnityWebRequest.BeginWebRequest () (at <e3fc8d90ba1040059febf63741118cfe>:0)
UnityEngine.Networking.UnityWebRequest.SendWebRequest () (at <e3fc8d90ba1040059febf63741118cfe>:0)
PixelCrushers.DialogueSystem.OpenAIAddon.OpenAI.SubmitChatAsync (System.String apiKey, PixelCrushers.DialogueSystem.OpenAIAddon.Model model, System.Single temperature, System.Single top_p, System.Single frequency_penalty, System.Single presence_penalty, System.Int32 maxTokens, System.Collections.Generic.List`1[T] messages, System.Action`1[T] callback) (at Assets/Plugins/Pixel Crushers/Dialogue System Addon for OpenAI/Scripts/OpenAI/OpenAI.cs:252)
PixelCrushers.DialogueSystem.OpenAIAddon.RuntimeAIConversation.StartFreeformTextInputConversation () (at Assets/Plugins/Pixel Crushers/Dialogue System Addon for OpenAI/Scripts/OpenAI/Runtime/RuntimeAIConversation.cs:359)
PixelCrushers.DialogueSystem.OpenAIAddon.RuntimeAIConversation.Play () (at Assets/Plugins/Pixel Crushers/Dialogue System Addon for OpenAI/Scripts/OpenAI/Runtime/RuntimeAIConversation.cs:117)
UnityEngine.Events.InvokableCall.Invoke () (at <44f3679c53d1477a9c6e72f269e3a3a9>:0)
UnityEngine.Events.UnityEvent.Invoke () (at <44f3679c53d1477a9c6e72f269e3a3a9>:0)
UnityEngine.UI.Button.Press () (at ./Library/PackageCache/com.unity.ugui/Runtime/UGUI/UI/Core/Button.cs:70)
UnityEngine.UI.Button.OnPointerClick (UnityEngine.EventSystems.PointerEventData eventData) (at ./Library/PackageCache/com.unity.ugui/Runtime/UGUI/UI/Core/Button.cs:114)
UnityEngine.EventSystems.ExecuteEvents.Execute (UnityEngine.EventSystems.IPointerClickHandler handler, UnityEngine.EventSystems.BaseEventData eventData) (at ./Library/PackageCache/com.unity.ugui/Runtime/UGUI/EventSystem/ExecuteEvents.cs:57)
UnityEngine.EventSystems.ExecuteEvents.Execute[T] (UnityEngine.GameObject target, UnityEngine.EventSystems.BaseEventData eventData, UnityEngine.EventSystems.ExecuteEvents+EventFunction`1[T1] functor) (at ./Library/PackageCache/com.unity.ugui/Runtime/UGUI/EventSystem/ExecuteEvents.cs:272)
UnityEngine.EventSystems.EventSystem:Update() (at ./Library/PackageCache/com.unity.ugui/Runtime/UGUI/EventSystem/EventSystem.cs:530)

Welp, I have run out of ideas. If anyone has any ideas, please respond.

[Tony Li]

Hi,

As a test, try manually setting OpenAI.BaseURL in a script before clicking the conversation button. Maybe BaseURL isn't getting updated for runtime.

Code: Select all

void Start()
{
    OpenAI.BaseURL = "https://127.0.0.1:5000/";
}

[alsoknownas-stefan]

Hi Tony,

As your suggestion, I added a script to an empty gameobject in the scene.

using UnityEngine;
using PixelCrushers.DialogueSystem.OpenAIAddon;

public class SetBaseURLOpenAI : MonoBehaviour
{
void Start()
{
OpenAI.BaseURL = "https://127.0.0.1:5000/";
}
}

I get the following error when pushing the Text Input Conversation button:

Unable to complete SSL connection

UnityEngine.Debug:Log (object)
PixelCrushers.DialogueSystem.OpenAIAddon.OpenAI/<>c__DisplayClass38_0:<SubmitChatAsync>b__0 (UnityEngine.AsyncOperation) (at Assets/Plugins/Pixel Crushers/Dialogue System Addon for OpenAI/Scripts/OpenAI/OpenAI.cs:258)
UnityEngine.AsyncOperation:InvokeCompletionEvent ()

In the terminal of Oogabooga, I get the message 'Invalid HTTP request received'.

So, it seems to find the LLM service now, but we are not quite there yet. I suspect that this is due to SSL certificates. Above I used https and then it finds the service. The service is not configured for SSL, though, which is probably why Oogabooga responds with 'Invalid HTTP request received'. If I use http in the script above, then it says 404 again.

I am running locally and do not want to make a public api url through Cloudfare in order to enable https or such things if I can avoid it (In the documentation of Oogabooga, the developers say that this is an option, but they do not explain it in detail). I suspect that Unity webrequests require SSL.

I found this discussion, maybe it helps: https://discussions.unity.com/t/unitywe ... /717472/97

EDIT: So, I went down the SSL certificate rabbit hole. I installed OpenSSL, made a key and a certfile that have CN 127.0.0.1 ( yes, I know this is a self-signed certificate that Unity might still reject). I booted up Oogabooga with --ssl-keyfile "pathtokeyfile\key.pem" --ssl-certfile "pathtocertfile\cert.pem" refering to my SSL certificates on the disk. I can curl just fine and get a correct response from the LLM: curl -X POST https://127.0.0.1:5000/v1/chat/completions -H "Content-Type: application/json" -d "{ \"model\": \"gpt-3.5-turbo\", \"messages\": [{ \"role\": \"user\", \"content\": \"Hello!\" }] }" -k

{"id":"chatcmpl-1738489604768838656","object":"chat.completion","created":1738489604,"model":"L3-8B-Stheno-v3.2-Q5_K_M-imat.gguf","choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"Hello! It's nice to meet you. Is there something I can help you with, or would you like to chat?"}}],"usage":{"prompt_tokens":44,"completion_tokens":26,"total_tokens":70}}

Note that I use //127.0.0.1:5000/v1/chat/completions, which seems to be the correct API url for chat completions, so I modify the script below to https://127.0.0.1:5000/v1/chat/completions (i also tried https://127.0.0.1:5000/v1/chat/completions/ with a forward slash and https://127.0.0.1:5000/ and https://127.0.0.1:5000 since it being a base url might be that i only have to add the first part of the url; also tried https://127.0.0.1:5000/v1/ and https://127.0.0.1:5000/v1/chat/)

using UnityEngine;
using PixelCrushers.DialogueSystem.OpenAIAddon;

public class SetBaseURLOpenAI : MonoBehaviour
{
void Start()
{
OpenAI.BaseURL = "https://127.0.0.1:5000/v1/chat/completions";
}
}

Still, no luck. I get an new error that is Curl error 60: Cert verify failed. Certificate is not correctly signed by a trusted CA. UnityTls error code: 7, when clicking the Text Input Conversation button during runtime, which could still indicate that self signed certificates are not allowed with Unity Webrequests. The terminal of Oogabooga shows no incoming request, which might mean that Unity throws the error before the message gets send.

EDIT #2:
I got it haha
I added the certfile.pem to Trusted Root Certification Authorities in certmgr.msc of windows. I then used https://127.0.0.1:5000/v1, and since this works, this means that self signed SSL certificates do not seem to be a problem locally.

using UnityEngine;
using PixelCrushers.DialogueSystem.OpenAIAddon;

public class SetBaseURLOpenAI : MonoBehaviour
{
void Start()
{
OpenAI.BaseURL = "https://127.0.0.1:5000/v1";

}
}

I got local LLM working with your addon Now to see if I can do this without the use of a SSL certificate. It might just be that I need to use http://127.0.0.1:5000/v1

EDIT #3:
My suspicions were correct. I do not need SSL certificates. Just need to launch Oogabooga with --api --listen

and then the script below attached to an empty gameobject:

using UnityEngine;
using PixelCrushers.DialogueSystem.OpenAIAddon;

public class SetBaseURLOpenAI : MonoBehaviour
{
void Start()
{
OpenAI.BaseURL = "http://127.0.0.1:5000/v1";
}
}

huzzah, local LLM integration for everyone using Oogabooga,
I suspect that KoboldCCP and other local LLM OpenAI API derivatives require a similar setup.

EDIT #4:
I tried GPT4ALL, whi has an OpenAI style API and if you turn on Local API server in the settings of GPT4ALL and adjust the port number to 5000 (does not have to be this, but I changed it for convenience so that I can use OpenAI.BaseURL = "http://127.0.0.1:5000/v1"; ) and turn on localdocs in GPT4ALL, then you can chat with your documents through the Dialogue System for Unity addon for OpenAI. For example, I added a pdf (a bill for something) to the localdocs directory that you can setup in GPT4ALL.

curl -X POST http://127.0.0.1:5000/v1/chat/completions -H "Content-Type: application/json" -d "{ \"model\": \"Llama 3 8B Instruct\", \"messages\": [{ \"role\": \"user\", \"content\": \"What do I have to pay according to the bill?\" }] }" -k

In Unity, I can just ask the question through the text prompt box in the Runtime Demo scene of the addon for OpenAI and get an answer back from Private Hart about my document.

[Tony Li]

Awesome! That is some great work you did. I'm going to save your steps and write them up in a general-purpose format that will be helpful for others who want to tie into their own local LLMs. (Ollama won't need any extra steps since it will be handled natively.)

[alsoknownas-stefan]

Credits go to you, haha. You gave the clue that was needed and built all the functionality in your asset in the first place. I only did some testing.

I did not know you are working on an Ollama integration. Good to know. Do you have an estimate on when you are releasing that?

Also, do you plan an Elevenlabs Flash integration (eleven_flash_v2_5 and eleven_flash_v2)? I want to make really fast responding text to speech AI chatbots.

I am also interested in an auto record/auto send feature that submits my speech to text automatically, basically opens and closes the microphone automatically maybe based on Db levels. Do you have something like that in the works?

[Tony Li]

Those should all be in the next release, scheduled tentatively for February 17.

[alsoknownas-stefan]

Very exciting I cannot wait to play with these features.

Another question: Do you plan a OpenAI Vision API integration? I am seeing AI chatbots that have vision and can interpret their surroundings.

Example: at 55 seconds.

I also found this snippet of unity code for the vision api:
https://dev.to/1geek/unity-openai-vision-and-voice-3930

[Tony Li]

I hadn't been thinking about it. I've been focused mostly on text and speech dialogue. But I can add it to the consideration list as something to research.