Oogabooga and other local LLM's
Posted: Sat Feb 01, 2025 5:04 pm
Hi there,
I am using the Dialogue System Addon for OpenAI. It has been working great for me. I want to venture towards local LLM's though. I got Oogabooga (LLM platform) running locally. Koboldccp is also an option for a local LLM platorm. See for how I installed it. And they run pretty good, no issues.
In the manual of Dialogue System Addon for OpenAI, it say this:
OpenAI API Base URL & Other Services
The Dialogue System Addon for OpenAI connects to the OpenAI API at
https://api.openai.com/v1/ by default. If you want to use a different service, such as Microsoft
Azure, that complies to the same API, you can set OpenAI.BaseURL. The URL should end with a
forward slash ( / ).
According to the devs of Oogabooga, their API is meant to be a drop-in replacement to the OpenAI API, including Chat and Completions endpoints.
https://github.com/oobabooga/text-gener ... OpenAI-API
Koboldccp also claims to use OpenAI formatting:
I am pretty sure I configured Oogabooga right and it seems to be running nicely. I got the --api and --listen flags turned on (openai flag is turned on automatically on the sessions tab and --openai does not have to be added if you launch Oogabooga through a .bat) on the sessions tab in the webinterface (and rebooted Oogabooga after that, the webinterface gives you a button to do that; You must also select and load a model on the models tab). Sillytavern, a popular interface, seems to play nicely with Oogabooga on /127.0.0.1:5000/ (which seems to be the default for Oogabooga's OpenAI-compatible API URL) and I can chat with my characters just fine.
I want to play the Runtime Demo scene of Dialogue System Addon for OpenAI and push the button Text Input Conversation. In the welcome window of your addon in Unity, I add an OpenAI key. I add the OpenAI key in Runtime AI Conversation settings Script on the Basic Standard Dialogue UI gameobject (because otherwise, I would get an error after pushing the button Text Input Conversation that I need to add an API key there). This works, I press the button Text Input Conversation and have a nice dialogue with Private Hart.
Now, I want to connect to my local LLM. I tick override Base URL in the welcome window.
I enter http://127.0.0.1:5000/ , start the scene and push the Text Input Conversation button. Unity gives me the following trace (sometimes, it gives me another error first that I need to add the OpenAI key in Runtime AI Conversation settings Script on the Basic Standard Dialogue UI gameobject; I think enabling the Override Base url tick erases the API key in the Runtime AI Conversation settings Script; so I add the key to the script component and then push the Text Input Conversation Button and then get the following error):
HTTP/1.1 404 Not Found
{"detail":"Not Found"}
UnityEngine.Debug:Log (object)
PixelCrushers.DialogueSystem.OpenAIAddon.OpenAI/<>c__DisplayClass38_0:<SubmitChatAsync>b__0 (UnityEngine.AsyncOperation) (at Assets/Plugins/Pixel Crushers/Dialogue System Addon for OpenAI/Scripts/OpenAI/OpenAI.cs:258)
UnityEngine.AsyncOperation:InvokeCompletionEvent ()
which is odd, since Sillytavern can connect just fine on this url.
I try http://127.0.0.1:7910/, which is the web interface, which gives me the same trace
I try http://0.0.0.0:5000/ or http://0.0.0.0:7910/, which gives me the following trace:
InvalidOperationException: Insecure connection not allowed
UnityEngine.Networking.UnityWebRequest.BeginWebRequest () (at <e3fc8d90ba1040059febf63741118cfe>:0)
UnityEngine.Networking.UnityWebRequest.SendWebRequest () (at <e3fc8d90ba1040059febf63741118cfe>:0)
PixelCrushers.DialogueSystem.OpenAIAddon.OpenAI.SubmitChatAsync (System.String apiKey, PixelCrushers.DialogueSystem.OpenAIAddon.Model model, System.Single temperature, System.Single top_p, System.Single frequency_penalty, System.Single presence_penalty, System.Int32 maxTokens, System.Collections.Generic.List`1[T] messages, System.Action`1[T] callback) (at Assets/Plugins/Pixel Crushers/Dialogue System Addon for OpenAI/Scripts/OpenAI/OpenAI.cs:252)
PixelCrushers.DialogueSystem.OpenAIAddon.RuntimeAIConversation.StartFreeformTextInputConversation () (at Assets/Plugins/Pixel Crushers/Dialogue System Addon for OpenAI/Scripts/OpenAI/Runtime/RuntimeAIConversation.cs:359)
PixelCrushers.DialogueSystem.OpenAIAddon.RuntimeAIConversation.Play () (at Assets/Plugins/Pixel Crushers/Dialogue System Addon for OpenAI/Scripts/OpenAI/Runtime/RuntimeAIConversation.cs:117)
UnityEngine.Events.InvokableCall.Invoke () (at <44f3679c53d1477a9c6e72f269e3a3a9>:0)
UnityEngine.Events.UnityEvent.Invoke () (at <44f3679c53d1477a9c6e72f269e3a3a9>:0)
UnityEngine.UI.Button.Press () (at ./Library/PackageCache/com.unity.ugui/Runtime/UGUI/UI/Core/Button.cs:70)
UnityEngine.UI.Button.OnPointerClick (UnityEngine.EventSystems.PointerEventData eventData) (at ./Library/PackageCache/com.unity.ugui/Runtime/UGUI/UI/Core/Button.cs:114)
UnityEngine.EventSystems.ExecuteEvents.Execute (UnityEngine.EventSystems.IPointerClickHandler handler, UnityEngine.EventSystems.BaseEventData eventData) (at ./Library/PackageCache/com.unity.ugui/Runtime/UGUI/EventSystem/ExecuteEvents.cs:57)
UnityEngine.EventSystems.ExecuteEvents.Execute[T] (UnityEngine.GameObject target, UnityEngine.EventSystems.BaseEventData eventData, UnityEngine.EventSystems.ExecuteEvents+EventFunction`1[T1] functor) (at ./Library/PackageCache/com.unity.ugui/Runtime/UGUI/EventSystem/ExecuteEvents.cs:272)
UnityEngine.EventSystems.EventSystem:Update() (at ./Library/PackageCache/com.unity.ugui/Runtime/UGUI/EventSystem/EventSystem.cs:530)
Welp, I have run out of ideas. If anyone has any ideas, please respond.
I am using the Dialogue System Addon for OpenAI. It has been working great for me. I want to venture towards local LLM's though. I got Oogabooga (LLM platform) running locally. Koboldccp is also an option for a local LLM platorm. See for how I installed it. And they run pretty good, no issues.
In the manual of Dialogue System Addon for OpenAI, it say this:
OpenAI API Base URL & Other Services
The Dialogue System Addon for OpenAI connects to the OpenAI API at
https://api.openai.com/v1/ by default. If you want to use a different service, such as Microsoft
Azure, that complies to the same API, you can set OpenAI.BaseURL. The URL should end with a
forward slash ( / ).
According to the devs of Oogabooga, their API is meant to be a drop-in replacement to the OpenAI API, including Chat and Completions endpoints.
https://github.com/oobabooga/text-gener ... OpenAI-API
Koboldccp also claims to use OpenAI formatting:
I am pretty sure I configured Oogabooga right and it seems to be running nicely. I got the --api and --listen flags turned on (openai flag is turned on automatically on the sessions tab and --openai does not have to be added if you launch Oogabooga through a .bat) on the sessions tab in the webinterface (and rebooted Oogabooga after that, the webinterface gives you a button to do that; You must also select and load a model on the models tab). Sillytavern, a popular interface, seems to play nicely with Oogabooga on /127.0.0.1:5000/ (which seems to be the default for Oogabooga's OpenAI-compatible API URL) and I can chat with my characters just fine.
I want to play the Runtime Demo scene of Dialogue System Addon for OpenAI and push the button Text Input Conversation. In the welcome window of your addon in Unity, I add an OpenAI key. I add the OpenAI key in Runtime AI Conversation settings Script on the Basic Standard Dialogue UI gameobject (because otherwise, I would get an error after pushing the button Text Input Conversation that I need to add an API key there). This works, I press the button Text Input Conversation and have a nice dialogue with Private Hart.
Now, I want to connect to my local LLM. I tick override Base URL in the welcome window.
I enter http://127.0.0.1:5000/ , start the scene and push the Text Input Conversation button. Unity gives me the following trace (sometimes, it gives me another error first that I need to add the OpenAI key in Runtime AI Conversation settings Script on the Basic Standard Dialogue UI gameobject; I think enabling the Override Base url tick erases the API key in the Runtime AI Conversation settings Script; so I add the key to the script component and then push the Text Input Conversation Button and then get the following error):
HTTP/1.1 404 Not Found
{"detail":"Not Found"}
UnityEngine.Debug:Log (object)
PixelCrushers.DialogueSystem.OpenAIAddon.OpenAI/<>c__DisplayClass38_0:<SubmitChatAsync>b__0 (UnityEngine.AsyncOperation) (at Assets/Plugins/Pixel Crushers/Dialogue System Addon for OpenAI/Scripts/OpenAI/OpenAI.cs:258)
UnityEngine.AsyncOperation:InvokeCompletionEvent ()
which is odd, since Sillytavern can connect just fine on this url.
I try http://127.0.0.1:7910/, which is the web interface, which gives me the same trace
I try http://0.0.0.0:5000/ or http://0.0.0.0:7910/, which gives me the following trace:
InvalidOperationException: Insecure connection not allowed
UnityEngine.Networking.UnityWebRequest.BeginWebRequest () (at <e3fc8d90ba1040059febf63741118cfe>:0)
UnityEngine.Networking.UnityWebRequest.SendWebRequest () (at <e3fc8d90ba1040059febf63741118cfe>:0)
PixelCrushers.DialogueSystem.OpenAIAddon.OpenAI.SubmitChatAsync (System.String apiKey, PixelCrushers.DialogueSystem.OpenAIAddon.Model model, System.Single temperature, System.Single top_p, System.Single frequency_penalty, System.Single presence_penalty, System.Int32 maxTokens, System.Collections.Generic.List`1[T] messages, System.Action`1[T] callback) (at Assets/Plugins/Pixel Crushers/Dialogue System Addon for OpenAI/Scripts/OpenAI/OpenAI.cs:252)
PixelCrushers.DialogueSystem.OpenAIAddon.RuntimeAIConversation.StartFreeformTextInputConversation () (at Assets/Plugins/Pixel Crushers/Dialogue System Addon for OpenAI/Scripts/OpenAI/Runtime/RuntimeAIConversation.cs:359)
PixelCrushers.DialogueSystem.OpenAIAddon.RuntimeAIConversation.Play () (at Assets/Plugins/Pixel Crushers/Dialogue System Addon for OpenAI/Scripts/OpenAI/Runtime/RuntimeAIConversation.cs:117)
UnityEngine.Events.InvokableCall.Invoke () (at <44f3679c53d1477a9c6e72f269e3a3a9>:0)
UnityEngine.Events.UnityEvent.Invoke () (at <44f3679c53d1477a9c6e72f269e3a3a9>:0)
UnityEngine.UI.Button.Press () (at ./Library/PackageCache/com.unity.ugui/Runtime/UGUI/UI/Core/Button.cs:70)
UnityEngine.UI.Button.OnPointerClick (UnityEngine.EventSystems.PointerEventData eventData) (at ./Library/PackageCache/com.unity.ugui/Runtime/UGUI/UI/Core/Button.cs:114)
UnityEngine.EventSystems.ExecuteEvents.Execute (UnityEngine.EventSystems.IPointerClickHandler handler, UnityEngine.EventSystems.BaseEventData eventData) (at ./Library/PackageCache/com.unity.ugui/Runtime/UGUI/EventSystem/ExecuteEvents.cs:57)
UnityEngine.EventSystems.ExecuteEvents.Execute[T] (UnityEngine.GameObject target, UnityEngine.EventSystems.BaseEventData eventData, UnityEngine.EventSystems.ExecuteEvents+EventFunction`1[T1] functor) (at ./Library/PackageCache/com.unity.ugui/Runtime/UGUI/EventSystem/ExecuteEvents.cs:272)
UnityEngine.EventSystems.EventSystem:Update() (at ./Library/PackageCache/com.unity.ugui/Runtime/UGUI/EventSystem/EventSystem.cs:530)
Welp, I have run out of ideas. If anyone has any ideas, please respond.