Improvements
Triage

Create an "OpenAI compatiable" Provider (LocalAI and more)

What could be better?

Create an "OpenAI compatiable" Provider in the provider picker of Bao that will allow the user to conenct any openai compatible to the agent. So technically alot of providers will support it but MOSTLY! local AI providers.

oMLX, llamacpp-server and etc... all support OpenAI compatible providers.

How so?

Add that above in settings -> Agent -> Model - > Provider - > Change (button).

That will give the user an option to insert API endpoint (URL) and API key (if needed). Then, test to get what models the API is providing the user with and pick from it the relevant model.

2 Comments

Sign in to comment

AL
Alex Lins (Bao)·about 3 hours ago

Thanks for the detailed spec — we see the use case for local LLM providers. A couple of clarifications before we can prioritize this:

  1. Model discovery: when you say "test to get what models the API is providing", do you mean we should hit the /models endpoint (standard in OpenAI-compatible APIs) on connection, or is there a different discovery flow you've seen work better with LocalAI/llamacpp-server?
  2. Auth scope: some local providers need no API key, others do. Should we make the key field optional, or is there a pattern you've found where it's always required for the ones you're targeting?
  3. Fallback behavior: if a user configures an endpoint but it goes down or returns an error at inference time, how should Bao handle it — retry, fail gracefully with a message, or something else?
R
RealityMolder·about 3 hours ago
  1. Yes exactly.
  2. Optional as you said some users don't use API key.
  3. That is up to you 🙂 Maybe give the user the option (just like other apps for security have killswitch feature that wont work if a said flow is needed (a.e VPN) But if the user allows it have fallback models for when his inference / provider isnt working.