Model specification via OpenRouter
It'd be great to have the ability to select the models used for at least chat myself. I've been working with and testing GLM 5.3 Flash and it's performing surprisingly well on long running tasks and in different domains besides coding, at 10% of the cost of GLM 5.3.
Just an example in this case but being able to configure a model myself would be very helpful.
4 Comments
Sign in to comment
hi Michael!
It's a thing which is still WIP. As I posted in Community channel,
> We don't have a built-in inference — apart from local model for vector embeds, i.e. hybrid/semantic search, but it's not LLM. On local ai/ollama/etc in Anytwo: we haven't figured a good user experience yet. Internally, or asking via bao/agent, it is easy to switch to anything. But giving a good experience of configuration is challenging because of model card differences. It's not a problem to set provider/model/url for inference, but models are different so their behavior must be taken in account in agent internals. So, still WIP and definitely not out of the radar
As for now you can ask bao or another external agent to switch the model, but take into account that it can breakthe agent state. And also keep in mind that bao needs some toolcalling/coding capabilities to do things.
Thanks!
Have you seen this?
Just posted a comment about how I expect to configure local LLMs. There are other reasons for allowing model selection, in the linked post. Some (or most) inference APIs allow you to list available models. Maybe just using that and turning model tier view into selectors based on available models from API?
But for local LLM users and more advanced use cases I believe the target audience is tech-savvy enough (at least today) to make it work with simple text fields, like the ones from my comment.
Regarding model custom parameters, maybe just select the model by id and trust the defaults from the provider.
When configuring local models for integration with other tools sometimes I add many entries in the provider using the same model just to get different settings - each setting with its own ID so the client can choose via API - to work around some parameter not directly supported by the harness
For example: I have many "models" in my API that are actually Gemma 12B, just with different context window sizes, temperature, system prompt, thinking options, etc. Almost anything the harness (in this case Anytwo) can't discover via metadata available in the API itself we can workaround using this approach.
I briefly tested 5.3-flash yesterday -- works great, but I've noticed that bao doesn't count the price of api properly, created an issue for this.
