I have set up openclaw on my slightly older server that has no GPU, but 64GiB RAM. I tried with various different models, and I think I now found something that works ok.

The problem is of course that most models are censored, and I can only run small models of which most don’t work that well with openclaw. I have arrived at ollama/huihui_ai/Qwen3.6-abliterated:35b, which does work quite well. It’s a mixture of experts model, so responds fast even though it is quite big. Responses to simple prompts come after a minute, more complex maybe take ten, or sometimes even more. But for me that’s fine, given that I’m just using unused computation.

One issue with this model is of course that it is Chinese, and it does for example avoid the topic of the Tiananmen Square Massacre. But I’m not intending to talk about this kind of heavy stuff with it, so it’s not that bad. And when asked directly, it also searches the web for information about it.

For roleplaying it works quite well so far. I am still testing out the tool calling capabilities.

Before I tried ollama/huihui_ai/gemma-4-abliterated:26b, but while it talked nicely, it was bad at calling tools and updating e.g. the SOUL.md and IDENTITY.md file, and also tried to write information about me into the IDENTITY.md file instead of the USER.md file. So I looked for alternatives.

But I am interested in your setups. What models work well for you, and what kind of hardware do you use for them? Do you have any experience to share, or any tips you can give?

  • tal@lemmy.todayEnglish
    0·
    4 days ago

    I’ve already got a 128GB Framework Desktop that I picked up prior to the memory shortage for AI compute stuff, but if I didn’t…

    In 2028, I expect memory prices to start coming down.

    The GPU hardware itself isn’t really the limiting factor, but rather available memory.

    I just don’t think that I’d be willing to put in massive amounts of effort trying to work around hardware limitations (or spend a bunch of money, which is the other route, outbidding people for the limited supply of memory) to get a local AI compute setup going in 2026 or 2027 when the hardware situation should change substantially in the not-so-distant future.

    Specifically on generating literary erotica, my own experience is that quality is substantially better with larger models. I’ve read that MoE models, like Qwen, are not very good at it, and my limited experimentation with Qwen supports that, though I haven’t tried to aggressively track the latest versions.

    If you look at this thread:

    https://www.reddit.com/r/SillyTavernAI/comments/1vp24rj/qwen_38_27b_is_the_next_stepping_stone_for_local/

    …while the OP in that thread is talking about using Qwen 3.6 for ERP, other users are not generally all that enthusiastic about Qwen 3.6.

    In general, I think that /r/SillyTavernAI favors /u/TheDrummer’s models—which are not merely abliterated base models, but ERP-oriented finetunes—for ERP.