If the model is adequately FOSS, i.e. open weights, and can run on a single consumer GPU (or NPU), and the “author” (quotations because I’m personally undecided if one can claim code generated by an aforementioned model is theirs) understands it, then I really don’t see what the problem is…
Personally I have nothing against SLMs/LLMs as a technologically, to me my grievances against ChatGPT or Claude are mostly about their environmental impacts and selling us back our own art, also keeping knowledge behind a for-profit black-box - if those aren’t appropriate for a specific model, then I say using that model is fair, and good for productivity.
I’d take a model whose training data is open source and legitimately obtained. The only ones I know about are Apertus and OLMo, and they aren’t really competitive.
Unfortunately not yet, no, true FOSS models are likely many years away, but I would argue that that follows typical FOSS lifecycles. Emerging technology is typically outperformed by proprietary endeavors, which creates an audience, and then that audience undertakes a FOSS implementation that initially underperforms, then just about competes, then eventually overtakes (i.e, GNU/Linux).
The problem with open weight models is that they are still hard to deploy. For DeepSeek for example you require NVIDIA HGX B200, these things are expensive.
If you are really using those things for work, a 32GB VRAM Blackwell currently costs around 3800-4000€. My own experience with local models on my 5070/12GB with 64GB DDR4 has shown me:
I can run a Qwen 3.8 35b Q_8 quant (BF16 is a tad too large) using CPU offloading, while still reaching acceptable speeds for private use.
Ternary Models are a game changer. Built to be used in edge computing, Bonsai-27B-Ternary from Prism ML runs completely on my GPU (using ca. 10GB of my 12GB VRAM incl. MMProj, 9 when disabling vision) and even leaves enough VRAM to run subagents (when killing my plasmashell lol -but i tend to offload these to the CPU in favor of increasing the context of the main model). currently you have to install a costum fork of llama.cpp to run ternary models, but the needed changes will be applied upstream soon. (There is a Bonsai quant which aims to run on iPhones with below 6GB VRAM footprint - the limit of what Ios currently allows)
If this scales linear, it would mean that the 32 GB of a smallish Blackwell would be enough to run Ternary models with around 70B parameters plus a ton of context window without breaking a sweat.
These are strange times. I absolutely love the technology and the advances, and i fucking hate the companies pusting these technologies.
Altman, Amodei, Musk, Nadella, Huang and all of the othe AI evangelists can go choke on a bag of dicks in my opinion. This technology never should have been this divisive, it should have just become another tool in humanities toolbox. I fear that these people have tainted the general populations outlook on this tech for a very long time.
If the model is adequately FOSS, i.e. open weights, and can run on a single consumer GPU (or NPU), and the “author” (quotations because I’m personally undecided if one can claim code generated by an aforementioned model is theirs) understands it, then I really don’t see what the problem is…
Personally I have nothing against SLMs/LLMs as a technologically, to me my grievances against ChatGPT or Claude are mostly about their environmental impacts and selling us back our own art, also keeping knowledge behind a for-profit black-box - if those aren’t appropriate for a specific model, then I say using that model is fair, and good for productivity.
I’d take a model whose training data is open source and legitimately obtained. The only ones I know about are Apertus and OLMo, and they aren’t really competitive.
Unfortunately not yet, no, true FOSS models are likely many years away, but I would argue that that follows typical FOSS lifecycles. Emerging technology is typically outperformed by proprietary endeavors, which creates an audience, and then that audience undertakes a FOSS implementation that initially underperforms, then just about competes, then eventually overtakes (i.e, GNU/Linux).
The problem with open weight models is that they are still hard to deploy. For DeepSeek for example you require NVIDIA HGX B200, these things are expensive.
If you are really using those things for work, a 32GB VRAM Blackwell currently costs around 3800-4000€. My own experience with local models on my 5070/12GB with 64GB DDR4 has shown me:
If this scales linear, it would mean that the 32 GB of a smallish Blackwell would be enough to run Ternary models with around 70B parameters plus a ton of context window without breaking a sweat.
These are strange times. I absolutely love the technology and the advances, and i fucking hate the companies pusting these technologies.
Altman, Amodei, Musk, Nadella, Huang and all of the othe AI evangelists can go choke on a bag of dicks in my opinion. This technology never should have been this divisive, it should have just become another tool in humanities toolbox. I fear that these people have tainted the general populations outlook on this tech for a very long time.