Quick Introduction We have all been on forums, chats, reddit, discord, youtube, or somewhere and heard “Oh! Model XYZ is AMAZEBALLZ!zomgwtfbbq” then downloaded it (or more likely, some quantized form of it) and said “eww… This sucks!” This post is going to be a rather technical series of experiments to demonstrate the impact of implementation-specific hazards with inference. I will be using the term “reference implementation” to describe the lab that published and offers first-party hosting of ...
    • frongt@lemmy.zipEnglish
      0·
      3 days ago

      Not any more than any other local service. Ollama makes it really easy to get started. The ministral models are small but still useful.

        • frongt@lemmy.zipEnglish
          0·
          3 days ago

          Yeah, I’m just saying it’s a really easy way to get started, without being overwhelmed by formats and quantizations and top-k and stuff. You run it, it goes, then you can move to something better.

          • brucethemoose@lemmy.worldEnglish
            0·
            3 days ago

            Its sleazy software, and IMO its defaults provide a really terrible 1st time experience. It convinces 1st time users that locally run LLMs are terrible, and goes out of its way to be incompatible with other stuff.

            I think LM Studio would be a much better “1st time” recommendation. Its not perfect either, but its lightyears better, not an active con like ollama, and its smoother to “graduate” from it to croco.cpp or llama.cpp or something.

    • brucethemoose@lemmy.worldEnglish
      0·
      3 days ago

      You shouldn’t let that discourage you. VLLM is really finicky, but there are easier backends to use (though which one depends on your hardware).