I realize, I need to upgrade my little NUC to something bigger for higher inference of bigger llama models. I want something that you still can have on your living room’s tv bench, so no monster rack please, but that has also the necessary muscle when needed for llama. Budget doesn’t matter right now, want to understand what’s good and what’s out there. Thanks
Problem with smaller footprint is cooling and how audible it becomes.
One idea is to use fiber optic hdmi cables and a usb extender to hide the pc away in another room.
If you want smaller footprint then the keyword to use is “Unified memory”, it can be reasonable fast for 30B models and a slow thinker mode for 70B ones.