It’s been a while! Let us know what you’re currently working on, what exploded, what cool stuff you found, or just brag that everything is running fine unattended.
I haven’t been doing a lot, renovate auto upgrades for patches and sometimes minor updates work great so far. I got involved in my union to spin up a mastodon server soon, so there’s a new project, but I already run mastodon so it shouldn’t be much of a deal, I just need to get comfortable with the infrastructure. I also set up ntfy but only sent some test notifications and didn’t configure any app yet - planned for Element Chat.
[Meta] I’m not sure how to apply the new tags. In the recent versions of this thread, people sometimes posted about their ollama setup or the like, but I’m not mentioning AI in the post body (except this meta thingy) and it shouldn’t be much of the discussion, but it’s an open post. Maybe curbstickle can chime in to let me know if I should apply one of them.
You must log in or # to comment.
I’ve had an eventful week, I setup a laptop as a Proxmox Backup Server, moved my VMs storage to my NAS so I could reconfigure the Proxmox server’s local LVM, and now I have qocw2 storage instead of raw disk files! It was super stressful, but seeing my incremental backups take less than 5 minutes was sooo worth it.
https://lemmy.world/post/49663813
My old post from my old account.
Since then, I’ve been moving my PC (not server) from arch to nix, and discovered that I was relying on fundamentally broken DNS resolution behaviour on netbird.
Fixed that and also changed my Caddy config to use FQDNs as well.
Apart from that, added Gollum as a wiki server, and the experience has been perfectly amazing!
Still waiting to get my own mini-PC and use that, current shitbox is a desktop in my college.
I thought I’d take a shortcut and use AI to help me setup an arch thing I’m working on. Jokes on me, took two days and in the end the thing was so fucked its irrecoverable. What a waste of a weekend.
Mealie, recipe manager and Social to Mealie. Scrapes recipes from instagram reels, tiktok, youtube shorts.
Bar Assistant, bar and cocktail management. You can add your ingredients and it shows what can you make, you can share a menu to your friends, its cool
I’m really enjoying mealie. I just wish there was an easier way to input nutritional information
I built a dashboard themed around my dog. It uses the Home Assistant API to get various stats from the house. I’m pleased with it!
Care you share any more details? Looks very aesthetically pleasing. I think I should make one themed around my cat.
It’s a PHP container On the front end I’m using Bootstrap and theming it with Catppuccin. The data for the home energy charts are populated from the Home Assistant API. They’re drawn with chart.js. The server stats are pulled from PHP exec commands, and drawn with Bootstrap progress bars. The temperature card also takes data from the Home Assistant API, which also has an endpoint for local weather data. This doesn’t update as often as I’d like. But it compares the outdoor temperature with the reported temperature of my thermostat, and then the messages below that are determined by a comparison of those values. I’m also using HTMX to load each card, and each tab of the energy card separately so that I don’t have no wait on all requests for the dashboard to load. Not that it’s slow - the home assistant API is surprisingly efficient at loading a great deal of logs. When you click my dogs face, she says WOOF WOOF WOOF in a modal to offer me a search with searxng.
There’s not nearly enough dog metrics on that dashboard
I’d add “time to next meal” but she seems to know that already down to the minute!
Setup an XMPP server for family and friends to use. While I really like it, I can’t seem to get Jitsi to play nice for group calls with the Openfire server. Atleast 1 on 1 calls seem to work fine with Dino and Conversations anyway.
I scored an old HP Microserver Gen10 from work with 4 4tb wd reds. Spent the weekend putting in an extra SSD for the OS and testing the drives. So far so good. Now I just need to figure out what I’m going to do woth it…
Building a new server, pinping up the specs to run more serious llm in parallel with all my services.
Waiting for the CPU (an amd Ryzen 9 5950X) that delayed in delivery, and building wooden crate/case to hold everything together with proper ventilation .
Ooo care to share pics of the case design? I’ve been interested in building one for awhile now
[AIT] I know this isn’t everyone’s cup of tea, but I’m excited about Deepseek V4 flash. It’s (for me) the perfect size and architecture to self-host an LLM.
My box (and brain) are chugging through a queue:
-
Figure out why my swap is going crazy, and how to ban processes from it [Done].
-
Figure out why Code OSS is unhappy [Partially Done].
-
Make an ik_llama.cpp iMatrix for Deepseek V4 [Done].
-
Figure out why quantization isn’t working [Done].
-
Make a test IQ2_KL/MXFP4_R8 quant to see how it does squeezed onto my box [in progress].
-
Test. Tune. Inevitably troubleshoot the dozen other things that go wrong. Figure out how much spare RAM that leaves me.
-
Make a higher quality IQ3_KT quantization. This will take all night on my CPU.
-
KLD test both of them vs the full precision, to quantify quantization loss. Likely an overnight task, too.
-
Try merging the new model release with the base model, 50/50, for a less “deep fried” model. imatrix, quant, test.
The goal is to host it on a single RTX 3090, Ryzen 7000 with 128GB RAM, for anyone curious. Though I may try smaller models too, like Laguna S1.
Deepseek V4 flash
What are the advantages of Deepseek V4 flash?
-
It’s just under 300B, trained at FP4; absolutely the perfect size for servers with 128GB-192GB CPU RAM to spare.
-
Its fast. I’m getting 17 tokens/sec on a single RTX 3090 GPU, all experts offloaded to RAM; for a 300B model this smart, that’s crazy fast.
-
Its attention mechanism is cutting edge, good for long context without too much processing time.
-
The benchmarks for coding/agenic usage are absolutely bonkers, within margin of error of frontier models or Deepseek Pro in some cases: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731
-
…Though I don’t put much stock in benchmarks. And I haven’t tested it enough to tell you if it lives up to that hype for specific use cases.
-
Deepseek also publishes its base model. That means I can “unfry” the model with a merge if I have to.
I am afraid of the the model being “overfit” to coding and agenic stuff.
For reference, my previous favorite model was Xiaomi MiMo V2.5 310B. It benched well, but it also feels “smart” outside of benchmarks, like in knowledge of trivia without tool usage/internet access or comprehension of weird questions.
Sweet.
-
-
[AIT] Well, let’s see…nothing really new on the server. Still waiting for something interesting besides *.arr. I did provision a new tower with the following specs:
- CPU: Ryzen 5 5600X
- GPU: ASUS ROG RX 580
- RAM: Came with 32 GB Corsair Vengeance RGB DDR4 / dropped another 96 GB in it.
- SSD: WD Black NVMe M.2 iirc it’s about a 1tb
- Asus motherboard
- Working on acquiring a GeForce RTX 5090 32GB GDDR7 mounted on a OCuLink external dock (tentative - may change)
This, obviously, will be an local AI machine
Be aware that you pay a significant performance penalty for going over occulink.
The hit modest for fully offloaded dense models (like Qwen 27B), but dramatic for hybrid inference of big MoEs.
Even my old 3090 got a noticeable performance gain going from a PCIe 3.0 x16 riser to a PCIe 4.0 one.
Be aware that you pay a significant performance penalty for going over occulink.
And that’s why it’s tentative. The reason I’m leaning towards the dock is that, while the 5090 will fit, physically in the case, I am worried about heat dissipation. Even now, when I fire up a moderate model and run it through some paces, I can see the temps rising fairly quickly. So, if I’m going to drop that kind of cabbage on a 5090, the one thing I don’t want to do is watch it turn into a shrink-i-dink.
Duct it!
I have a 400W 3090 with zero case fans.
Its always cool, because I bought like $10 of weather stripping to “seal” its intakes against the edge of an SFF case. So it’s always sucking in ambient air:
You can also severely undervolt a 5090 to like 250-300W, with very little performance impact. Honestly its stock speeds are kind of crazy.
…But again, you aren’t gaining much over a 4090. A 4090 + DDR4 threadripper would be way faster than a 5090 + occulink, for many reasons. And TR CPUs are pretty reasonably priced compared to GPUs these days.
But even if you do go 5090, I’d highly recommend finding some way to shove it in the case and duct some air into its intakes. Its going to be way faster on a PCIe slot.
You could even get a riser and mount it somewhere else in the case, theoretically.
Shit I deleted my post. Yes I have been enjoying some greenery so you just have to hear me out. I have an old countertop ice maker that stripped it’s gears. The refrigeration loop is fine. Pretty wild how small it is. I was wondering…what if I provisioned some brackets on which to mount a muffin fan of unknown CFM, to blow across the coils thru several sheets of mesh of varying gauges to collect moisture drained off to an unknown location, and duct that indirectly into the computer, like a miniature Liebert server room cooling unit. I worry about the moisture tho. Constructing a desiccant wheel seems improbable. I could reheat the cool air upstream or mix warmer air in with the stream to so the outlet air doesn’t stay at a very low dew point and “rewet” downstream components. Perhaps with a small filament heat coil. HMMmmmmm
Nice. Did you have to sell a house to afford the parts?
LOL Not quite, however, I am choking on the 5090 but as I understand, that’s the better option, and 32 GB seems to be the sweetspot for AI. I actually picked up the tower and original RAM for less than $500 USD.
First of all, I mean zero offense with any purchase decision. A 5090 is very good.
…But if I were paying that kind of money, I’d probably get a 4090 and a new motherboard/CPU instead. Maybe a used DRR4 threadripper system.
Hybrid (CPU + GPU) inference is where it’s at these days. It opens up a whole world of huge MoE models, whereas on an 5090 you are stuck with Qwen 27B.
Having a fast CPU, with lots of RAM channels, with full PCIe bandwidth is much more important for that than having a 5090, where a 3090 or 4090 will get the job done.
Even if pure speed is your primary concern, you can tune a sparse 120B model (like Laguna) to run almost as fast as Qwen 27B on a 5090, and get at-least-good results.
It’s more finicky and involved, though. For sure.
Running an LLM on a 5090 is a task. Hybrid CPU + GPU inference is a hobby.
Hybrid (CPU + GPU) inference is where it’s at these days. It opens up a whole world of huge MoE models, whereas on an 5090 you are stuck with Qwen 27B.
Why you do this to me? LOL Injecting your options. But seriously thank you for the advice. I’m really green in the AI arena. So I’m trying to feel my way around, trying not to spend money on equipment I’ll regret later.
I installed cameras in my condo recently, and I need to buy and install a larger HDD for capture storage. I ordered a 6TB WD Purple off Amazon, only to have it arrive used (it was absolutely not marked used or refurbished.) So I returned it, naturally. I think that the safest bet is just to order it directly from one of their vendors.
Also, I need to tinker with my postgres install on my Unraid server a bit. Right now a couple other services I host on Unraid use it, but moving forward I need it available to other devices on the LAN.
You might try ordering from ebay. I find a lot of vendors sell there, and at least you know you’re actually getting the product from the person you ordered from, rather than a box randomly stuffed with “similar” products from all sellers.
Over the last few weeks I’ve been making the horrifying leap from PHP5 to PHP8. I knew it was going to be a lot of work to upgrade all my various scripts and web pages but I’m nearing the end, just have some random CLI scripts still failing that I have to work through.
I’ve been making a big push this year to get all my servers caught up to supported operating system levels. Debian will truly run forever even when it really shouldn’t. I was happy to finally get my mail servers modernized last month. The old code I had relied on for managing spam scanning and such had been abandoned but I finally discovered rspamd which replaced everything plus does so much more. The first couple weeks saw an awful lot of server emails being marked as spam while I trained the new system, seems to be settling in nicely now though, and I created some new spam-trap mailboxes and sprinkled the addresses around several web pages. Nice to see their activity picking up again.
Slowly uploading my media collection from a slowly dying TrūNAS box up to B2… slowly 😬
Tired. Tired, tired.
A little minor struggle with Immich. It happens sometimes that I cannot upload MOV files. Not all, just some. I’ve tended to circumvent its making big refusals by switching between my reverse proxied address and my local IP address for it but that didn’t work today, and since these are videos of my baby, I’m a little frustrated. I don’t really have the time to dig into that though.
I wonder whether it’s possible to split tunnel traffic bound for the app from web traffic so I can serve it to my family members without exposing it to the web. Right now I don’t trust its security—the authors even state you really shouldn’t—so I have it behind a Cloudflare Zero Trust MFA prompt. But I get comments sometimes from family that it feels like a hoop to jump through since the longest CF will let them stay authorized is 30 days. Not a real ‘today’ thing, but just a general thing. It’s the app I use the most besides Jellyfin, both to free myself from Microsoft, Google and Apple’s monthly subscriptions, but also, to share privately my baby photos with the family so I can protect the kid’s privacy and all that. I wish it was smoother for my end users but I’ve done what I can for now.
I tried migrating to new hardware recently because I wanted to virtualize most everything. I thought there was sense in being able to get snapshots in proxmox for rollback and ease of backup purposes. It ended up that only about half my servers made it over, and Immich and Jellyfin didn’t make the cut because they need a Quadro T400 that lives in the original machine. Chatbot led me to think passing that GPU through virtualization was going to be a major complicating factor.
I’ve got a backup NAS off-site at my parent’s house ~20 miles away but it has been offline since I changed ISPs because my silly ass hard-coded my public IP address into WireGuard’s config (it auto-launches WG on boot) rather than a DNS address that gets dynamic DNS updating and resolves to whatever my home IP is. I should get to make it out to fix that on Thursday, finally. I went to all this trouble to buy and set up a backup NAS and it has been totally without backups for probably two months now.
Back to the subject of Cloudflare, I know they have a laundry list of security things I need to look at and I want to take those things seriously. With the new baby, and with how demoralized I have felt from work, and with of course all the normal obligations of life, I haven’t ever had the time or felt like dealing with it, but I have a degree in cybersecurity and I want to take my home network / home lab security seriously. For protecting myself of course, but also because I think it’ll be a big growing opportunity technically; maybe some stuff I learn can end up on the résumé. I desperately need to get out of my current role and anything that would help the résumé would be good. But there never seems to be time. Or if there is time there is no will.
I had a Cisco 2960x in my basement that I never did much with because I was suspicious it was drawing a lot of power (likely coincidence but it seemed like the electric bill went up every month I tried plugging it in), and it didn’t seem to have enough memory to run without constantly causing me issues. I wanted to learn Cisco switches with it since I don’t get enough hands-on time at work but it just didn’t seem to work out. I was going to sell it on Facebook but never made time to do that either, so today I just put it back in the rack to have one less thing on the floor. Ironically, now we’ve upgraded to 3Gbps Internet and none of my home network stuff supports it. If I’d thought about that I would have likely let the ISP install their own mesh APs but I had put so much money and time into my own setup, I thought, surely I don’t need them putting their hardware in my home. Well, here we are, and neither my little baby switch, nor my Cisco switch, nor even my OPNsense router support 3Gbps and there’s no money to upgrade any of it 🥲
That’s about all I can think of for today. Check in next Sunday, and I bet the update would be precisely the same :pDiscovered you can self host Joplin and I can’t wait to set it up in the next days.
Also I tried some alternatives to ActualBudget, because I didn’t need to many functionalities. Long story short, I went back to it cause none offers a neat interface and multiple accounts at the same time.
