comments:
ThinkPad P52, 96gb DDR4, 1tb ssd with 20tb external, Ubuntu 26.04 RTX Pro 5000 Blackwell 72gb, which I'll admit, I was very fortunate to buy for $6,500 about a month ago (open box). I regret buying this eGPU. I've come to learn that its TB3 connection isn't compatible with the TB3 connection on the P52. So, I'm at TB1 speeds, which is a killer at model loading times (2-3 minutes instead of 7 seconds). I'll be upgrading that shortly. Since I'm new to Blackwell, I've been running NVFP4 models, and I'm pretty shocked by a) how well they perform, 2) the prefill is sub-second even on large prompts, despite the TB1 speed connection, 3) 32 concurrent calls isn't unreasonable. I think my CPU is the gate on concurrency, but 8 concurrent sub-agents are effortless for the card. 4) serving the eGPU via my Tailnet, so I hit the endpoint from every device on my Tailnet. Gemma4_31b-NVFP4 and Qwen3.6_27b_NVFP4. I find the Gemma model to be my "go-to" with thinking on for Hermes. Of course it can't perform as well as frontier models for building, but for routine work with already-built Skills, it's shockingly strong. The context window is an issue when running Hermes. I'm using 131k context length to be conservative on KV cache (concurrency with sub-agents needs headspace). I made this post which has cost breakdown: link