MoE models changed what “fits on my GPU” means, and most hardware advice hasn’t caught up
When I first got into self-hosting my own Large Language Models, I’d use distilled variants of Deepseek-R1 and switch to a different model if it couldn’t handle my productivity tasks. Unfortunately, the only alternatives I could find were in the sub-7B range, as my old GPUs couldn’t accommodate bulkier models without massively slowing down the inference operations.
When I first got into self-hosting my own Large Language Models, I’d use distilled variants of Deepseek-R1 and switch to a different model if it couldn’t handle my productivity tasks. Unfortunately, the only alternatives I could find were in the sub-7B range, as my old GPUs couldn’t accommodate bulkier models without massively slowing down the inference operations.
Sophia Wilson
Atlanta
Atlanta
Published by: aplhsindia.in
