Inference lab
llama.cpp, Ollama, vLLM and Strata on two machines: an old rig at home with an RTX 4080 Super, and a Lambda Vector workstation at the office with an RTX 4000 Ada.
llama.cpp, Ollama, vLLM and Strata, on a home rig and an office workstation. Started with quantized 8B to 12B models; now running more capable quantized 27B dense or 125B MoE models, depending on the job.
Hardware
- Home: an old rig. AMD Ryzen 7 5700 on an ASRock A320M/ac board, 32 GB RAM, a 512 GB PCIe Gen3 x4 NVMe SSD, and an NVIDIA RTX 4080 Super with 16 GB.
- Office: Lambda Vector workstation. NVIDIA RTX 4000 Ada Generation with 20 GB, and 128 GB RAM.
- Extended with Hugging Face. I extend the lab with rented GPUs on Hugging Face Spaces.