Self-Hosting Llama

[Sevam]

Running heavy open-source models isn't a task for beginners. Configuring the right hardware environment, selecting optimal GPU instances, and setting up an inference framework that doesn't crash under load requires genuine systems administration skills. If you make a mistake in your server architecture, your cloud hosting bills will be disproportionately high compared to the actual output, and your project will fail.

I want to use this thread to collect our shared technical blueprints for hosting Llama models smoothly:

What bare-metal specs or cloud GPU setups (RunPod, Vast.ai, AWS, custom VDS) are you utilizing? What is the cheapest configuration to run a Llama 70B model with acceptable generation speeds?
What software stack are you running under the hood? Ollama, vLLM, TGI, or LM Studio? Which one handles concurrent API requests from multiple scraping scripts the best?
How badly does your local setup trail behind OpenAI's endpoints in tokens per second? Is the absolute privacy and lack of boundaries worth the infrastructure maintenance headache?

If you've successfully anchored a private Llama instance into your web production setup, share your roadmap with the community. Where should a beginner start?