Choosing Your First Model

You've learned about where to find models, how size and format impact performance, the role of quantization, and the significance of licenses. Now, let's put that knowledge into practice to select your first Large Language Model for local experimentation. The goal here isn't to find the absolute "best" model, but rather a suitable starting point that works reasonably well on your hardware and allows you to begin interacting with a local LLM.

Match the Model to Your Machine

The most significant factor influencing your first model choice is your computer's hardware, specifically RAM (System Memory) and, if available, VRAM (GPU Memory). As discussed, larger models with more parameters require more memory. Quantization helps reduce this, but hardware limitations remain the primary filter.

• Low RAM / No Dedicated GPU (e.g., < 8GB RAM): You'll likely need to stick to the smallest models available, perhaps in the $1B$ to $3B$ parameter range, and heavily quantized (like Q2 or Q3 GGUF formats). Performance might be slow, but it's a starting point.
• Moderate RAM / Basic GPU (e.g., 8-16GB RAM, < 6GB VRAM): Models in the $7B$ parameter range, especially quantized versions (like Q4 or Q5 GGUF), are often manageable. You might run these primarily on the CPU, potentially with some layers offloaded to the GPU if VRAM allows.
• High RAM / Capable GPU (e.g., 16GB+ RAM, 8GB+ VRAM): You can comfortably run $7B$ models, experiment with $13B$ models (quantized), and potentially even larger ones depending on your specific VRAM amount. Higher VRAM allows more of the model to run on the faster GPU.

The chart below gives a rough idea of the memory footprint for different model sizes using a common quantization level. Remember these are estimates; actual usage depends on the specific model, quantization method, and the software you use.

Estimated memory requirements for running quantized GGUF models (e.g., Q4_K_M). Actual usage can vary based on specific model and software.

Start Small and Quantized

For your first foray into local LLMs, it's highly recommended to start with a smaller model, typically in the $7B$ parameter range, and choose a quantized GGUF version.

Why?

1. Manageability: Smaller, quantized models download faster and require less disk space.
2. Performance: They load more quickly and generate text faster on typical consumer hardware compared to larger, unquantized models.
3. Accessibility: A $7B$ quantized model often strikes a good balance between capability and resource requirements, running adequately even without a powerful GPU.
4. Learning: It provides a good platform to learn the basics of downloading, loading, and interacting with a model without excessive waiting or complex setup.

Look for GGUF files with quantization levels like Q4_K_M or Q5_K_M. These generally offer a good trade-off between reduced size/resource usage and maintaining model quality. You can find these on model repositories like Hugging Face, often provided by community members who specialize in creating these optimized formats (searching for terms like "GGUF" alongside a base model name is effective).

Check the Model's Purpose and License

When browsing models (e.g., on Hugging Face), pay attention to the model card:

• Intended Use: Look for models described as "chat" or "instruct" models. These are fine-tuned to follow instructions and carry on conversations, making them ideal for getting started. Avoid models designed for highly specific tasks (like code generation only, or medical text analysis) unless that's your specific goal.
• License: Double-check the model's license. For initial experimentation and personal use, many popular models have permissive licenses (like Apache 2.0, MIT, or specific Llama/Mistral licenses). Ensure the license allows for your intended use, especially if you plan to build anything more than simple tests.

Example Starting Points (Model Families)

While specific model recommendations change rapidly, here are types of models that often make good starting points in the $7B$, quantized GGUF format:

• Mistral-based models: Models derived from Mistral AI's releases (like Mistral 7B) are known for strong performance relative to their size. Look for instruct-tuned GGUF versions.
• Llama-based models: Meta's Llama models (Llama 2, Llama 3) form the basis for many fine-tuned variants. Again, look for $7B$ instruct or chat GGUF versions.
• Phi-based models: Microsoft's Phi models offer good capabilities in smaller sizes ($~3B$ parameters). Check for chat-tuned GGUF formats if available.

Always prioritize finding the GGUF quantized version of these base models, usually available through community contributors on Hugging Face.

Expect to Iterate

Choosing your first model is just that, a first step. Don't worry about getting it perfect immediately. Download a candidate model based on the criteria above. In the next chapter, you'll learn how to run it.

• If it runs smoothly, great! Start experimenting.
• If it's too slow or uses too much memory, try a model with fewer parameters or a more aggressive quantization level (e.g., Q3 instead of Q4).
• If it doesn't seem to understand instructions well, perhaps try a different instruct-tuned model from the same family or a different family altogether.

Experimentation is a normal part of working with these models.

Decision Checklist

To summarize, when choosing your first model:

1. Assess Hardware: Note your available RAM and VRAM.
2. Target Size: Aim for a smaller model initially (e.g., $7B$ parameter range).
3. Format: Look for GGUF format.
4. Quantization: Choose a balanced quantization level (e.g., Q4_K_M).
5. Purpose: Select a model fine-tuned for 'chat' or 'instruct' based on its model card.
6. License: Verify the license permits your intended use.
7. Download: Get the model file (usually from Hugging Face).

With these steps, you'll be well-prepared to select a model that allows you to successfully run your first local LLM in the next chapter.