🎛️ LLM Parameters Demystified — A Beginner's Guide

 Every knob you can turn, explained with one running example

If you've ever tinkered with an LLM API and wondered "What does temperature actually do?" or "When do I use LoRA rank?" — this post is for you.

We'll split every important LLM parameter into two clean buckets and then walk through a single concrete example to see them in action.

---

📚 What We'll Learn

🎨 Bucket 1: Inference Parameters (used at generation time)

Control how the model produces text from an already-trained model.

1. 🌡️ Temperature
2. 🎯 Top-K
3. 🎯 Top-P (Nucleus Sampling)
4. 🚫 Frequency Penalty
5. 🚫 Presence Penalty
6. 📏 Max Tokens
7. 🛑 Stop Sequences
8. 🎲 Seed

🏋️ Bucket 2: Training / Fine-Tuning Parameters (used while teaching the model)

Control how the model learns from data.

1. 📈 Learning Rate
2. 📦 Batch Size
3. 🔄 Epochs
4. 📉 Warmup Steps
5. 🎯 Weight Decay
6. ⏹️ Gradient Clipping
7. 🎯 LoRA Rank (r)
8. ⚖️ LoRA Alpha (α)
9. 💧 LoRA Dropout
10. 💾 Quantization Bits (QLoRA)

Let's go! 🚀

---

🎨 Part 1: Inference Parameters

A Step-by-Step Walkthrough with ONE Example

---

🎬 Our Running Example

Input prompt:

Code

"I love pizza. I love pasta. I love"

Task: Predict the next word.

---

📊 Step 0: What Are Logits?

After the LLM processes your input, its final layer spits out logits — raw, unnormalized scores for every word in the vocabulary. They're NOT probabilities yet — just numbers indicating how "excited" the model is about each word.

For our prompt, let's say the model produces:

Code

Logits (raw scores):
  "love":     9.0   ← model wants to repeat!
  "cooking":  7.0
  "eating":   6.5
  "pizza":    6.0
  "sushi":    5.0
  "cars":     2.0
  ... (49,995 more words)

Higher logit = model thinks it's a better next word. Now the sampling pipeline kicks in.

---

🚫 Step 1: Apply Frequency Penalty

What it does: Penalizes words based on how many times they already appeared.

Formula: new_logit = logit - frequency_penalty × count

Our example — count from prompt: "love" appeared 2 times.

With frequency_penalty = 1.0:

Code

"love":     9.0 - (1.0 × 2)  = 7.0   ← big penalty, appeared 2×
"cooking":  7.0 - (1.0 × 0)  = 7.0   ← new word, no penalty
"eating":   6.5 - (1.0 × 0)  = 6.5
"pizza":    6.0 - (1.0 × 1)  = 5.0   ← appeared 1×, small penalty
"sushi":    5.0 - (1.0 × 0)  = 5.0
"cars":     2.0 - (1.0 × 0)  = 2.0

💡 Intuition: A "tax" that grows each time you reuse a word. Repeat "love" 5 times? Big fat penalty.

---

🚫 Step 2: Apply Presence Penalty

What it does: Flat penalty for any word that has appeared at all (regardless of count).

Formula: new_logit = logit - presence_penalty (if seen)

With presence_penalty = 0.5:

Code

"love":     7.0 - 0.5 = 6.5   ← appeared, flat penalty
"cooking":  7.0 - 0.0 = 7.0   ← never appeared ✅
"eating":   6.5 - 0.0 = 6.5
"pizza":    5.0 - 0.5 = 4.5   ← appeared, flat penalty
"sushi":    5.0 - 0.0 = 5.0
"cars":     2.0 - 0.0 = 2.0

💡 Intuition: A "one-time fee" that encourages exploring new words. It doesn't care how many times, just "have you seen this before?"

New ranking:

Code

cooking (7.0) > love (6.5) = eating (6.5) > sushi (5.0) > pizza (4.5) > cars (2.0)

Notice: "cooking" is now the leader! 🎉

---

🌡️ Step 3: Apply Temperature

What it does: Reshapes the "sharpness" of the distribution. Divides logits before softmax.

Formula: adjusted_logit = logit / temperature

With temperature = 1.0 (default, no change):

Code

cooking: 7.0, love: 6.5, eating: 6.5, sushi: 5.0, pizza: 4.5, cars: 2.0

With temperature = 0.5 (COLD — sharpens):

Code

cooking: 14.0, love: 13.0, eating: 13.0, sushi: 10.0, pizza: 9.0, cars: 4.0

Gaps grow → top choice dominates more.

With temperature = 2.0 (HOT — flattens):

Code

cooking: 3.5, love: 3.25, eating: 3.25, sushi: 2.5, pizza: 2.25, cars: 1.0

Gaps shrink → other words get a real chance.

💡 Intuition: Temperature is a "volume knob for randomness".

• 0.0 → always pick #1 (deterministic, greedy)
• 1.0 → natural randomness
• 2.0 → wild, creative, occasionally bizarre

Let's continue with temperature = 1.0.

---

🎯 Step 4: Apply Top-K Filter

What it does: Keep only the K highest-scoring tokens, discard the rest.

With top_k = 4:

Code

Keep top 4:
  cooking: 7.0 ✅
  love:    6.5 ✅
  eating:  6.5 ✅
  sushi:   5.0 ✅
  pizza:   4.5 ❌ (discarded)
  cars:    2.0 ❌ (discarded)

💡 Intuition: "Only consider the top 4 candidates."

---

🎯 Step 5: Apply Top-P (Nucleus) Filter

What it does: Keep the smallest set of tokens whose probabilities sum to P.

First, we peek at probabilities (softmax of remaining logits):

Code

cooking: 0.42
love:    0.26
eating:  0.26
sushi:   0.06

With top_p = 0.9:

Code

Cumulative sum:
  cooking:  0.42       → keep ✅
  love:     0.68       → keep ✅
  eating:   0.94       → keep ✅ (crossed 0.9)
  sushi:    1.00       → discard ❌

💡 Intuition: "Keep just enough candidates to cover 90% of the probability mass." Adaptively shrinks or grows the pool.

---

🎲 Step 6: Softmax → Probabilities

Convert the remaining logits into a proper probability distribution:

Code

Logits (survivors): cooking: 7.0, love: 6.5, eating: 6.5

Softmax → 
  cooking: 45%
  love:    27%
  eating:  27%
  (sums to 100%)

Visually:

Code

┌──────────────────────────────────────────────┐
│████████████████████ cooking (45%)            │
│████████████ love (27%)                       │
│████████████ eating (27%)                     │
└──────────────────────────────────────────────┘

---

🎲 Step 7: Sample (Weighted Random Pick)

The model spins a "weighted wheel" and picks based on probabilities.

Code

Random number generated: 0.31
     │
     ▼ Falls in "love" range (0.45–0.72)
     │
Picked: "love" ✅

Wait — but "cooking" had the highest probability! Why "love"?

Because sampling is random, weighted by probability. "cooking" wins 45% of the time, but the other 55% goes to "love" or "eating". That's what creates variety.

💡 Set seed = 42 to fix the random number generator → same input always gives same output. Great for testing! 🧪

---

🛑 Step 8: Check Stop Sequences

Not modifying logits — this is a post-check on the generated text.

Code

Generated so far: "I love pizza. I love pasta. I love love"
Stop sequences:   ["\n\n", "END", "User:"]

Does output end with any stop sequence? → No
Continue generating!

---

📏 Step 9: Check Max Tokens & Loop

Code

Tokens generated so far: 1
max_tokens = 50

Under limit? → Yes
Loop back to Step 0 with the new token appended

Update our tracking:

• Now "love" count = 3
• Next iteration, "love" will have an even bigger frequency penalty

---

📊 The Full Inference Pipeline (Visual)

Code

Input: "I love pizza. I love pasta. I love"
                     │
                     ▼
        ┌────────────────────────┐
        │   LLM produces logits  │
        │   (50,000 raw scores)  │
        └────────────┬───────────┘
                     ▼
     🚫 Step 1: Frequency Penalty (subtract based on count)
                     ▼
     🚫 Step 2: Presence Penalty (subtract if seen)
                     ▼
     🌡️ Step 3: Temperature (divide logits)
                     ▼
     🎯 Step 4: Top-K Filter (keep top K)
                     ▼
     🎯 Step 5: Top-P Filter (keep P mass)
                     ▼
     📊 Step 6: Softmax (→ probabilities)
                     ▼
     🎲 Step 7: Sample (weighted random)
                     ▼
     🛑 Step 8: Stop Sequences Check
                     ▼
     📏 Step 9: Max Tokens Check → Loop
                     │
                     ▼
              Next token: "love"

---

🎯 Inference Params — Quick Cheat Sheet

Parameter	Typical Value	Effect
🌡️ Temperature	0.0 – 1.0	Randomness (0 = deterministic)
🎯 Top-P	0.9	Diversity via nucleus
🎯 Top-K	40	Diversity via cutoff
🚫 Frequency Penalty	0.0 – 1.0	Anti-repetition (grows with count)
🚫 Presence Penalty	0.0 – 1.0	Encourages new words
📏 Max Tokens	500 – 4096	Output length
🛑 Stop Sequences	["\n\n"]	Halt condition
🎲 Seed	42	Reproducibility

---

🏋️ Part 2: Training / Fine-Tuning Parameters

Now, before an LLM can generate anything, it has to be trained. These parameters control that phase.

Imagine we're fine-tuning an LLM on cooking recipes. Here's what each parameter does.

---

📈 1. Learning Rate

How big a step to take when updating weights after each training example.

Code

Too small (1e-6):  🐢 Learns painfully slowly
Too large (1e-1):  💥 Overshoots, unstable
Just right (1e-4): ✅ Steady progress

Analogy: Rolling a ball down a hill to find the bottom. Small steps = careful. Big steps = might bounce past the valley.

Typical values:

• Pre-training: 1e-4
• Fine-tuning: 2e-5
• LoRA: 2e-4

---

📦 2. Batch Size

Number of examples processed together per training step.

Code

Batch = 1:   Update weights after every recipe (noisy)
Batch = 32:  Update after 32 recipes averaged (stable)
Batch = 512: Update after 512 recipes (very stable, needs huge GPU)

Trade-off: Bigger batch = smoother learning but more memory. Smaller batch = noisier but fits on small GPUs.

---

🔄 3. Epochs

Number of times the model sees the entire dataset.

Code

Epochs = 1:  Basic pass
Epochs = 3:  Common sweet spot
Epochs = 10: Risk of overfitting (memorizing recipes instead of learning cooking)

Analogy: Rereading a textbook. Once = get the gist. 3 times = solid grasp. 10 times = you memorize page numbers but stop learning.

---

📉 4. Warmup Steps

Gradually ramp up learning rate at the start to avoid early instability.

Code

Step 0:    LR = 0
Step 100:  LR = 5e-5   
Step 500:  LR = 2e-4   ← fully warmed up
Step 1000+: LR decays back down

Analogy: A car doesn't slam into 5th gear from a standstill — it eases up through the gears.

---

🎯 5. Weight Decay

Slightly shrinks weights each step to prevent them from growing too large.

Code

weight_decay = 0.01
→ Every step: weights *= 0.9999
→ Discourages memorization, keeps model general

Analogy: A gentle "pruning" of the model's memory to prevent hoarding useless details.

---

⏹️ 6. Gradient Clipping

Caps how big any gradient update can be to prevent training explosions.

Code

max_grad_norm = 1.0
→ If gradient > 1.0, scale it down
→ Prevents "boom!" moments where loss spikes to infinity

---

🎯 7. LoRA Rank (r) — The Star of Fine-Tuning!

Instead of updating all billions of weights, LoRA adds tiny adapter matrices.

Code

Original weight W: (4096, 4096) = 16.8 M params (frozen ❄️)

LoRA adds two small matrices (trained 🔥):
   A: (4096, r) and B: (r, 4096)

If r = 8:
   A + B = 65,536 params  (256× smaller!)

Final: W_new = W + A · B

Visually:

Code

      Big Matrix W (frozen)          Small A · B (trained)
      ┌──────────────────┐            ┌──┐    ┌────────────┐
      │                  │      +     │  │  × │            │
      │  4096 × 4096     │            │r │    │   r × 4096 │
      │                  │            │  │    │            │
      └──────────────────┘            └──┘    └────────────┘
         16.8M params                    ~65K params (r=8)

Rank r	Capacity	Use Case
4	Tiny	Style tweaks
8	Small	Simple tasks (default)
16	Medium	Domain adaptation
64	Large	Complex domains
128+	Huge	Near full fine-tuning

💡 Intuition: Rank = "how much room" the adapter has to store new knowledge.

---

⚖️ 8. LoRA Alpha (α)

Scaling factor for LoRA updates.

Code

W_new = W + (α / r) × A · B

Rule of thumb: alpha = 2 × rank. Higher alpha = stronger LoRA influence on the model.

---

💧 9. LoRA Dropout

Randomly zero out some LoRA activations during training to prevent overfitting.

Code

dropout = 0.0  → no dropout
dropout = 0.1  → common default

Analogy: A basketball team practicing with random players sitting out — makes everyone versatile.

---

💾 10. Quantization Bits (QLoRA)

Compress model weights to fewer bits to fit in less GPU memory.

Code

FP32 (32-bit): full precision — huge
FP16 (16-bit): half precision — 2× smaller
INT8  (8-bit): 4× smaller
INT4  (4-bit): 8× smaller (QLoRA!) — fits Llama-3-8B in 4GB 💪

Trade-off: Less precision = slight quality loss, but massive memory savings.

---

🎯 Training Params — Quick Cheat Sheet

Parameter	Typical Value	Effect
📈 Learning Rate	1e-5 – 2e-4	Speed of learning
📦 Batch Size	8 – 64	Stability of gradient
🔄 Epochs	1 – 3	Passes over data
📉 Warmup Steps	100 – 500	Ease into training
🎯 Weight Decay	0.01	Prevent overfitting
⏹️ Gradient Clipping	1.0	Prevent explosions
🎯 LoRA Rank	8 – 64	Adapter capacity
⚖️ LoRA Alpha	2 × rank	Adapter strength
💧 LoRA Dropout	0.05 – 0.1	Prevent overfit
💾 Quantization	4-bit / 8-bit	Memory savings

---

🎬 The Big Picture

Code

┌─────────────────────────────────────────────────────────────┐
│                                                              │
│  🏋️  TRAINING TIME             🎨  INFERENCE TIME            │
│  ────────────────              ─────────────────              │
│                                                              │
│  📈 Learning rate              🌡️ Temperature                │
│  📦 Batch size                 🎯 Top-K / Top-P              │
│  🔄 Epochs                     🚫 Frequency penalty          │
│  📉 Warmup                     🚫 Presence penalty           │
│  🎯 Weight decay               📏 Max tokens                 │
│  ⏹️ Gradient clip              🛑 Stop sequences             │
│  🎯 LoRA rank                  🎲 Seed                       │
│  ⚖️ LoRA alpha                                               │
│  💧 LoRA dropout                                             │
│  💾 Quantization                                             │
│                                                              │
│  Goal: teach model              Goal: control how it talks   │
│                                                              │
└─────────────────────────────────────────────────────────────┘

---

💡 Real-World Recipes

🤖 Production Chatbot (with RAG)

YAML

temperature: 0.3
top_p: 0.9
max_tokens: 500
frequency_penalty: 0.3
presence_penalty: 0.2

✍️ Creative Writing

YAML

temperature: 0.9
top_p: 0.95
max_tokens: 2000
frequency_penalty: 0.5

🧮 Code / Math

YAML

temperature: 0.0
top_p: 1.0
max_tokens: 1000

🔧 LoRA Fine-Tuning on Custom Data

YAML

learning_rate: 2e-4
batch_size: 16
epochs: 3
lora_rank: 16
lora_alpha: 32
lora_dropout: 0.1
quantization: 4-bit  # QLoRA

---

🎯 TL;DR

• Inference params shape logits → probabilities → sampling. Master temperature, top_p, frequency_penalty and you can control 90% of output behavior.
• Training params shape how weights update. Master learning_rate, batch_size, LoRA rank and you can fine-tune any model on any domain.
• Same architecture underneath — you're just tweaking dials on the machine. 🎛️

---

✨ Pro tip: Start with defaults (temp=0.7, top_p=0.9, LoRA rank=8). Only change one knob at a time and observe. That's how the pros tune LLMs. 🚀