Training a Small Multilingual LLM
Training a Small Multilingual LLM for Structured Order Extraction (English, German, Arabic) on Google Colab, for CPU Deployment
Contents
Abstract
Online stores receive free-text order requests in several languages ("I want a red T-shert, size m, to be delivered to mainstreet 34, 23456 Hamburg, I would like to pay in mastercard/paypal"). This article describes a complete, reproducible recipe for turning such messages into a fixed JSON object (brand, product, color, size, quantity, deliver_address, payment_method) with a model that is small enough to run on a 2–4 core CPU server. We fine-tune Qwen3-0.6B, a 0.6-billion-parameter open-weight model with Apache-2.0 licence and 100+ language coverage, using Low-Rank Adaptation (LoRA) on the free Google Colab T4 GPU. Because no labelled data exists for a specific store, we generate training data synthetically from a catalogue configuration: a structured order is sampled first, then rendered into English, German or Arabic text with language-specific grammar (German adjective declension, Arabic gender agreement, Arabic-Indic digits) and realistic noise (typos, casing, missing fields, chit-chat). Labels are therefore correct by construction. The model is trained with a completion-only loss on 6,000 examples, evaluated with field-level exact match on a hand-written 45-message gold set, merged, converted to GGUF and quantized for llama.cpp. The pipeline was validated end-to-end in a 2-vCPU sandbox; the GPU training run itself is executed by the reader on Colab, and the article explains how to read its results. Article 2 covers serving the exported model on CPU with grammar-constrained JSON decoding.
Keywords: instruction fine-tuning, LoRA, synthetic data, information extraction, multilingual NLP, Qwen3, GGUF, llama.cpp, Google Colab
1.Introduction
1.1 The problem
A customer writes a message in their own words and language; the shop system needs a machine-readable order. The classical approach — regular expressions and keyword lists per language — breaks on the first typo ("T-shert"), on word order ("zwei schwarze Hoodies von Nordwind in XL"), on grammatical agreement ("سترة حمراء" versus "تيشيرت أحمر") and on mixed scripts (an Arabic sentence containing a German address in Latin letters).
1.2 Goals and constraints
The requirements set for this study are:
1. Three input languages
English, German, Arabic (including dialectal phrasing such as "أبغى", "بدي", "عايز").
2. One fixed output schema
Exactly seven keys, returned strictly formatted as valid JSON.
3. Store-specific behaviour
Product, colour and payment names normalised to the store's catalogue vocabulary.
4. Small hardware
CPU-only server with 2–4 cores & 4–8 GB RAM; training fits in free Google Colab.
5. A learning objective
The recipe teaches how to fine-tune and ship a model for a specific domain, justifying every design choice.
1.3 Contributions
The article contributes:
- (i) Schema design with explicit normalisation rules.
- (ii) Configuration-driven synthetic-data generator that produces grammatically plausible messages in three languages with correct labels.
- (iii) LoRA fine-tuning recipe for Qwen3-0.6B running on a T4 GPU in well under an hour.
- (iv) Evaluation protocol measuring what is actually shipped (quantized GGUF on CPU).
- (v) Tested notebook and script set validated end-to-end.
2.Background and Related Work
Instruction fine-tuning
Fine-tuning a pre-trained language model on (instruction, answer) pairs makes it follow the instruction format reliably, including for unseen inputs of the same kind [1]. For a single narrow task, a few thousand high-quality examples are sufficient; Zhou et al. showed that 1,000 carefully curated examples can align a 65B model [2], and the task here is far narrower than open-ended chat.
Parameter-efficient fine-tuning
LoRA freezes pre-trained weights and learns a low-rank update ΔW = BA for selected weight matrices, cutting trainable parameters by orders of magnitude without quality loss [3]. QLoRA extends this to 4-bit quantized base models [4]; for a 0.6B model this is unnecessary as full-precision weights fit comfortably.
Synthetic training data
Self-Instruct demonstrated instruction data can be generated rather than annotated [5]. We use a stricter variant: the label (structured order) is sampled first and text is generated from it, so labels are exact by construction with zero annotator error.
Constrained decoding
A fine-tuned model usually emits valid JSON, but "usually" is not a guarantee. Grammar-constrained decoding masks every token violating a formal grammar [6] and works even without fine-tuning [7]. llama.cpp converts a JSON Schema into such a grammar [8]. We combine both: fine-tuning gives correct content, the grammar gives correct structure.
Small multilingual models
Qwen3 family covers 119 languages with an Apache-2.0 0.6B dense model [9][10]. Gemma 3 270M is positioned by Google for task-specific fine-tuning [11]. Qwen3.5 adds 0.8B–9B hybrid Gated-DeltaNet/attention models [12]. Section 4.1 compares them for this task.
Quantization
Running on CPU requires integer quantization. For Qwen3, 8-bit quantization is reported as near-lossless, while 4-bit shows noticeable degradation, and smaller models are less stable under quantization than larger ones [13]. This drives the Q8_0 recommendation in Section 4.6.
Alternative: token classification
An encoder such as XLM-R [14] fine-tuned for span tagging would be faster, but extracts spans verbatim and cannot normalise "rot" to red or "Nachnahme" to cash_on_delivery without a second rule layer. A generative model does extraction and normalisation in one step.
3.Task Definition
3.1 Input and output
Input: one customer message (plain text, any of the three languages, mixed scripts allowed).
Output: one JSON object with exactly the seven keys below, always in this order.
| Field | Type | Normalisation rule | Example |
|---|---|---|---|
brand | string | null | Spelled as in the catalogue ("Nordwind"), null if not mentioned | "Nordwind" |
product | string | null | Store's English catalogue key, lowercase ("t-shirt", "hoodie", "sneakers") | "t-shirt" |
color | string | null | English catalogue colour, lowercase; rot, أحمر, red all become "red" | "red" |
size | string | null | Letter code XS…XXL in upper case, or a number as a string ("42"); Arabic-Indic digits converted | "M" |
quantity | integer | null | Explicit count; singular noun with article ("a T-shirt", "ein T-Shirt", "تيشيرت") = 1; plural without number = null | 1 |
deliver_address | string | null | Copied verbatim from the message (whitespace collapsed), no translation or reformatting | "mainstreet 34, 23456 Hamburg" |
payment_method | list of ids | Ids from closed set (mastercard, visa, paypal, credit_card, cash_on_delivery, bank_transfer, klarna, invoice); [] if none | ["mastercard", "paypal"] |
{"brand": null, "product": "t-shirt", "color": "red", "size": "M", "quantity": 1,
"deliver_address": "mainstreet 34, 23456 Hamburg", "payment_method": ["mastercard", "paypal"]}3.2 Design decisions
items list once the basic recipe works.4.Method
4.1 Base model selection
Selection criteria (in order of importance): Arabic and German quality, CPU footprint after quantization, licence, and tool chain maturity (transformers, PEFT/TRL, llama.cpp, Ollama).
| Model | Params | Languages | Licence | Architecture / Notes | Fit for this task |
|---|---|---|---|---|---|
| Qwen3-0.6B [10] | 0.6B (0.44B non-embed) | 100+ | Apache-2.0 | Dense decoder, text-only, thinking mode switchable off; universal tool support | Selected |
| Qwen3.5-0.8B [12] | 0.8B | 201 | Apache-2.0 | Hybrid Gated DeltaNet + attention, native vision; needs source transformers | Second Iteration |
| Gemma 3 270M [11][23] | 0.27B (0.17B embed) | 140+ | Gemma terms, gated | Designed for fine-tuning; smallest CPU footprint; weaker free-form Arabic | Edge / Pi |
| SmolLM2-360M [24] | 0.36B | Primarily English | Apache-2.0 | English-centric pre-training | Not suitable |
| Llama 3.2 1B [25] | 1.2B | 8 official (no Arabic) | Llama 3.2 licence | Arabic not officially supported | Not suitable |
Decision: Qwen3-0.6B
Confidence: 80%The remaining 20% accounts for Qwen3.5-0.8B giving potentially better Arabic at a similar CPU footprint once its toolchain matures. Qwen3-0.6B's thinking mode must be turned off via enable_thinking=False to prevent unwanted <think> prefix tokens.
4.2 Prompt format: identical at training and inference
Chat models are trained on specific token layouts; using a different layout at inference degrades them badly [15]. We freeze the prompt format string in extractor_common.py:
<|im_start|>system
{system prompt}<|im_end|>
<|im_start|>user
{customer message}<|im_end|>
<|im_start|>assistant
<think>
</think>
The system prompt is 148 tokens and never changes, which enables llama.cpp to cache its KV state across all customer requests.
4.3 Synthetic data generation
Principle: Sample the answer first, render the question from it. Because text is derived from the label, labels are correct by construction.
Catalogue as Configuration
catalog.json holds brands, products with surface forms per language, colours, sizes, payment methods, German cities and street names. Replacing this file adapts the model to another shop.
Noise & Negative Examples
30% typos, 20% lowercased, shuffled clauses. 7% chit-chat negative examples (all null) prevent hallucinating an order from casual inquiries ("Wo ist mein Paket?").
- German adjective declension: Handles gender and article ("ein rotes T-Shirt", "einen roten Hoodie", "eine rote Jacke").
- Arabic gender agreement: Color adjective follows noun gender ("تيشيرت أحمر" vs "سترة حمراء") and non-human plurals take feminine singular ("جوارب سوداء").
- Arabic-Indic digits: Sizes/quantities like "مقاس ٤٤" mapped to ASCII "42" while verbatim address preserves original digits.
- Mixed scripts: 70% of Arabic orders feature Latin-script German delivery addresses.
- Dialects: Openers such as "أبغى", "بدي", "عايز", "محتاج" in addition to Modern Standard Arabic.
4.4 Fine-tuning with LoRA
LoRA trains matrices A and B with rank r=16 on all linear attention and MLP projections, yielding ~10M trainable parameters (≈1.7% of the model).
| Hyper-parameter | Value | Reason |
|---|---|---|
Rank r / α / dropout | 16 / 32 / 0.05 | Standard; α = 2r keeps update scale moderate |
Target modules | all attention & MLP projections | Best quality per parameter in QLoRA study [4] |
Learning rate | 2e-4 (cosine, 3% warmup) | LoRA tolerates higher learning rate than full FT |
Epochs / batch | 3 epochs, batch 16 (1,125 steps) | Conservative budget for 6,000 short examples; keeps best checkpoint |
Max sequence length | 512 tokens | 99th percentile is 309 tokens, max 341; avoids truncation |
Loss | completion-only | Learn to answer, not predict customer text |
Precision | fp32 + fp16 autocast (T4) / bf16 (Ampere+) | Prevents fp16 overflow without full memory penalty |
Seed | 42 | Reproducibility |
4.5 Evaluation protocol
Greedy decoding (temperature 0) evaluates outputs on five metrics:
- JSON validityShare of outputs that parse and contain all seven required keys.
- Field accuracyExact match per field (
nullcounts as a valid value). - Exact match (Overall)All seven fields correct simultaneously — the core business metric.
- Exact match per languageReveals whether Arabic or German lags behind English.
- LatencyMean and 95th percentile inference time per message.
4.6 Export for CPU
Adapter weights are merged via merge_and_unload(), saved in bf16, converted to GGUF using llama.cpp, and quantized.
Recommendation: Ship Q8_0
Confidence: 85%8-bit quantization is near-lossless (~0.6 GB file). The 240 MB savings of Q4_K_M (~0.4 GB) introduces measurable accuracy degradation and is only recommended for systems with < 2 GB RAM.
5.Step-by-Step on Google Colab
The companion notebook train_order_extractor.ipynb executes these steps in 45–60 minutes on free tier T4 GPU:
Runtime Setup
Select Runtime → Change runtime type → T4 GPU. Verify in Colab:
!nvidia-smi -LPinned Libraries
pip install -q "transformers==5.19.0" "trl==1.15.0" "peft==0.21.2" "datasets==5.1.0" "accelerate==1.15.0" "pydantic>=2.7"Project Files
All modular scripts written to disk via %%writefile.
FIELDS = ["brand", "product", "color", "size", "quantity", "deliver_address", "payment_method"]
PROMPT_TEMPLATE = (
"<|im_start|>system\n{system}<|im_end|>\n"
"<|im_start|>user\n{message}<|im_end|>\n"
"<|im_start|>assistant\n<think>\n\n</think>\n\n"
)
EOS_TOKEN = "<|im_end|>"
def format_prompt(message, system_prompt=SYSTEM_PROMPT):
return PROMPT_TEMPLATE.format(system=system_prompt, message=message.strip())
def target_json(output):
ordered = {k: output.get(k) for k in FIELDS} # fixed key order
return json.dumps(ordered, ensure_ascii=False, separators=(", ", ": "))Generate Synthetic Data & Gold Test Set
python generate_data.py --n_train 6000 --n_val 600 --out data --seed 42
python make_gold.pySample Generated Data:
→ {"brand": null, "product": "hoodie", "color": "blue", "size": "XS", "quantity": 1, "deliver_address": "Mozartstrasse 79 90462 Nürnberg, Deutschland", "payment_method": []}
→ {"brand": "Elbe & Co", "product": "t-shirt", "color": "white", "size": null, "quantity": 3, "deliver_address": "شارع شيلر 66، 12367 برلين", "payment_method": ["klarna"]}
→ {"brand": "Al Noor", "product": "hoodie", "color": "yellow", "size": "L", "quantity": 3, "deliver_address": "Bahnhofstr. 162, 90474 Nürnberg", "payment_method": []}
Zero-Shot Baseline Evaluation
python evaluate.py --backend hf --model Qwen/Qwen3-0.6B --data data/gold_test.jsonl --report reports/baseline_gold.jsonLoRA Training & Adapter Merge
python train.py --base_model Qwen/Qwen3-0.6B --data data --out models/order-extractor \
--epochs 3 --batch 16 --lr 2e-4 --rank 16 --alpha 32Evaluation (GPU, Hugging Face format)
python evaluate.py --backend hf --model models/order-extractor/merged --data data/val.jsonl --limit 300 --report reports/finetuned_val.json
python evaluate.py --backend hf --model models/order-extractor/merged --data data/gold_test.jsonl --report reports/finetuned_gold.jsonExport to GGUF and Quantize
bash export_gguf.sh models/order-extractor/merged models order-extractorEvaluate CPU Shipping Candidate
pip install -q llama-cpp-python
python evaluate.py --backend gguf --model models/order-extractor-Q8_0.gguf --data data/gold_test.jsonl --threads 2 --report reports/gguf_q8_gold.json
python evaluate.py --backend gguf --model models/order-extractor-Q4_K_M.gguf --data data/gold_test.jsonl --threads 2 --report reports/gguf_q4_gold.jsonPreserve Artefacts
Download or sync to Google Drive: order-extractor-Q8_0.gguf (~600 MB), lora_adapter/, and reports/.
6.Results
6.1 How to report
Fill the evaluation table from the generated reports/*.json files:
| System | JSON valid | Exact match (45 gold) | EN | DE | AR | Latency (mean) |
|---|---|---|---|---|---|---|
| Qwen3-0.6B zero-shot (GPU) | — | — | — | — | — | — |
| Fine-tuned HF bf16 (GPU) | — | — | — | — | — | — |
| Fine-tuned GGUF Q8_0 + grammar fallback (2-core CPU) | 100 % | — | — | — | — | — |
| Fine-tuned GGUF Q4_K_M + grammar fallback (2-core CPU) | 100 % | — | — | — | — | — |
Performance Targets:
- Field accuracy > 95% for catalogue items (product, color, size, payment).
- Errors concentrate in
deliver_addressandquantity(e.g. plural counting). - GGUF Q8_0 performance should track HF bf16 within 1–2 percentage points.
6.2 Pipeline validation (measured)
The full pipeline was verified end-to-end on 2 vCPUs using a proxy model. Key validation fact: with the grammar fallback enabled (--grammar always), the model returned a parsable JSON object with all seven keys and valid catalogue payment IDs in 100% of requests — proving that structure comes from the grammar and content comes from training.
7.Discussion and Limitations
Synthetic-to-real gap
Template text is cleaner than customer reality. Gold set acts as defense; post-launch active learning mitigates drift.
Catalogue drift
Serving layer checks values against the live DB; unrecognized products trigger retraining in minutes.
Hallucination
Constrained by negative training examples, explicit null labels, and grammar validation.
Address handling
Verbatim copying preserves leading zeros in German postal codes and Arabic street strings.
Multiple items
Schema v1 supports 1 item. v2 can switch item keys to an items: [] array of objects.
Privacy & GDPR
No customer data used for training. CPU local inference ensures zero customer addresses are sent to 3rd party APIs.
8.Conclusion
A store-specific, trilingual order-extraction model can be built in an afternoon without any annotated data:
- Catalogue configuration defines vocabulary.
- Grammar-aware generator produces training pairs with correct labels by construction.
- LoRA fine-tuning of Qwen3-0.6B on free Colab GPU finishes in < 30 minutes.
- Result is a ~0.6 GB GGUF file running locally on a 2–4 core CPU with guaranteed JSON schema conformity.
References
- J. Wei et al, “Finetuned Language Models Are Zero-Shot Learners,”ICLR 2022. arXiv:2109.01652
- C. Zhou et al., “LIMA: Less Is More for Alignment,”NeurIPS 2023. arXiv:2305.11206