I fine-tuned a small 1.2B model to make better tool calls. On 500 held-out examples, exact-argument accuracy went from 62.8% to 72.8%. That was the easy part.
The hard part was trusting the numbers. Halfway through, a second benchmark said the fine-tuned model had become worse at general tool calling. It hadn’t. My scorer was wrong. This post is about both.
Why I did it
Agents depend on tool calls. A model picks a function, fills in the arguments, and your code runs it. If one argument is wrong, the whole step fails.
Big models are good at this. I wanted to know how far a small model can go with a cheap, focused fine-tune, and how carefully I need to measure to believe the result.
What I tested
- Model: LiquidAI’s LFM2.5-1.2B-Instruct.
- Training data: ToolACE, about 10,000 tool-calling conversations. I trained only on the first assistant turn.
- Method: a LoRA adapter (rank 16), trained for 1,200 steps on a single A10G GPU in about 28 minutes.
- Main test: 500 held-out ToolACE examples. A call counts as correct only if the tool names and every argument match exactly.
- Second test: 500 examples from BFCL, a different benchmark, to check that the model did not get worse outside its training data.
Every gain comes with a 95% paired bootstrap confidence interval. Both models are scored on the same resampled examples, so the interval shows whether a gain is real or just luck in which examples I picked.
Results
| Model | ToolACE exact arguments | BFCL arguments |
|---|---|---|
| Base LFM2.5-1.2B | 62.8% | 87.8% |
| + ToolACE LoRA | 72.8% | 90.4% |
| + Mixed-data LoRA | 72.6% | 90.0% |
On ToolACE, the LoRA gained +10.0 points, CI [+6.6, +13.6]. The whole interval is above zero, so this gain is real. The correct-tool-name rate also went from 78% to 89%.
On BFCL, the gain is small: +2.6 points, CI [+0.2, +5.0]. That is only just above zero, so I call it tentative. The important part is that it did not go down.
I also compared against bigger models on an earlier 200-example split. After training, the 1.2B model (71%) beat Qwen3-4B (65%) and Llama-3.1-8B (48.5%) on this task. These models are 3 to 7 times larger.
The bug that faked a regression
My first BFCL numbers looked bad. With my original scorer, the base model got 59.6% and the fine-tuned model got 54.2%. That looked like a 5-point drop: the fine-tune seemed to make the model worse at general tool calling.
I almost wrote that as a finding. Then I checked the examples it marked wrong.
BFCL does not give one correct answer per argument. It gives a list of accepted answers. An empty string in that list means the argument is optional. My converter kept only the first non-empty answer from each list and threw the rest away.
Example (simplified):
BFCL accepts: unit: ["miles", "mi", ""]
Model wrote: unit="mi" -> correct
My scorer saw: expected "miles" -> wrongSo a correct answer like "mi" was marked wrong. A model that correctly skipped an optional argument was also marked wrong. The fine-tuned model was hit harder by this, so the bug looked like a regression.
After fixing the scorer, the drop disappeared: base 87.8%, fine-tuned 90.4%.
This was not the only bug. Llama first scored 35.5%, because my parser read only the first tool call when the model made several. After that fix, it scored 48.5%. Both times, the model was fine and my measurement was wrong. When a result surprises you, read the failing examples before you believe the number.
What did not work, or is not proven
- Mixed data did not help. I added Hermes and Glaive tool-calling data to fix the “regression.” Because the regression was fake, the mix had nothing to fix. It scored the same as ToolACE-only.
- One training seed. I did not measure how much the results change across runs.
- Data size is a rough signal. With 1,000 and 3,000 examples, the scores were 67.5% and 70%. Most of the gain came early, but these were ordered prefixes from one run, not random samples.
- These are my local metrics, not official BFCL leaderboard scores.
What I learned about evaluation
- The scorer is code, and code has bugs. Test it like the model.
- Look at examples, not only averages. Both of my bugs were obvious after reading ten failures.
- Report a confidence interval. “+10 points” means little without one.
- Write down the limits. One seed and a non-random ablation are fine if you say so.
Links
The adapter is public on Hugging Face, with a model card that lists the training setup, the scores, and a runnable example.
- Code, data prep, and full write-up: toolcall-distill on GitHub
- Fine-tuned adapter: rahatneuron/lfm25-1.2b-toolace-control-s7
Experiment date: October 31, 2026.
ToolACE: Liu et al., ToolACE: Winning the Points of LLM Function Calling (2024), arXiv:2409.00920. Base model: LiquidAI/LFM2.5-1.2B-Instruct, LFM Open License v1.0.