You queued the research agent because the tools post told you to. Anything slow belongs on a queue. The docs agree: ->queue() keeps the app “fast and responsive”.
So it runs on a worker now. Same three Firecrawl tools, same database check, plus one crawl start per run. On Haiku a run takes 47 to 98 seconds. On Sonnet the provider calls alone take 53 to 58.
The worker’s default timeout is 60.
Here is one run with --tries=2, which is what a Horizon supervisor gives you unless you say otherwise:
12:56:16 attempt 1 step 1 FirecrawlSearch ×3
12:56:24 attempt 1 step 2 FirecrawlScrape, FirecrawlCrawl
12:56:59 attempt 1 step 3 FirecrawlScrape ×2, FirecrawlCrawl
12:57:18 attempt 1 job.timed_out timeout=60
12:57:18 attempt 1 Worker STOPPED Job timed out
12:57:46 attempt 2 step 1 messages=1
12:57:49 attempt 2 step 1 FirecrawlSearch ×3
12:57:59 attempt 2 step 2 FirecrawlScrape, FirecrawlCrawl
12:58:19 attempt 2 step 3 DatabaseQueryTool ×2
12:58:41 attempt 2 then text_chars=5827Attempt 1 ran seven tools and started two crawls. Then the worker was killed.
Attempt 2 knew none of that. It ran seven more tools and started a third crawl. None of the fourteen calls shared arguments with another, because the model planned from scratch.
Nothing in your app recorded that attempt 1 happened.
The job the SDK dispatches
->queue() does one thing:
return new QueuedAgentResponse(
InvokeAgent::dispatch($this, $prompt, $attachments, $provider, $model)
);And Laravel\Ai\Jobs\InvokeAgent, in full:
class InvokeAgent implements ShouldQueue
{
use Concerns\InvokesQueuedResponseCallbacks;
use Queueable;
public function __construct(
public Agent $agent,
public Decisions|string $prompt = '',
public array $attachments = [],
public Lab|array|string|null $provider = null,
public ?string $model = null) {}
public function handle(): void
{
$this->withCallbacks(fn (): AgentResponse => $this->agent->prompt(
$this->prompt, $this->attachments, $this->provider, $this->model
));
}
}No $timeout. No $tries. No $backoff. No ShouldBeUnique. No Interruptible.
Your agent object, tools and all, serialized into the payload, and one call to prompt().
So the worker decides everything. queue:work defaults to --timeout=60 --tries=1. Horizon defaults to --tries=0, and zero means unlimited, which oussama-mater.tech learned the hard way. Same job, two runtimes, two different disasters.
What happens at second 60
The worker arms pcntl_alarm before every job. When it fires, the handler does four things in order.
It fails the job if this attempt was its last. It dispatches JobTimedOut. It dispatches WorkerStopping with the reason TimedOut. Then it sends SIGKILL to its own process.
With --tries=1, the job is failed first. The SDK’s catch callback runs with a TimeoutExceededException, and then the process dies.
That is the pure-default run, and on Sonnet it is not a maybe. Same agent, queue:work with no flags. Two searches, a scrape, the database check, a crawl started at 13:05:49, then the answer step. The alarm fired at 13:06:34, 78 seconds in, while the model was writing.
catch fired. failed_jobs got a row. The crawl kept running on Firecrawl’s side with nobody left to collect it.
With any other tries, nothing fails the job. The process dies with the job still reserved. retry_after is 90 seconds by default, so 90 seconds after the original reservation another worker picks it up and runs handle() from the top.
In the run above, attempt 1 was reserved at 12:56:16. Attempt 2 started at 12:57:46. Ninety seconds exactly.
Under Horizon’s default, that loop has no end.
Nothing from the run survives
The agent above uses RemembersConversations. A middleware stores its history, and the middleware stores it like this:
return $next($prompt)->then(function (AgentResponse $response) use ($prompt): void {
// ...
$this->store->storeUserMessage(...);
$this->store->storeAssistantMessage(...);
});After $next($prompt) returns. The assistant message is one row, carrying every tool call and every tool result of the whole run as JSON. The docs say it plainly: messages are stored “after each interaction”.
A run that dies at step 3 has not finished an interaction. It writes zero rows. Not the user message, not the seven tool results.
The agent_conversation_messages table had 214 rows before attempt 1 and 214 rows after it.
Attempt 2 loads the conversation, finds nothing, appends the prompt, and starts over.
If your agent is not Conversational, it is worse. The only outputs of a queued run are the then callbacks, and those run after prompt() returns. Kill the worker and the run never happened, except on the provider’s invoice and inside every tool that wrote something.
One of those tools was FirecrawlCrawl, which I wrote. In an earlier run it reported cURL error 28: Operation timed out after 30000 milliseconds. Firecrawl’s active-crawl endpoint listed that crawl 42 seconds after the request started.
The tool told the model it failed. The crawl was running. Timeout on the response, not on the action, and it happened without a worker kill at all.
The step cap is not this bug
You will reach for MaxSteps. It is the wrong tool for this, and it is worth knowing why.
When the loop hits its last step and the model still asks for a tool, the SDK does not run it. It writes a placeholder result instead:
The agent reached its maximum number of steps without running this tool call.The tool result is marked failed, the run returns normally, and everything gets persisted. A step-capped run is a finished run. The cost-math post has a Haiku run that ended exactly this way, with a DatabaseQueryTool call on step 5 that never executed.
The cap protects you from the model. It does nothing about the worker.
The SDK already knows how to do this
Read the human-approval section of the docs and you find this sentence:
Laravel stores the result of an approved tool before asking the model to continue. If generation then fails, the approval has already been resolved.
That is the fix. Persist the tool result before the next model call, so a crash after the tool ran costs nothing on retry.
The implementation is real. On a resume, the loop executes the approved tools before its first step. It hands the results to a recorder that writes them into the paused conversation row, inside a transaction, with lockForUpdate().
The repo has a test named for it: a resume that fails after the tool runs does not re-execute the tool on retry. Tool runs once. Provider returns 500. The same decisions go in again. The tool count is still one.
Now count how many of your agent’s tool calls go through an approval. For the research agent, none.
The SDK checkpoints the one tool call a human clicked “approve” on, and not one of the calls the agent made on its own.
That is not a complaint. It tells you exactly what to build for the other calls: a durable record of the tool result, written before the loop asks the model what to do next.
toolCallId() is the wrong key
The tool Request has a method for this, and its docblock says so:
/**
* Get the stable provider tool-call ID, usable as an external idempotency key.
*/
public function toolCallId(): ?stringStable within one invocation. On an approval resume the same paused turn is replayed, so the id matches.
On a job retry the model plans again and the provider mints new ids. In the run above, attempt 2’s first search was FirecrawlSearch with limit: 8. Attempt 1’s was limit: 5. Different id, different arguments, same intent.
An idempotency key that survives a retry has to come from the job, not from the model. Use the run, the tool, and a hash of the arguments.
Claim before you execute
Wrap each tool. The wrapper inserts a claim row first, runs the tool second, stores the result third. The unique index is the lock.
use Illuminate\Contracts\JsonSchema\JsonSchema;
use Illuminate\Database\UniqueConstraintViolationException;
use Illuminate\Support\Facades\DB;
use Laravel\Ai\Contracts\Tool;
use Laravel\Ai\Tools\Request;
use Laravel\Ai\Tools\ToolNameResolver;
use Stringable;
use Throwable;
class IdempotentTool implements Tool
{
public function __construct(
private readonly Tool $tool,
private readonly string $runKey,
) {}
public function name(): string
{
return ToolNameResolver::resolve($this->tool);
}
public function description(): Stringable|string
{
return $this->tool->description();
}
public function schema(JsonSchema $schema): array
{
return $this->tool->schema($schema);
}
public function handle(Request $request): Stringable|string
{
$args = $request->all();
ksort($args);
$claim = [
'run_key' => $this->runKey,
'tool' => $this->name(),
'args_sha' => sha1(json_encode($args, JSON_UNESCAPED_SLASHES)),
];
try {
DB::table('agent_tool_invocations')->insert([
...$claim,
'args' => json_encode($args, JSON_UNESCAPED_SLASHES),
'status' => 'claimed',
'created_at' => now(),
'updated_at' => now(),
]);
} catch (UniqueConstraintViolationException) {
$existing = DB::table('agent_tool_invocations')->where($claim)->first();
if ($existing->status === 'done') {
return $existing->result;
}
// Claimed, no result: the last attempt died between claim and store.
// The outcome is unknown, so run it again and take over the row.
}
try {
$result = (string) $this->tool->handle($request);
} catch (Throwable $e) {
DB::table('agent_tool_invocations')->where($claim)->update(['status' => 'failed']);
throw $e;
}
DB::table('agent_tool_invocations')->where($claim)->update([
'status' => 'done',
'result' => $result,
'updated_at' => now(),
]);
return $result;
}
}The name() method matters. The SDK resolves a tool’s name from name() when it exists and from the class basename when it does not. Without it, the model would see four tools called IdempotentTool.
The order matters more. Insert, then execute, then update.
Execute first and record second, and a crash between the two leaves no trace, so the retry runs the tool again. buildmvpfast calls this “executing before claiming” and says they have seen it four times in production codebases. Claim first, and the worst case is a claimed row with no result. The retry treats that as unknown and re-runs it. Once, for one tool, instead of for all of them.
The migration is a table with run_key, tool, args_sha, args, status, result, and unique(['run_key', 'tool', 'args_sha']).
Checkpoint before every step, replay on retry
The ledger stops a tool from running twice. It does not stop the retry from paying for steps 1 and 2 again. And it does nothing when the model asks for something slightly different, which it did on every call in the run above.
For that you need the history. The SDK hands it to you before every step:
class StartingStep
{
/**
* @param Message[] $messages The messages being sent for this step, including the tool results of the steps before it.
*/
public function __construct(
public string $invocationId,
public int $stepNumber,
// ...
public array $messages,
// ...
) {}
}That array is the full replayable transcript so far. Store it, keyed by the run:
use Illuminate\Support\Facades\DB;
use Laravel\Ai\Events\StartingStep;
class StepCheckpoint
{
public function starting(StartingStep $event): void
{
$runKey = CurrentRun::key();
if ($runKey === null) {
return;
}
DB::table('agent_step_checkpoints')->updateOrInsert(
['run_key' => $runKey, 'step' => $event->stepNumber + 1],
[
'message_count' => count($event->messages),
'messages' => base64_encode(serialize($event->messages)),
'created_at' => now(),
'updated_at' => now(),
],
);
if (CurrentRun::stopRequested()) {
throw RunInterrupted::beforeStep($event->stepNumber + 1, SIGTERM);
}
}
public static function latest(string $runKey): ?array
{
// Step numbers restart at 1 on every attempt, so the newest checkpoint
// is the one with the most history, not the highest step.
$row = DB::table('agent_step_checkpoints')
->where('run_key', $runKey)
->orderByDesc('message_count')
->orderByDesc('id')
->first();
return $row === null ? null : ['step' => (int) $row->step, 'messages' => unserialize(base64_decode($row->messages))];
}
}CurrentRun is a static holder for the run key and a stop flag. A worker runs one job at a time, and the SDK’s events carry an invocation id but not the job, so process-local state is the right place for it. RunInterrupted is a RuntimeException with a message. Register the listener in a service provider: Event::listen(StartingStep::class, [StepCheckpoint::class, 'starting']).
The messages are plain value objects: UserMessage, AssistantMessage with a collection of ToolCall, ToolResultMessage with a collection of ToolResult. serialize() round-trips them. The base64 is there because serialized protected properties carry NUL bytes, and a NUL in a text column is a bug you find at 2am.
Then the agent replays them. Implement Conversational and return the checkpoint from messages():
use App\Ai\Durable\IdempotentTool;
use App\Ai\Durable\StepCheckpoint;
use Laravel\Ai\Contracts\Tool;
class DurableResearchAgent extends QueuedResearchAgent
{
public function __construct(public readonly string $runKey)
{
parent::__construct();
}
public function tools(): iterable
{
foreach (parent::tools() as $tool) {
yield $tool instanceof Tool ? new IdempotentTool($tool, $this->runKey) : $tool;
}
}
public function messages(): iterable
{
return StepCheckpoint::latest($this->runKey)['messages'] ?? [];
}
public function isResuming(): bool
{
return StepCheckpoint::latest($this->runKey) !== null;
}
}One thing to get right about the prompt.
The loop builds its message list as your messages() followed by a new UserMessage with the prompt. The checkpoint already starts with the original prompt. Send it again on a retry and the model sees its own work followed by the task it was given, and starts over.
So the retry sends a continuation instead:
$prompt = $agent->isResuming()
? 'The previous worker stopped before you finished. Continue from where you were. Do not repeat tool calls whose results are already in this conversation.'
: "Research this blog topic and recommend angles we haven't covered: {$this->topic}";This is what the approval docs tell you to do after a failed resume: “continue the conversation with a normal text prompt”. Same mechanism, no approval required.
Own the job
->queue() gives you InvokeAgent. Write your own instead. Fifty lines, and every one of them is something the SDK’s job left out.
use App\Ai\Agents\DurableResearchAgent;
use App\Ai\Durable\CurrentRun;
use Illuminate\Contracts\Queue\Interruptible;
use Illuminate\Contracts\Queue\ShouldBeUnique;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
use Illuminate\Support\Facades\Storage;
class RunDurableResearch implements ShouldQueue, ShouldBeUnique, Interruptible
{
use Queueable;
public int $timeout = 240;
public int $tries = 3;
public array $backoff = [5, 30];
public int $uniqueFor = 600;
public function __construct(
public readonly string $runKey,
public readonly string $topic,
public readonly string $model = 'claude-haiku-4-5-20251001',
) {}
public function uniqueId(): string
{
return $this->runKey;
}
public function handle(): void
{
CurrentRun::begin($this->runKey);
$agent = new DurableResearchAgent($this->runKey);
$prompt = $agent->isResuming()
? 'The previous worker stopped before you finished. Continue from where you were. Do not repeat tool calls whose results are already in this conversation.'
: "Research this blog topic and recommend angles we haven't covered: {$this->topic}";
$response = $agent->prompt($prompt, provider: 'anthropic', model: $this->model, timeout: 120);
Storage::put("runs/{$this->runKey}-result.md", $response->text);
}
public function interrupted(int $signal): void
{
CurrentRun::requestStop();
}
}Four things in there.
The timeout is above the measured worst case. The longest recorded run was 98 seconds. The per-step provider timeout is 120. Keep retry_after on the connection above $timeout, or a second worker picks the job up while the first is still running it. That is oussama’s gotcha #4 and Deploynix’s double bill, and it applies unchanged.
Tries are explicit. Three, with a backoff. The retry that resumes from the checkpoint is a decision you made, not Horizon’s zero.
One run per key. ShouldBeUnique on the run key stops a double dispatch from starting two runs that fight over one ledger. Set uniqueFor. A unique lock with no expiry survives a SIGKILL forever, which is oussama’s gotcha #3.
Interruptible. Laravel 13.31 added this contract and the JobInterrupted event. When the worker receives SIGTERM, SIGINT or SIGQUIT while your job is running, it calls interrupted($signal) on the job. Only on jobs that implement the contract. InvokeAgent does not, so the hook in the release notes is unreachable from ->queue().
Here it sets a flag. The StepCheckpoint listener checks that flag at the next step boundary, after writing the checkpoint and before the next provider call, and throws. The exception leaves the loop cleanly. The worker releases the job with the backoff. The retry resumes from a checkpoint that includes the tool results of the step that was running when the signal arrived.
No half-billed provider call. No half-run tool. A deploy becomes a pause.
Kill it again
Same topic, same tools, same worker. This time the job is RunDurableResearch, and instead of a timeout the script sends kill -9 to the worker twenty seconds in. The supervisor’s version of a bad day.
13:02:25 attempt 1 durable.start resuming=false
13:02:29 attempt 1 step 1 FirecrawlSearch ×3 ledger rows 1-3
13:02:40 attempt 1 step 2 FirecrawlScrape ×3 ledger rows 4-6
13:02:44 attempt 1 step 3 starting, messages=5 checkpoint written
kill -9 (mid provider call)
13:03:55 attempt 2 durable.start resuming=true
13:03:55 attempt 2 step 1 starting, messages=6
13:03:59 attempt 2 step 1 FirecrawlCrawl, DatabaseQueryTool ledger rows 7-8
13:04:26 attempt 2 durable.done text_chars=6145Attempt 2 loaded five messages from the checkpoint, added the continuation prompt, and its first model call asked for the two tools attempt 1 never reached. Not the three searches. Not the three scrapes.
The ledger has eight rows for the whole run, each executed once. The crawl started once.
The same kill against the SDK’s job, earlier in the session: attempt 1 ran four tools, attempt 2 ran eight. Twelve executions for one answer, two of them with byte-identical arguments and the rest re-doing the same intent with different queries.
Now the deploy case. Same job, kill -TERM at twenty seconds, which is what queue:restart, horizon:terminate and every process supervisor send first.
13:09:53 attempt 1 durable.start resuming=false
13:09:56 attempt 1 step 1 FirecrawlSearch ×3
13:10:06 attempt 1 step 2 FirecrawlScrape ×3
13:10:13 attempt 1 step 3 starting, messages=5
13:10:17 attempt 1 worker.interrupted signal=15
13:10:17 attempt 1 durable.interrupted
13:10:17 attempt 1 step 3 completed: FirecrawlCrawl, DatabaseQueryTool
13:10:22 attempt 1 step 4 starting, messages=7 checkpoint written
13:10:22 attempt 1 RunInterrupted: stopping before step 4
13:10:22 attempt 1 job.released backoff=5
13:10:22 attempt 1 Worker STOPPED Interrupted
13:10:27 attempt 2 durable.start resuming=true
13:10:27 attempt 2 step 1 starting, messages=8
13:10:45 attempt 2 durable.done text_chars=5146, tool_calls=0The signal arrived during step 3’s provider call. The worker’s handler set the flag, the step finished, its two tools ran and landed in the ledger, the checkpoint for step 4 was written with all seven messages, and then the loop threw. Five seconds later a new worker loaded eight messages and wrote the answer without calling a single tool.
Eight tool executions, one answer, and a stop that landed between steps instead of wherever the supervisor’s patience ran out.
One thing the log makes plain: a signal is handled when PHP gets control back, which in practice is when the in-flight HTTP call returns. In an earlier run the SIGTERM arrived during a crawl request that took 30 seconds to time out, and the job saw it 23 seconds late. Your stop is as prompt as your slowest tool. That is an argument for the HTTP timeouts on your tools, not against the pattern.
Log why it died
Since 13.30, queue:work prints the reason a worker stopped as its last line. You saw it above: Worker STOPPED Job timed out. The same WorkerStopReason rides on the WorkerStopping event with the exit status, the jobs processed, and the memory in use.
One listener, one line per stop, and “the agent job disappeared” becomes “timed out on attempt 2 with 38 MB in use”. Put it next to the StepCompleted rows from the cost-math post and every death has a step number.
What this gives you
You can queue a multi-step agent and kill the worker at any point.
The retry finds the tool results on disk and does not run them again. It finds the history on disk and does not re-plan from the prompt. A deploy signal stops it between steps instead of mid-tool. And the log says which step it died on and why.
The SDK ships the events that make all of this possible and a job that uses none of them. Read the approval path, build it for the rest of your tools, and stop paying for step 1 twice.
The ledger, the checkpoint, the job and the kill script are in the companion repo, laravel-ai-research-agent. Queue it, kill -9 the worker, and watch it resume.