ollama_embed retries with the same fixed 30s request timeout forever: a slow but healthy embedding server never succeeds (use adaptive timeouts)

Problem and intended result

hivesense_app.ollama_embed (db/ollama.sql) gives every embedding request the same fixed timeout, urlopen(req, timeout=30) at about line 159, on the first attempt and on every retry. The backoff (5s doubling to 120s) only lengthens the sleep between attempts. So when the embedding server is alive but a batch takes longer than 30s, every attempt fails the same way:

  • Sync path (max_retries => NULL, infinite retries): the worker loops forever on that batch. Each attempt is abandoned at 30s, and Ollama is likely still computing the abandoned request when the next one arrives, which adds to the load that made it slow. That is a livelock, not a retry.
  • Search path (finite max_retries): fails after N identical attempts that could never have succeeded.

Observed 2026-10-15 in the AIDEV test stack (.aidev/test-stack.compose.yml, CPU Ollama with all-minilm:l6-v2 and mxbai-embed-xsmall-v1), on aidev/integration at 548021d. The host was steem-17 at load average 48-52 on 32 CPUs. embed-search and embed-slice both failed like this:

NOTICE:  [Batch 0:37] network error: timed out, retrying in 5s (1/inf)
NOTICE:  [Batch 0:37] network error: timed out, retrying in 10s (2/inf)
NOTICE:  [Batch 0:37] network error: timed out, retrying in 20s (3/inf)
NOTICE:  [Batch 0:37] network error: timed out, retrying in 40s (4/inf)
ERROR:  plpy.Error: canceling statement due to statement timeout

The same steps on the same commit at load ≈26 passed, with the whole embed step taking 6-7s. The server was healthy; it was CPU-starved. The 120s cap is .aidev/run-checks.sh's statement_timeout, which #62 (closed) added so a dead server fails fast.

Intended result: request timeouts adapt to how slow the server actually is. A slow but working server eventually answers, and a dead one still fails fast where the caller asks for that.

Required behavior

  1. Per-attempt timeout grows on timeouts. In ollama_embed, start each batch at the configured base timeout (default 30s, as today). After an attempt that timed out, double the next attempt's timeout, capped at a configured maximum (default 600s). Errors that aren't timeouts (connection refused, HTTP 5xx) keep the current timeout, because waiting longer can't fix them. Reset to the base for the next batch, as the backoff delay already is. Include the timeout used in the retry NOTICE (… timed out after 60s, retrying …).
  2. Base scales with batch size. Base timeout = max(base, per_text_seconds * len(batch)), so a 100-text batch isn't held to the same 30s as a single search query. Defaults: base 30s, per-text 1s.
  3. Configurable, with today's behavior as the floor. Store the three values (base, per-text, max) in hivesense_app.hivesense_app_status, settable through scripts/install_app.sh options, following how embedding_batch_size is handled. Existing installs get the defaults.
  4. Search path stays bounded. The search path (finite max_retries) must keep its worst-case total wait bounded, so a slow Ollama can't hold a snapshot open (the reason given in the existing comment). Cap its timeout growth separately (default max 60s), or derive the cap from max_retries so the total is no worse than today's order of magnitude. Document which one.
  5. Test stack. .aidev/run-checks.sh's 120s statement_timeout around generate_embeddings_for_posts must leave room for the adaptive chain on a loaded host. Raise it to cover at least two grown attempts (for example 600s). The dead-server case still fails fast through probe_embedding_server, which #62 (closed) added for exactly that. Update .aidev/README.md's embed-search description to match.

Acceptance criteria

  • If the embedding server takes longer than the base timeout but less than the maximum to answer a batch, the sync path embeds the batch on a later attempt instead of retrying forever. The NOTICEs show the growing timeout.
  • If the server refuses connections, the per-attempt timeout doesn't grow, and the existing retry and backoff behavior is unchanged.
  • The search path's total wait against an unresponsive server stays bounded at the documented cap.
  • A batch of N texts gets an initial timeout of at least per_text_seconds * N.
  • With the defaults, an install that doesn't set the new options behaves like today when the server answers within 30s.
  • In the AIDEV full slot, embed-search, embed-slice and embed-pca still fail within about 10s with the #62 (closed) message when the stack's Ollama is down.

Verification

  • A unit-level test of the timeout schedule (base, scaling, growth only on timeout, cap, reset per batch), in tests/unit if the logic is factored into an importable helper. Otherwise, a SQL test against a stub HTTP server that sleeps longer than the base timeout before answering. The stub must show that today's code never succeeds and the new code does.
  • Run the full slot by hand, with every step .aidev/project.yaml declares, and record the junit results. The workflow's own gate runs the slot approved on aidev/integration.
  • Live behavior under real CPU starvation can't be reproduced on demand. The stub server is the stand-in, and the first loaded-host gate run after promote is the live evidence.

Integration points

  • db/ollama.sql hivesense_app.ollama_embed — adaptive per-attempt timeout
  • db/database_schema.sql hivesense_app.hivesense_app_status — timeout configuration columns
  • scripts/install_app.sh — options for the timeout configuration
  • .aidev/run-checks.sh — statement_timeout around the embedding call
  • .aidev/README.md — embed-search description

Scope boundary

The backoff delay schedule, the batch size, and the openai-style endpoint's request shape stay as they are. Adding CPU limits or reservations to the test stack's Ollama is a separate question.