lucas/omp-deck

Watch

Fork 0

Provider login gets lost sometimes #245

New issue

Open

opened 2026-11-10 10:39:35 +01:00 by lucas · 1 comment

lucas commented

2026-11-10 10:39:35 +01:00

Owner

For the second time now, the Claude login just dropped and I had to log in again.

Ideally this is fixed, but if not the Forgejo scheduler should be made capable to both notify the user but also delay triggers until the login is back to avoid the user having to chase all the aborted ones

For the second time now, the Claude login just dropped and I had to log in again. Ideally this is fixed, but if not the Forgejo scheduler should be made capable to both notify the user but also delay triggers until the login is back to avoid the user having to chase all the aborted ones

lucas added the

Kind

Bug

label

2026-11-10 10:39:40 +01:00

lucas added the

Agent

Triage

label

2026-11-10 19:21:07 +01:00

omp-deck commented

2026-11-10 19:23:05 +01:00

Collaborator

Triage

Two separable problems in one issue, and I'd keep them separable:

Root cause — the Anthropic OAuth credential disappears/stops working at some point. Needs diagnosis; nothing in this repo manages the credential lifecycle (the SDK's AuthStorage owns auth.db), so the fix may well be upstream or environmental.
Blast radius — a dropped login makes every Forgejo trigger that fires meanwhile burn, with no retry, so you have to re-trigger each one by hand. That part is entirely ours and is fixable regardless of (1).

#238 — "Account for usage quota in Forgejo scheduler": effectively the same mechanism, different precondition. Same shape: scheduler checks a readiness precondition → defers instead of starting the turn → posts a temporary comment that gets removed when the turn actually runs. Strong recommendation: design one "dispatcher readiness gate" with pluggable preconditions (quota below threshold, provider authenticated, …) and let #245 and #238 be two conditions on it, rather than two separate halt mechanisms. If you agree, the scheduler half of this issue could be folded into #238 and #245 kept purely for the root cause.
#246 is unrelated (surfaced only because the search matched "provider").

No duplicate of the login-loss bug itself exists.

What the code does today

Credentials are not stored or refreshed by the deck. auth-singleton.ts memoizes discoverAuthStorage() from the SDK; routes-auth-oauth.ts only drives AuthStorage.login() and a DELETE /providers/:provider that calls credentials.remove(). Everything else — token refresh, rotation, expiry — happens inside the SDK against ~/.omp/agent/auth.db, which is also what the omp CLI uses.
The deck has no health signal on credentials: deriveAuthState in routes-auth-oauth.ts collapses a credential to oauth | api-key | unconfigured with no expiry, no last-refresh, no probe. So a dead-but-present credential looks identical to a healthy one, and a credential that vanished is only noticed when a turn fails.
The only reaction to an auth failure is after the fact: looksLikeAuthError + maybeSuggestSubscriptionFallback in bridge/in-process.ts emit a notification if the same model id is reachable through an authenticated subscription provider. For the case "the subscription provider itself lost its login" that hint can't fire — so you get a bare inline error.
The dispatcher never retries: triggers interrupted or failed are surfaced (thread comment → inbox ticket → notification) and dropped, by design, because the agent may already have posted (forgejo/dispatcher.ts header). That design is right for crashes but wrong for "the turn never started because we had no credential" — in that case nothing was posted and deferring is safe.

Clarifying questions

When it dropped: did Settings → Providers show Anthropic as unconfigured (credential gone from auth.db), or still connected but every turn 401'd? That distinguishes "something deleted the row" from "refresh failed / refresh token revoked".
How is the deck deployed in your setup, and is ~/.omp/agent/ shared with anything else — an omp CLI you use in a terminal, a second deck process/container, a mounted volume? Concurrent refreshes with rotating refresh tokens are the classic way an OAuth credential dies: the process that refreshes second presents an already-rotated token and gets invalid_grant.
Was there a deck restart, container recreate, or image update around either occurrence?
Roughly how long had the login been in place before each drop (hours? days? weeks?) — a reproducible interval points at expiry/refresh, a random one at a race or an external revoke.
Do you still have server logs from either occurrence? oauth-routes, auth-singleton and the session error around the first failing turn would say a lot. If not, worth adding logging now so occurrence #3 is diagnosable.
For the scheduler half: should the gate cover routines (actionKind: "prompt" via routines-runner.ts) and manually started sessions too, or only Forgejo triggers? A routine firing into a dead login burns silently in exactly the same way.
Deferral policy: hold rows pending indefinitely until login returns, or expire after N hours? And should rows that already failed from the two past occurrences be re-drivable, or is re-triggering by hand acceptable for history?

Initial feedback

Part (2) is worth doing even if (1) gets fixed: quota (#238), transient provider outages and revoked credentials all produce the same "precondition not met" situation, and burning a trigger queue on it is never the right behaviour.
The deferral mechanism has precedent in the dispatcher: the gated set already holds rows back for the checkbox settle window, so "hold a claimed row, don't finish it, re-pump on a wake signal" is an existing pattern rather than new machinery. The natural wake signal for this condition is the models_changed broadcast that routes-auth-oauth.ts already emits after a successful login.
The precondition check should be cheap and local — registry.hasConfiguredAuth / isUsingOAuth for the session's configured model — not a live API probe per trigger. That catches "credential gone" immediately; it does not catch "credential present but rejected". For the second case the honest approach is reactive: classify the first turn failure with looksLikeAuthError, and if it matches, flip the gate closed and re-queue that trigger instead of posting a failure comment.
For the user-visible side, please consider reusing one notification per outage rather than one per deferred trigger, plus the temporary thread comment described in #238 — otherwise a login drop with ten queued triggers produces ten notifications, which is the same chase in a different inbox.
Diagnosis suggestion for (1), low cost and useful on its own: log every AuthStorage credential transition the deck can observe (login, remove, and the auth state seen at each session start) with provider and timestamp. Today there is no record at all of when the credential went away, which is why two occurrences have produced no evidence.

I have not reproduced anything here — this is a read of the current code plus the issue text. Happy to go deeper on either half once you've answered 1–3.

## Triage Two separable problems in one issue, and I'd keep them separable: 1. **Root cause** — the Anthropic OAuth credential disappears/stops working at some point. Needs diagnosis; nothing in this repo manages the credential lifecycle (the SDK's `AuthStorage` owns `auth.db`), so the fix may well be upstream or environmental. 2. **Blast radius** — a dropped login makes every Forgejo trigger that fires meanwhile burn, with no retry, so you have to re-trigger each one by hand. That part is entirely ours and is fixable regardless of (1). ### Related issues - **#238 — "Account for usage quota in Forgejo scheduler"**: effectively the same mechanism, different precondition. Same shape: *scheduler checks a readiness precondition → defers instead of starting the turn → posts a temporary comment that gets removed when the turn actually runs*. Strong recommendation: design one "dispatcher readiness gate" with pluggable preconditions (quota below threshold, provider authenticated, …) and let #245 and #238 be two conditions on it, rather than two separate halt mechanisms. If you agree, the scheduler half of this issue could be folded into #238 and #245 kept purely for the root cause. - #246 is unrelated (surfaced only because the search matched "provider"). No duplicate of the login-loss bug itself exists. ### What the code does today - Credentials are not stored or refreshed by the deck. `auth-singleton.ts` memoizes `discoverAuthStorage()` from the SDK; `routes-auth-oauth.ts` only drives `AuthStorage.login()` and a `DELETE /providers/:provider` that calls `credentials.remove()`. Everything else — token refresh, rotation, expiry — happens inside the SDK against `~/.omp/agent/auth.db`, which is also what the `omp` CLI uses. - The deck has **no health signal** on credentials: `deriveAuthState` in `routes-auth-oauth.ts` collapses a credential to `oauth | api-key | unconfigured` with no expiry, no last-refresh, no probe. So a dead-but-present credential looks identical to a healthy one, and a credential that vanished is only noticed when a turn fails. - The only reaction to an auth failure is after the fact: `looksLikeAuthError` + `maybeSuggestSubscriptionFallback` in `bridge/in-process.ts` emit a notification *if* the same model id is reachable through an authenticated subscription provider. For the case "the subscription provider itself lost its login" that hint can't fire — so you get a bare inline error. - The dispatcher never retries: triggers interrupted or failed are surfaced (thread comment → inbox ticket → notification) and dropped, by design, because the agent may already have posted (`forgejo/dispatcher.ts` header). That design is right for crashes but wrong for "the turn never started because we had no credential" — in that case nothing was posted and deferring is safe. ### Clarifying questions 1. When it dropped: did Settings → Providers show Anthropic as **unconfigured** (credential gone from `auth.db`), or still connected but every turn 401'd? That distinguishes "something deleted the row" from "refresh failed / refresh token revoked". 2. How is the deck deployed in your setup, and is `~/.omp/agent/` shared with anything else — an `omp` CLI you use in a terminal, a second deck process/container, a mounted volume? Concurrent refreshes with rotating refresh tokens are the classic way an OAuth credential dies: the process that refreshes second presents an already-rotated token and gets `invalid_grant`. 3. Was there a deck restart, container recreate, or image update around either occurrence? 4. Roughly how long had the login been in place before each drop (hours? days? weeks?) — a reproducible interval points at expiry/refresh, a random one at a race or an external revoke. 5. Do you still have server logs from either occurrence? `oauth-routes`, `auth-singleton` and the session error around the first failing turn would say a lot. If not, worth adding logging now so occurrence #3 is diagnosable. 6. For the scheduler half: should the gate cover **routines** (`actionKind: "prompt"` via `routines-runner.ts`) and manually started sessions too, or only Forgejo triggers? A routine firing into a dead login burns silently in exactly the same way. 7. Deferral policy: hold rows pending indefinitely until login returns, or expire after N hours? And should rows that already failed from the two past occurrences be re-drivable, or is re-triggering by hand acceptable for history? ### Initial feedback - Part (2) is worth doing **even if (1) gets fixed**: quota (#238), transient provider outages and revoked credentials all produce the same "precondition not met" situation, and burning a trigger queue on it is never the right behaviour. - The deferral mechanism has precedent in the dispatcher: the `gated` set already holds rows back for the checkbox settle window, so "hold a claimed row, don't finish it, re-pump on a wake signal" is an existing pattern rather than new machinery. The natural wake signal for this condition is the `models_changed` broadcast that `routes-auth-oauth.ts` already emits after a successful login. - The precondition check should be cheap and local — `registry.hasConfiguredAuth` / `isUsingOAuth` for the session's configured model — not a live API probe per trigger. That catches "credential gone" immediately; it does **not** catch "credential present but rejected". For the second case the honest approach is reactive: classify the first turn failure with `looksLikeAuthError`, and if it matches, flip the gate closed and re-queue that trigger instead of posting a failure comment. - For the user-visible side, please consider reusing one notification per outage rather than one per deferred trigger, plus the temporary thread comment described in #238 — otherwise a login drop with ten queued triggers produces ten notifications, which is the same chase in a different inbox. - Diagnosis suggestion for (1), low cost and useful on its own: log every `AuthStorage` credential transition the deck can observe (login, remove, and the auth state seen at each session start) with provider and timestamp. Today there is no record at all of *when* the credential went away, which is why two occurrences have produced no evidence. I have not reproduced anything here — this is a read of the current code plus the issue text. Happy to go deeper on either half once you've answered 1–3.

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

lucas/omp-deck#245

No description provided.

Rows
Columns