Skip to content

Add red 'stalled' status to admin dashboard banner #499

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dereuromark merged 1 commit into from
May 14, 2026
Merged

Add red 'stalled' status to admin dashboard banner #499

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions config/app.example.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,23 @@
// auto-refresh dashboard in seconds (0 = disabled)
'dashboardAutoRefresh' => 0,

// Status-banner thresholds on the admin dashboard, in seconds. The
// banner has three colors: green (running), yellow (idle), red
// (stalled — action required).
// running: fresh heartbeat (< dashboardIdleAfter)
// idle: stale heartbeat, no backlog (>= dashboardIdleAfter)
// stalled: >= dashboardStalledAfter with a pending backlog and no
// in-flight job, OR no worker reporting with backlog
// Defaults (60 / 120) are deliberate UI policy — human-perceptible
// 1-min / 2-min boundaries — not derived from queue mechanics, since
// no existing config knob (workerLifetime, defaultRequeueTimeout,
// sleeptime) actually means "heartbeat freshness." Override for
// unusual cadences (e.g. slow cron in `exitwhennothingtodo` mode —
// raise dashboardStalledAfter past the cron interval to avoid
// false-red between ticks).
'dashboardIdleAfter' => 60,
'dashboardStalledAfter' => 120,

// Standalone mode for admin controllers:
// - false (default): Extends App\Controller\AppController, inherits app auth/components
// - true: Isolated admin, skips app's AppController setup
Expand Down
184 changes: 147 additions & 37 deletions templates/Admin/Queue/index.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,58 +25,168 @@

use Cake\Core\Configure;

// Banner thresholds are a UI policy, not a system mechanic: how long the
// dashboard waits before nagging the admin. Defaults are 60s yellow / 120s
// red — human-perceptible minute boundaries — and don't derive from queue
// config knobs because none of them actually mean "heartbeat freshness":
// - workerLifetime is an exit policy (a 1h-lifetime worker still
// heartbeats every ~sleeptime when idle).
// - defaultRequeueTimeout is the job-reassignment safeguard, tuned for
// max job duration (often 5-10 min).
// - sleeptime is closest to the real heartbeat cadence for an idle
// worker, but busy workers don't sleep — and we already cover the
// busy-worker case via the `runningJobs > 0` escape hatch below.
// Override these for installations with unusual cron cadence (e.g. slow
// `exitwhennothingtodo` cron — raise dashboardStalledAfter past the cron
// interval to avoid false-red between ticks).
$idleAfterSeconds = (int)Configure::read('Queue.dashboardIdleAfter', 60);
$stalledAfterSeconds = (int)Configure::read('Queue.dashboardStalledAfter', 120);
?>

<!-- Status Banner -->
<?php if ($status): ?>
<?php
<?php
/**
* Three-state status (running / idle / stalled). State is computed for both the
* "have a recent heartbeat" path and the "no active worker rows at all" path so
* that a total cron outage — the worst case — surfaces as red, not as a muted
* info notice.
*
* running <idleAfterSeconds green
* idle idleAfterSeconds-stalledAfterSeconds,
* OR no heartbeat & no backlog yellow
* stalled ≥stalledAfterSeconds with pending backlog and no
* in-flight job, OR no heartbeat at all with pending red
*
* Thresholds default to 60s yellow / 120s red — human-perceptible minute
* boundaries — and can be tuned via Queue.dashboardIdleAfter and
* Queue.dashboardStalledAfter for installs with unusual cron cadence.
*
* Notes:
* - In cron-driven mode workers are short-lived; `workers == 0` is the normal
* idle state for a quiet system, so red requires a real pending backlog too.
* - `runningJobs > 0` (derived in the controller from
* `fetched IS NOT NULL AND completed IS NULL`) keeps a busy worker out of
* red: heartbeats fire at the top of each loop, not during long jobs, so
* a >2 min task with more pending behind it would look stalled by heartbeat
* age alone.
* - When `$status` is empty, `QueueProcessesTable::status()` filtered every
* worker row past `Queue.defaultRequeueTimeout`. In that case a pending
* backlog or stuck in-flight job is unambiguously a problem.
*/
$state = 'idle';
$time = null;
$relTime = null;

if ($status) {
/** @var \Cake\I18n\DateTime $time */
$time = $status['time'];
$running = $time->addMinutes(1)->isFuture();
$now = new \Cake\I18n\DateTime();
$secondsSinceActivity = max(0, $now->getTimestamp() - $time->getTimestamp());

$state = 'running';
if ($secondsSinceActivity >= $idleAfterSeconds) {
$state = 'idle';
}
if ($secondsSinceActivity >= $stalledAfterSeconds && $pendingJobs > 0 && $runningJobs === 0) {
$state = 'stalled';
}

$relTime = method_exists($this->Time, 'relLengthOfTime')
? $this->Time->relLengthOfTime($status['time'])
: $this->Time->timeAgoInWords($status['time']);
?>
<div class="status-banner <?= $running ? 'status-running' : 'status-idle' ?>">
<div class="d-flex align-items-center justify-content-between">
<div class="d-flex align-items-center">
<span class="status-icon me-3">
<?php if ($running): ?>
<i class="fas fa-check-circle text-success"></i>
<?php else: ?>
<i class="fas fa-pause-circle text-warning"></i>
<?php endif; ?>
</span>
<div>
<strong><?= $running ? __d('queue', 'Queue Running') : __d('queue', 'Queue Idle') ?></strong>
<div class="text-muted small">
} elseif ($pendingJobs > 0 || $runningJobs > 0) {
// No worker has reported within `Queue.defaultRequeueTimeout`, yet jobs are
// either waiting (pending) or marked in-flight (fetched but not completed).
// Pending + no heartbeat = cron likely dead. Running + no heartbeat = worker
// died mid-job and left a stale fetched row, OR a job legitimately ran past
// the requeue timeout (which is itself a misconfiguration worth surfacing).
$state = 'stalled';
}

$stateMeta = [
'running' => ['icon' => 'check-circle', 'iconColor' => 'text-success', 'label' => __d('queue', 'Queue Running')],
'idle' => ['icon' => 'pause-circle', 'iconColor' => 'text-warning', 'label' => __d('queue', 'Queue Idle')],
'stalled' => ['icon' => 'exclamation-circle', 'iconColor' => 'text-danger', 'label' => __d('queue', 'Queue Stalled')],
][$state];
?>
<div class="status-banner status-<?= $state ?>">
<div class="d-flex align-items-center justify-content-between">
<div class="d-flex align-items-center">
<span class="status-icon me-3">
<i class="fas fa-<?= $stateMeta['icon'] ?> <?= $stateMeta['iconColor'] ?>"></i>
</span>
<div>
<strong><?= $stateMeta['label'] ?></strong>
<?php if ($state === 'stalled'): ?>
<span class="badge bg-danger ms-2"><?= __d('queue', 'action required') ?></span>
<?php endif; ?>
<div class="text-muted small">
<?php if ($status): ?>
<?= __d('queue', 'Last activity {0}', $relTime) ?>
&bull;
<?= $this->Html->link(
__d('queue', '{0} worker(s)', $workers),
['action' => 'processes'],
['class' => 'text-decoration-none']
) ?>
<?php else: ?>
<?= __d('queue', 'No worker reporting') ?>
&bull;
<?= __d('queue', '{0} server(s)', count($servers)) ?>
</div>
<?php endif; ?>
<?= $this->Html->link(
__d('queue', '{0} worker(s)', $workers),
['action' => 'processes'],
['class' => 'text-decoration-none']
) ?>
&bull;
<?= __d('queue', '{0} server(s)', count($servers)) ?>
</div>
</div>
<div>
<?= $this->Html->link(
'<i class="fas fa-cogs me-1"></i>' . __d('queue', 'Manage Workers'),
['action' => 'processes'],
['class' => 'btn btn-sm btn-outline-dark', 'escapeTitle' => false]
) ?>
</div>
</div>
<div>
<?= $this->Html->link(
'<i class="fas fa-cogs me-1"></i>' . __d('queue', 'Manage Workers'),
['action' => 'processes'],
['class' => 'btn btn-sm btn-outline-dark', 'escapeTitle' => false]
) ?>
</div>
</div>
<?php else: ?>
<div class="alert alert-secondary">
<i class="fas fa-info-circle me-2"></i>
<?= __d('queue', 'No queue status available. Workers may not have started yet.') ?>
</div>
<?php endif; ?>
<?php if ($state === 'stalled'): ?>
<?php
if (!$status && $runningJobs > 0) {
$causeHint = __d('queue', 'A job is marked in-flight but no worker is reporting. The worker likely crashed mid-job — reset stale fetched jobs and check cron.');
} elseif (!$status) {
$causeHint = __d('queue', 'No worker has reported in. Cron is likely not firing — check that {0} runs on at least one server.', '<code>bin/cake queue run</code>');
} elseif ($workers === 0) {
$causeHint = __d('queue', 'Jobs are waiting but no workers are running. Check that {0} cron is firing on at least one server.', '<code>bin/cake queue run</code>');
} else {
$causeHint = __d('queue', "Jobs are waiting but aren't being picked up. Workers may have crashed — restart the queue or clean up stale processes.");
}
?>
<div class="stalled-details mt-3 pt-3 border-top border-danger-subtle">
<dl class="row mb-2 small">
<dt class="col-sm-3 text-muted fw-normal"><?= __d('queue', 'Last activity') ?></dt>
<dd class="col-sm-9 mb-1">
<?php if ($time): ?>
<code><?= h($time->i18nFormat('yyyy-MM-dd HH:mm:ss')) ?></code>
<span class="text-muted">· <?= $relTime ?></span>
<?php else: ?>
<span class="text-danger fw-medium"><?= __d('queue', 'No worker has reported recently') ?></span>
<?php endif; ?>
</dd>
<dt class="col-sm-3 text-muted fw-normal"><?= __d('queue', 'Workers') ?></dt>
<dd class="col-sm-9 mb-1">
<strong class="<?= $workers === 0 ? 'text-danger' : '' ?>"><?= $workers ?></strong>
<?= __d('queue', 'on {0} server(s)', count($servers)) ?>
</dd>
<dt class="col-sm-3 text-muted fw-normal"><?= __d('queue', 'Pending') ?></dt>
<dd class="col-sm-9 mb-0">
<strong class="<?= $pendingJobs > 0 ? 'text-danger' : '' ?>"><?= $pendingJobs ?></strong>
<?= __d('queue', 'jobs waiting') ?>
</dd>
</dl>
<div class="small">
<i class="fas fa-info-circle me-1 text-danger"></i>
<?= $causeHint ?>
</div>
</div>
<?php endif; ?>
</div>

<!-- Stats Cards -->
<div class="row g-3 mb-4">
Expand Down
16 changes: 16 additions & 0 deletions templates/layout/queue.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,22 @@
border: 1px solid #ffc107;
}

.status-banner.status-stalled {
background: linear-gradient(135deg, #f8d7da 0%, #f5c6cb 100%);
border: 1px solid #dc3545;
border-left-width: 4px;
}

.status-banner.status-stalled .stalled-details dt {
padding-top: 0.125rem;
}

.status-banner.status-stalled .stalled-details code {
background: rgba(220, 53, 69, 0.08);
padding: 0.125rem 0.375rem;
border-radius: 0.25rem;
}

.status-banner .status-icon {
font-size: 1.5rem;
}
Expand Down
Loading