feat: backup hardening + verify, stats overview, queue health

Backups:
- restore --fresh wipes a dirty/partial schema (same mysql client + $cfg as the
  import, scoped to DATABASE(), so it can't wipe the wrong DB) after a verified
  pre-restore safety snapshot; unmissable recovery message on post-wipe failure
- workkit:db:verify (+ --verify on backup): decrypt + xz integrity + completion
  marker + sha256 sidecar check, no DB needed
- metadata sidecar per backup; prune --keep-min floor so prune can't leave zero
- FK_CHECKS=0 import prepend; deadlock-proof concurrent stdout/stderr drain

Ops:
- workkit:stats: per-model row counts (estimate/--exact), largest tables, DB size,
  queue + backup summary; MySQL/MariaDB info_schema, graceful elsewhere
- workkit:queue:health: per-queue depth/due/delayed/reserved + oldest-due age,
  failed-job counts, STALLED detection, DB-error breach, non-zero exit for cron

README + CHANGELOG; new config read with code-side defaults (nested-merge safe).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Fabian Wagner 2026-06-29 12:39:03 +02:00
parent 2589d4baf4
commit b85609f229
13 changed files with 2071 additions and 106 deletions

85
CHANGELOG.md Normal file
View File

@ -0,0 +1,85 @@
# Changelog
All notable changes to `laravel-workkit` are documented here. This project
follows [Keep a Changelog](https://keepachangelog.com/) and keeps every release
backward-compatible with the previous minor (see
`PRINCIPLES/laravel-composer-packages.md` § Backward compatibility).
## Unreleased
### Added
- **`workkit:stats`** (+ `StatsService`) — terminal "admin overview": per-Eloquent-model
row counts (fast `information_schema` estimate on MySQL, `--exact` for `COUNT(*)`),
largest tables by size, total database size, and a compact queue + backup summary.
Models are auto-discovered from `app/Models` or `config('workkit.stats.models')`.
`--json` for machine consumption; degrades gracefully on non-MySQL drivers.
- **`workkit:queue:health`** (+ `QueueHealthService`) — per-queue depth / due / delayed /
reserved counts and oldest-due-job age for the database queue driver, failed-jobs
totals (+ last *N* hours), and STALLED-queue detection (due work past
`max_age_minutes` with nothing reserved → worker likely down). Exits non-zero on any
breach for cron/monitoring; thresholds under `config('workkit.queue.*')` with per-queue
depth overrides. `QueueHealthService::evaluate()` is a pure, unit-testable function.
- New config blocks `workkit.stats.*` and `workkit.queue.*` (read with explicit
code-side defaults, so consumers who published an older config are unaffected).
### Added — backups
- **`workkit:db:restore --fresh`** — drops every table + view in the target
before importing, the only reliable fix for a restore failing with
`errno 150` / error `3780` against a dirty or partially-migrated schema. The
wipe is performed via the **same** `mysql` client and database the import
uses (built from the connection's `$cfg`, scoped to `DATABASE()`), so a
`url` DSN / `unix_socket` / read-write split can never make the wipe and the
import target different databases.
- **Pre-restore safety snapshot**`--fresh` first takes *and verifies* a
snapshot of the current database (named `*.pre-restore.sql.xz.enc`) and
aborts before dropping anything if it can't be written; opt out with
`--no-safety-backup`. On a post-wipe import failure the command prints the
exact `workkit:db:restore --file=<snapshot>` recovery command.
- **`workkit:db:verify`** (+ `workkit:db:backup --verify`) — proves a backup
decrypts with this host's `APP_KEY` and is a complete, non-truncated xz
stream, without touching the database; flags a missing mysqldump completion
marker and compares the on-disk sha256 against the metadata sidecar.
`--all` checks every backup.
- **Metadata sidecar**`workkit:db:backup` writes a best-effort
`<file>.meta.json` (size, sha256, created_at, cipher params) next to each
backup. Never fails a good backup if it can't be written.
- **`workkit:db:prune-backups --keep-min`** (config
`backup.retention_min_keep`, default 5) — always keeps the N newest backups
regardless of age, so an aggressive `--days` can never leave zero recovery
points. Prune now also removes a backup's sidecar alongside it and sweeps
orphaned sidecars.
- **Restore safeguards** — the source backup is verified before any
destructive step; a dirty (non-empty) target is reported with a `--fresh`
hint; an import that incompatibly fails prints actionable guidance; a
non-trivial backup that imports zero tables is flagged as a possible silent
failure. The confirm line now shows the backup size.
- New public `BackupService` helpers (all additive): `verify()`,
`dropAllTables()`, `countTables()`, `estimateDatabaseBytes()`,
`backupFiles()`, `restoreCandidates()`, `metaPathFor()`, `humanBytes()`,
`runResult()`.
### Changed
- **`BackupService::decryptDecompressImport()` now prepends
`SET FOREIGN_KEY_CHECKS=0;`** to the import stream so a hand-rolled dump
lacking mysqldump's own header still imports into an empty database
regardless of table order. Harmless for mysqldump output (which sets/restores
its own session vars), and the whole import is one short-lived session so the
relaxed check never leaks. This does **not** rescue a restore onto a dirty
schema with incompatible column types — use `--fresh` for that.
- File selection in `restore`/`prune` now ignores `*.meta.json` sidecars while
still finding custom `--out`-named backups (it denies the sidecar suffix
rather than allow-listing one extension). The `restore` auto-pick also skips
`*.pre-restore.*` snapshots so a recovery snapshot of a known-bad database is
never restored by accident.
### Notes
- Existing `BackupService` method signatures and all config keys are unchanged;
new behavior is additive or confined to the new `--fresh` path. New nested
config keys are read with explicit code-side defaults, so a consumer who
published an older `config/workkit.php` is unaffected.
- The README is now backup-documented and brought up to the standard Blax OSS
package skeleton (badges, feature list, Star History).

199
README.md
View File

@ -2,4 +2,203 @@
# Laravel Workkit # Laravel Workkit
[![PHP Version](https://img.shields.io/badge/php-%3E%3D8.0-blue)](https://php.net)
[![Laravel](https://img.shields.io/badge/laravel-10.x--13.x-orange)](https://laravel.com)
[![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)
A Laravel collection of helpers and utilities to reduce redundant code over multiple projects. A Laravel collection of helpers and utilities to reduce redundant code over multiple projects.
## Features
- 💾 **Streaming DB backups**`mysqldump | xz | openssl` in one pipe, APP_KEY-encrypted, zero DB bytes held in PHP memory (multi-GB safe)
- ♻️ **Safe restores**`--fresh` wipes a dirty/partial schema before importing, after taking + verifying a recovery snapshot; failed imports never leave you stranded
- 🔎 **Backup verification**`workkit:db:verify` proves a backup decrypts + is a complete xz stream *without* a database, so "is it corrupt?" gets a real answer
- 🧹 **Retention pruning** — age-based cleanup that always keeps the N newest, so prune can never delete your last recovery point
- 📊 **Stats overview**`workkit:stats` prints per-model row counts, the largest tables, total DB size + a queue/backup summary
- 🩺 **Queue health**`workkit:queue:health` reports per-queue depth/age/failed jobs and exits non-zero on breach (cron-friendly)
- 📄 **Variable pagination** — a `#[VariablePaginatable]` attribute + `request()->perPage()` macro for per-route, user-overridable page sizes
- 🧩 **Reusable middleware & traits** — bearer-token auth, force-JSON responses, `HasExpiration`, `HasMeta`, `HasMetaTranslation`, and small service helpers
## Quick Start
```bash
composer require blax-software/laravel-workkit
```
```bash
php artisan workkit:db:backup # storage/backups/db_mysql_<timestamp>.sql.xz.enc
php artisan workkit:db:verify # is the newest backup intact?
php artisan workkit:db:restore --fresh # wipe + restore the newest backup
```
Backups are encrypted with a key derived from your app's `APP_KEY`; a backup is
restorable only by a deployment that knows the same `APP_KEY`.
## Database Backups
A streaming, compressed, encrypted MySQL backup/restore suite. The dump,
compression and encryption happen in a single shell pipe, so PHP never holds
the database content in memory regardless of dump size.
### Commands
| Command | What it does |
|---|---|
| `workkit:db:backup` | Stream `mysqldump → xz → openssl` into `storage/backups`, write a `.meta.json` sidecar (size, sha256, params). |
| `workkit:db:restore` | Stream `openssl → xz → mysql`. Verifies the source first; `--fresh` wipes the target after a verified safety snapshot. |
| `workkit:db:verify` | Prove a backup decrypts and is a complete xz stream — no database touched. |
| `workkit:db:prune-backups` | Delete backups older than the retention window, always keeping the N newest. |
### Backup
```bash
php artisan workkit:db:backup # default connection
php artisan workkit:db:backup --connection=mysql --xz-level=6
php artisan workkit:db:backup --verify # confirm the backup right after writing
```
### Restore
```bash
# Newest restorable backup into the default connection (prompts unless --force)
php artisan workkit:db:restore
# A specific file
php artisan workkit:db:restore --file=db_mysql_2026-06-28_09-21-49.sql.xz.enc
# Wipe the target first — the ONLY thing that fixes a restore failing with
# errno 150 / error 3780 against a dirty or partially-migrated schema.
php artisan workkit:db:restore --fresh
```
`--fresh` is deliberately careful, because dropping every table is irreversible:
1. The **source backup is verified** before anything is touched — a file that
can't be decrypted never triggers a wipe.
2. A **pre-restore safety snapshot** of the current database is taken *and
verified* (skip with `--no-safety-backup`; the restore aborts if the
snapshot can't be written, rather than wiping with no recovery point).
3. Only then are all tables + views dropped — via the **same** `mysql` client
and database the import uses, so the wipe and the import provably hit the
same server (a `url` DSN / `unix_socket` / read-write split can't make them
diverge).
4. If the import fails *after* the wipe, the command prints the exact
`workkit:db:restore --file=<snapshot>` command to recover.
Interactively, `--fresh` asks you to type the database name to confirm; under
`--force` it proceeds non-interactively for deploy scripts.
> **Why `--fresh` and not just FK checks?** A leftover table with an
> incompatible column type makes MySQL raise `errno 150` / error `3780` at
> `CREATE TABLE` even with `FOREIGN_KEY_CHECKS=0` (which restores already set).
> The only fix is an empty target. If a restore fails this way, the command
> tells you to re-run with `--fresh`.
### Verify
```bash
php artisan workkit:db:verify # newest backup
php artisan workkit:db:verify --file=db_mysql_2026-06-28_09-21-49.sql.xz.enc
php artisan workkit:db:verify --all # every backup in the directory
```
Verify proves **integrity** (decrypts with this host's `APP_KEY` + a complete,
non-truncated xz stream), flags a missing mysqldump completion marker
(source-truncation), and compares the on-disk sha256 against the `.meta.json`
sidecar. It does **not** prove restorability — a valid backup of the wrong or
empty database still verifies "OK".
### Prune
```bash
php artisan workkit:db:prune-backups --dry-run
php artisan workkit:db:prune-backups --days=30 --keep-min=5
```
The `--keep-min` floor (default 5) is always kept regardless of age, so an
aggressive `--days` can never leave you with zero backups. Schedule it:
```php
Schedule::command('workkit:db:prune-backups')->daily();
```
### Configuration
```bash
php artisan vendor:publish --tag=workkit-config
```
| Key | Env | Default | Purpose |
|---|---|---|---|
| `backup.path` | `WORKKIT_BACKUP_PATH` | `storage/backups` | Where backups live |
| `backup.retention_days` | `WORKKIT_BACKUP_RETENTION_DAYS` | `30` | Age cutoff for prune |
| `backup.retention_min_keep` | `WORKKIT_BACKUP_RETENTION_MIN_KEEP` | `5` | Newest backups always kept |
| `backup.xz_level` | `WORKKIT_BACKUP_XZ_LEVEL` | `3` | xz compression level (09) |
### Requirements
The host needs `mysqldump`, `mysql`, `xz`, `openssl` and `bash` on `PATH`
standard on any reasonable Linux server. A non-empty `APP_KEY` is required
(backups are unrecoverable without the key that produced them).
### Restoring without the package
The output is plain `openssl enc -salt` format, so any host with the same
`APP_KEY` can restore it directly:
```bash
openssl enc -d -aes-256-cbc -pbkdf2 -iter 600000 -pass env:WK_KEY \
-in db_mysql_2026-06-28_09-21-49.sql.xz.enc | xz -d | mysql <database>
# WK_KEY = your APP_KEY with the "base64:" prefix stripped
```
## Ops & Observability
### Stats overview
```bash
php artisan workkit:stats # models + largest tables + DB size + queue/backup summary
php artisan workkit:stats --exact # true COUNT(*) per table (heavier) instead of the estimate
php artisan workkit:stats --models --json # just per-model counts, machine-readable
php artisan workkit:stats --tables --limit=30
```
Model row counts are **estimated** from `information_schema` on MySQL (instant);
pass `--exact` for a real `COUNT(*)`. Models are auto-discovered from
`app/Models` — override with `config('workkit.stats.models')` (an explicit FQCN
list) or `models_path`. Table/DB **sizes** are MySQL-only and degrade to "n/a"
elsewhere.
### Queue health
```bash
php artisan workkit:queue:health # human table; exits 0 / non-zero
php artisan workkit:queue:health --json || notify-ops # cron / monitoring probe
php artisan workkit:queue:health --max-age=10 --max-depth=500 --max-failed=50
```
For the `database` queue driver it reports per-queue **pending / due / delayed /
reserved** counts and the **oldest due job's age**, plus failed jobs (total +
last *N* hours). It flags a queue **STALLED** — due work older than
`max_age_minutes` with nothing reserved (the worker is probably down) — and
**exits non-zero on any breach** so it slots straight into cron or a health
probe. Thresholds live under `config('workkit.queue.*')`, with per-queue depth
overrides for intentionally-slow queues. Schedule it:
```php
Schedule::command('workkit:queue:health')->everyFiveMinutes();
```
## Changelog
See [CHANGELOG.md](CHANGELOG.md).
## Star History
<a href="https://www.star-history.com/?repos=blax-software%2Flaravel-workkit&type=date&legend=top-left">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=blax-software/laravel-workkit&type=date&theme=dark&legend=top-left" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=blax-software/laravel-workkit&type=date&legend=top-left" />
<img alt="Star History Chart" src="https://api.star-history.com/chart?repos=blax-software/laravel-workkit&type=date&legend=top-left" />
</picture>
</a>

View File

@ -6,18 +6,63 @@ return [
| Backup Settings | Backup Settings
|-------------------------------------------------------------------------- |--------------------------------------------------------------------------
| |
| Used by workkit:db:backup, workkit:db:restore and | Used by workkit:db:backup, workkit:db:restore, workkit:db:verify and
| workkit:db:prune-backups. The default `path` is storage_path('backups') | workkit:db:prune-backups. The default `path` is storage_path('backups')
| so backups live alongside the rest of the app's storage. `retention_days` | so backups live alongside the rest of the app's storage. `retention_days`
| is the threshold workkit:db:prune-backups uses by default anything | is the threshold workkit:db:prune-backups uses by default anything
| older than that gets deleted on the next prune run. | older than that gets deleted on the next prune run, EXCEPT the
| `retention_min_keep` newest, which are kept regardless of age so prune
| can never leave you with zero recovery points.
|
| NOTE: every command reads these nested keys with an explicit code-side
| default (e.g. config('workkit.backup.retention_min_keep', 5)) because
| mergeConfigFrom does not deep-merge nested arrays a consumer who
| published an older config without a new key still gets a sane value.
| |
*/ */
'backup' => [ 'backup' => [
'path' => env('WORKKIT_BACKUP_PATH'), // null → storage_path('backups') 'path' => env('WORKKIT_BACKUP_PATH'), // null → storage_path('backups')
'retention_days' => (int) env('WORKKIT_BACKUP_RETENTION_DAYS', 30), 'retention_days' => (int) env('WORKKIT_BACKUP_RETENTION_DAYS', 30),
// Always keep at least this many newest backups regardless of age.
// 0 disables the floor (prune may then leave zero backups).
'retention_min_keep' => (int) env('WORKKIT_BACKUP_RETENTION_MIN_KEEP', 5),
// xz compression level. Lower = faster + larger output. 3 is a // xz compression level. Lower = faster + larger output. 3 is a
// good default for SQL dumps (~10× ratio at ~3× the speed of -9). // good default for SQL dumps (~10× ratio at ~3× the speed of -9).
'xz_level' => (int) env('WORKKIT_BACKUP_XZ_LEVEL', 3), 'xz_level' => (int) env('WORKKIT_BACKUP_XZ_LEVEL', 3),
], ],
/*
|--------------------------------------------------------------------------
| Stats overview (workkit:stats)
|--------------------------------------------------------------------------
|
| `models` is an explicit list of Eloquent model FQCNs to count. When null
| the command auto-discovers models by scanning `models_path` (default:
| app_path('Models')). Set an explicit list to avoid scanning, or to count
| models that live outside app/Models.
|
*/
'stats' => [
'models' => null, // e.g. [\App\Models\User::class, \App\Models\Order::class]
'models_path' => null, // null → app_path('Models')
],
/*
|--------------------------------------------------------------------------
| Queue health thresholds (workkit:queue:health)
|--------------------------------------------------------------------------
|
| The command exits non-zero when any of these is breached. `max_depth`
| applies per queue; `max_depth_per_queue` overrides it for named queues
| (e.g. an intentionally-slow, rate-limited queue allowed a deeper backlog).
| A queue is flagged STALLED when its oldest *due* job is older than
| `max_age_minutes` while nothing is reserved (worker likely down).
|
*/
'queue' => [
'max_depth' => (int) env('WORKKIT_QUEUE_MAX_DEPTH', 250),
'max_depth_per_queue' => [], // e.g. ['geoip' => 5000]
'max_age_minutes' => (int) env('WORKKIT_QUEUE_MAX_AGE_MINUTES', 15),
'max_failed' => (int) env('WORKKIT_QUEUE_MAX_FAILED', 25),
],
]; ];

View File

@ -14,6 +14,10 @@ use RuntimeException;
* is one shell pipe (mysqldump | xz | openssl); PHP holds zero bytes * is one shell pipe (mysqldump | xz | openssl); PHP holds zero bytes
* of database content in memory regardless of dump size. * of database content in memory regardless of dump size.
* *
* A best-effort <file>.meta.json sidecar (size, sha256, created_at,
* cipher params) is written alongside each backup so its integrity can
* later be confirmed and a suspiciously-small dump spotted at a glance.
*
* Output filename: * Output filename:
* storage/backups/db_<connection>_<timestamp>.sql.xz.enc * storage/backups/db_<connection>_<timestamp>.sql.xz.enc
*/ */
@ -22,7 +26,8 @@ class BackupCommand extends Command
protected $signature = 'workkit:db:backup protected $signature = 'workkit:db:backup
{--connection= : DB connection to back up (defaults to config(database.default))} {--connection= : DB connection to back up (defaults to config(database.default))}
{--out= : Custom output path (overrides storage/backups default)} {--out= : Custom output path (overrides storage/backups default)}
{--xz-level= : xz compression level 09 (default: 3 fast, ~10× ratio for SQL)}'; {--xz-level= : xz compression level 09 (default: 3 fast, ~10× ratio for SQL)}
{--verify : After writing, verify the backup decrypts + is a valid xz stream}';
protected $description = 'Create a streamed, compressed + APP_KEY-encrypted backup of the configured MySQL database.'; protected $description = 'Create a streamed, compressed + APP_KEY-encrypted backup of the configured MySQL database.';
@ -33,17 +38,19 @@ class BackupCommand extends Command
if (! $cfg) { if (! $cfg) {
$this->error("Unknown database connection: {$connection}"); $this->error("Unknown database connection: {$connection}");
return self::FAILURE; return self::FAILURE;
} }
if (($cfg['driver'] ?? null) !== 'mysql') { if (($cfg['driver'] ?? null) !== 'mysql') {
$this->error("workkit:db:backup currently supports only MySQL connections (got: {$cfg['driver']})."); $this->error("workkit:db:backup currently supports only MySQL connections (got: {$cfg['driver']}).");
return self::FAILURE; return self::FAILURE;
} }
$stamp = date('Y-m-d_H-i-s'); $stamp = date('Y-m-d_H-i-s');
$base = BackupService::backupDirectory(); $base = BackupService::backupDirectory();
$outPath = $this->option('out') $outPath = $this->option('out')
?: "{$base}/db_{$connection}_{$stamp}.sql.xz.enc"; ?: "{$base}/db_{$connection}_{$stamp}".BackupService::BACKUP_EXT;
$xzLevel = (int) ($this->option('xz-level') ?? config('workkit.backup.xz_level', 3)); $xzLevel = (int) ($this->option('xz-level') ?? config('workkit.backup.xz_level', 3));
@ -58,28 +65,33 @@ class BackupCommand extends Command
BackupService::dumpCompressEncrypt($cfg, $outPath, $xzLevel); BackupService::dumpCompressEncrypt($cfg, $outPath, $xzLevel);
} catch (RuntimeException $e) { } catch (RuntimeException $e) {
$this->error($e->getMessage()); $this->error($e->getMessage());
return self::FAILURE; return self::FAILURE;
} }
$elapsed = microtime(true) - $startedAt; $elapsed = microtime(true) - $startedAt;
$size = filesize($outPath); $size = (int) filesize($outPath);
$this->info(sprintf( $this->info(sprintf(
'Backup complete in %.1fs: %s (%s)', 'Backup complete in %.1fs: %s (%s)',
$elapsed, $elapsed,
$outPath, $outPath,
self::humanBytes((int) $size), BackupService::humanBytes($size),
)); ));
return self::SUCCESS;
if ($this->option('verify')) {
$r = BackupService::verify($outPath);
if (! $r['integrity_ok']) {
$this->error('Verification FAILED: '.$r['message']);
return self::FAILURE;
}
if (! $r['looks_complete']) {
$this->warn('Verification: '.$r['message']);
} else {
$this->info('Verified: decrypts + valid xz + completion marker present.');
}
} }
private static function humanBytes(int $bytes): string return self::SUCCESS;
{
$units = ['B', 'KB', 'MB', 'GB', 'TB'];
$i = 0;
while ($bytes >= 1024 && $i < count($units) - 1) {
$bytes /= 1024;
$i++;
}
return sprintf('%.2f %s', $bytes, $units[$i]);
} }
} }

View File

@ -8,9 +8,15 @@ use Blax\Workkit\Services\BackupService;
use Illuminate\Console\Command; use Illuminate\Console\Command;
/** /**
* Drop backup files older than the retention window. Defaults to 30 * Drop backup files older than the retention window, while ALWAYS keeping
* days; the host can override via --days or by setting * the N newest regardless of age (retention_min_keep, default 5) so an
* `workkit.backup.retention_days` in the published config. * aggressive --days can never leave you with zero recovery points the
* floor that turns "I deleted my only good backup" into a non-event.
*
* Each backup's .meta.json sidecar is removed alongside it, and orphaned
* sidecars (whose backup is already gone) are swept up too. Pre-restore
* safety snapshots are pruned by the same rules (they're emergency
* artifacts, not long-term backups) but still count toward the keep floor.
* *
* Designed to be wired into the scheduler so storage doesn't fill up * Designed to be wired into the scheduler so storage doesn't fill up
* with stale dumps, e.g. Schedule::command('workkit:db:prune-backups')->daily(); * with stale dumps, e.g. Schedule::command('workkit:db:prune-backups')->daily();
@ -19,34 +25,63 @@ class PruneBackupsCommand extends Command
{ {
protected $signature = 'workkit:db:prune-backups protected $signature = 'workkit:db:prune-backups
{--days= : Retention window in days (default: workkit.backup.retention_days, falling back to 30)} {--days= : Retention window in days (default: workkit.backup.retention_days, falling back to 30)}
{--keep-min= : Always keep at least this many newest backups regardless of age (default: workkit.backup.retention_min_keep, falling back to 5; 0 disables the floor)}
{--dry-run : Show what would be deleted without removing anything}'; {--dry-run : Show what would be deleted without removing anything}';
protected $description = 'Delete backup files older than the retention window.'; protected $description = 'Delete backup files older than the retention window, always keeping the N newest.';
public function handle(): int public function handle(): int
{ {
$days = (int) ($this->option('days') ?: config('workkit.backup.retention_days', 30)); // Distinguish "not passed" (null) from an explicit --days=0, which must
// error via the < 1 guard rather than being silently swallowed by ?:.
$daysOpt = $this->option('days');
$days = (int) ($daysOpt !== null ? $daysOpt : config('workkit.backup.retention_days', 30));
if ($days < 1) { if ($days < 1) {
$this->error('--days must be >= 1'); $this->error('--days must be >= 1');
return self::FAILURE; return self::FAILURE;
} }
$base = BackupService::backupDirectory(); // Read with an explicit code-side default: a consumer who published an
$files = glob($base . '/*') ?: []; // OLD config (a `backup` array without this key) would otherwise get
// null here, because mergeConfigFrom does not deep-merge nested keys.
$keepMinOpt = $this->option('keep-min');
$keepMin = (int) ($keepMinOpt !== null ? $keepMinOpt : config('workkit.backup.retention_min_keep', 5));
if ($keepMin < 0) {
$keepMin = 0;
}
if ($keepMin === 0) {
$this->warn('keep-min is 0 — prune may delete every aged backup, leaving zero recovery points.');
}
$dry = (bool) $this->option('dry-run');
// Newest-first, sidecars excluded. Same ordering the restore resolver
// uses, so the always-kept set and the restore candidate agree.
$files = BackupService::backupFiles();
$cutoff = time() - $days * 86400; $cutoff = time() - $days * 86400;
$protected = array_slice($files, 0, $keepMin);
$protectedSet = array_flip($protected);
$removed = 0; $removed = 0;
$kept = 0; $kept = 0;
foreach ($files as $f) { foreach ($files as $f) {
if (! is_file($f)) { if (isset($protectedSet[$f])) {
$kept++;
continue; continue;
} }
if (filemtime($f) < $cutoff) { if (filemtime($f) < $cutoff) {
if ($this->option('dry-run')) { $meta = BackupService::metaPathFor($f);
if ($dry) {
$this->line("would remove: {$f}"); $this->line("would remove: {$f}");
} else { } else {
@unlink($f); @unlink($f);
if (is_file($meta)) {
@unlink($meta);
}
$this->line("removed: {$f}"); $this->line("removed: {$f}");
} }
$removed++; $removed++;
@ -55,12 +90,32 @@ class PruneBackupsCommand extends Command
} }
} }
// Sweep orphaned sidecars (backup already gone, e.g. removed by hand).
$base = BackupService::backupDirectory();
$orphans = 0;
foreach (glob($base.'/*'.BackupService::META_EXT) ?: [] as $meta) {
$backup = substr($meta, 0, -strlen(BackupService::META_EXT));
if (! is_file($backup)) {
if ($dry) {
$this->line("would remove orphan sidecar: {$meta}");
} else {
@unlink($meta);
$this->line("removed orphan sidecar: {$meta}");
}
$orphans++;
}
}
$verb = $dry ? 'would-remove' : 'removed';
$this->info(sprintf( $this->info(sprintf(
'Backups: %d kept, %d %s (cutoff: %d days).', 'Backups: %d kept, %d %s, %d orphan sidecar(s) %s (cutoff: %d days, keep-min: %d).',
$kept, $kept,
$removed, $removed,
$this->option('dry-run') ? 'would-remove' : 'removed', $verb,
$orphans,
$verb,
$days, $days,
$keepMin,
)); ));
return self::SUCCESS; return self::SUCCESS;

View File

@ -7,6 +7,7 @@ namespace Blax\Workkit\Commands\Database;
use Blax\Workkit\Services\BackupService; use Blax\Workkit\Services\BackupService;
use Illuminate\Console\Command; use Illuminate\Console\Command;
use RuntimeException; use RuntimeException;
use Throwable;
/** /**
* Restore a backup produced by workkit:db:backup. Streams the file * Restore a backup produced by workkit:db:backup. Streams the file
@ -14,18 +15,29 @@ use RuntimeException;
* matching the backup pipeline exactly. PHP allocates nothing for the * matching the backup pipeline exactly. PHP allocates nothing for the
* payload, so even multi-GB backups restore without bumping memory_limit. * payload, so even multi-GB backups restore without bumping memory_limit.
* *
* Without --file, picks the newest backup in storage/backups by mtime. * Without --file, picks the newest *restorable* backup in storage/backups
* Refuses to run unless --force is passed: a restore overwrites whatever * by mtime (pre-restore safety snapshots are excluded from auto-pick).
* is currently in the target database. * Refuses to run unless --force is passed or the operator confirms: a
* restore overwrites whatever is currently in the target database.
*
* --fresh drops every table + view in the target first, which is the only
* thing that fixes a restore failing with errno 150 / error 3780 against a
* dirty or partially-migrated schema (incompatible leftover column types).
* Before it drops anything it (a) verifies the source backup decrypts and
* is a valid xz stream and (b) takes + verifies a pre-restore safety
* snapshot, so a failed import never leaves you with an empty database and
* no recovery point the exact trap that loses data.
*/ */
class RestoreCommand extends Command class RestoreCommand extends Command
{ {
protected $signature = 'workkit:db:restore protected $signature = 'workkit:db:restore
{--connection= : DB connection to restore into (defaults to config(database.default))} {--connection= : DB connection to restore into (defaults to config(database.default))}
{--file= : Specific backup filename inside the backups directory (default: newest by mtime)} {--file= : Specific backup filename inside the backups directory (default: newest restorable file by mtime)}
{--force : Skip the confirmation prompt}'; {--fresh : Drop ALL tables and views in the target before importing. Fixes restore-onto-dirty-schema failures (errno 150 / error 3780). Takes a verified pre-restore safety snapshot first.}
{--no-safety-backup : With --fresh, skip the automatic pre-restore safety snapshot. DANGEROUS: no recovery point if the import fails after the wipe.}
{--force : Skip confirmation prompts (non-interactive).}';
protected $description = 'Restore a streaming, APP_KEY-encrypted database backup. Defaults to the newest file in storage/backups.'; protected $description = 'Restore a streaming, APP_KEY-encrypted database backup. Defaults to the newest restorable file in storage/backups.';
public function handle(): int public function handle(): int
{ {
@ -34,54 +46,167 @@ class RestoreCommand extends Command
if (! $cfg) { if (! $cfg) {
$this->error("Unknown database connection: {$connection}"); $this->error("Unknown database connection: {$connection}");
return self::FAILURE; return self::FAILURE;
} }
if (($cfg['driver'] ?? null) !== 'mysql') { if (($cfg['driver'] ?? null) !== 'mysql') {
$this->error("workkit:db:restore currently supports only MySQL connections (got: {$cfg['driver']})."); $this->error("workkit:db:restore currently supports only MySQL connections (got: {$cfg['driver']}).");
return self::FAILURE; return self::FAILURE;
} }
$file = $this->resolveFile(); $file = $this->resolveFile();
if (! $file) { if (! $file) {
$this->error('No backup found.'); $this->error('No backup found.');
return self::FAILURE; return self::FAILURE;
} }
if (! file_exists($file)) { if (! file_exists($file)) {
$this->error("Backup file not found: {$file}"); $this->error("Backup file not found: {$file}");
return self::FAILURE; return self::FAILURE;
} }
$fresh = (bool) $this->option('fresh');
// 1) Verify the SOURCE before doing anything destructive. Never drop
// tables on the basis of a file that can't even be decrypted.
$v = BackupService::verify($file);
if (! $v['integrity_ok']) {
$this->error('Backup failed verification — refusing to restore.');
$this->line($v['message']);
return self::FAILURE;
}
if (! $v['looks_complete']) {
$this->warn('Note: '.$v['message']);
// A source with no completion marker may be truncated-at-source.
// Importing it is recoverable when a safety snapshot is taken, but
// --fresh --no-safety-backup would wipe the target and repopulate
// it from a partial dump with no way back — refuse that outright.
if ($fresh && $this->option('no-safety-backup')) {
$this->error('Refusing: --fresh --no-safety-backup on a source that looks incomplete (no mysqldump completion marker).');
$this->line('This would wipe the database and import a possibly-truncated dump with no recovery point.');
$this->line('Run workkit:db:verify on the backup, or drop --no-safety-backup so a recovery snapshot is taken first.');
return self::FAILURE;
}
}
if ($v['checksum_matches'] === false) {
$this->warn($v['message']);
}
$this->warn(sprintf( $this->warn(sprintf(
'About to restore `%s`@%s from: %s', 'About to restore `%s`@%s from: %s (%s)',
$cfg['database'], $cfg['database'],
$cfg['host'], $cfg['host'] ?? 'localhost',
$file, $file,
BackupService::humanBytes((int) $v['bytes']),
)); ));
$this->warn('This will OVERWRITE any data that conflicts with the dump.'); $this->warn('This will OVERWRITE any data that conflicts with the dump.');
if (! $this->option('force') && ! $this->confirm('Proceed?', false)) { // 2) Dirty-target detection — purely informational; never blocks or
$this->info('Aborted.'); // prompts under --force (existing automation restores over data).
return self::SUCCESS; $tableCount = $this->safeCountTables($cfg);
if ($tableCount === null) {
$this->line('Target table count: unknown (could not query the database).');
} elseif ($tableCount > 0 && ! $fresh) {
$this->warn(sprintf(
'Target is not empty (%d tables/views). Restoring onto an existing schema can fail with '
.'errno 150 / error 3780 when column types differ — re-run with --fresh if that happens.',
$tableCount,
));
}
if ($fresh) {
$this->warn(sprintf(
'--fresh will DROP ALL %stables/views in `%s`@%s before importing.',
$tableCount === null ? '' : $tableCount.' ',
$cfg['database'],
$cfg['host'] ?? 'localhost',
));
} }
// 3) Confirmation. --fresh is a full wipe, so require typed name
// confirmation interactively; --force is the non-interactive gate.
if (! $this->option('force')) {
if ($fresh) {
$typed = (string) $this->ask(sprintf(
"Type the database name '%s' to confirm dropping ALL its tables",
$cfg['database'],
));
if ($typed !== $cfg['database']) {
$this->info('Aborted (name did not match).');
return self::SUCCESS;
}
if (! $v['looks_complete'] && ! $this->confirm('The source dump looks incomplete (no completion marker). Wipe and import it anyway?', false)) {
$this->info('Aborted.');
return self::SUCCESS;
}
} elseif (! $this->confirm('Proceed?', false)) {
$this->info('Aborted.');
return self::SUCCESS;
}
}
// 4) --fresh: snapshot → verify snapshot → drop. Capture the snapshot
// path BEFORE dropping so every post-drop error can point at it.
$snapshotPath = null;
if ($fresh) {
if (! $this->option('no-safety-backup')) {
try {
$snapshotPath = $this->createSafetySnapshot($connection, $cfg);
} catch (Throwable $e) {
$this->error('Pre-restore safety snapshot failed — aborting before any data is dropped.');
$this->line($e->getMessage());
$this->line('Fix the cause (often disk space or a missing mysqldump), or re-run with '
.'--no-safety-backup to proceed WITHOUT a recovery point (dangerous).');
return self::FAILURE;
}
} else {
$this->warn('Skipping pre-restore safety snapshot (--no-safety-backup). No recovery point if the import fails.');
}
try {
BackupService::dropAllTables($cfg);
$this->line('Dropped all tables/views in the target database.');
} catch (Throwable $e) {
$this->error('Failed to drop existing tables — aborting (no import attempted; your data is intact).');
$this->line($e->getMessage());
return self::FAILURE;
}
}
// 5) Import.
$startedAt = microtime(true); $startedAt = microtime(true);
try { try {
BackupService::decryptDecompressImport($file, $cfg); BackupService::decryptDecompressImport($file, $cfg);
} catch (RuntimeException $e) { } catch (RuntimeException $e) {
$msg = $e->getMessage(); $this->reportImportFailure($e, $connection, $cfg, $file, $fresh, $snapshotPath);
// openssl's password mismatch error has a recognisable shape;
// translate it into something an operator can act on.
if (str_contains($msg, 'bad decrypt') || str_contains($msg, 'bad magic')) {
$this->error('Decryption failed — likely an APP_KEY mismatch between this host and the host that produced the backup.');
$this->line($msg);
} else {
$this->error($msg);
}
return self::FAILURE; return self::FAILURE;
} }
// 6) Post-import sanity: a non-trivial backup that imported nothing is
// a silent-failure signal (swallowed pipe error / empty stream).
if ((int) $v['bytes'] > 1024) {
$after = $this->safeCountTables($cfg);
if ($after === 0) {
$this->warn('Restore reported success but the database has 0 tables — the import may have applied '
.'nothing. Verify the backup (workkit:db:verify) and the APP_KEY.');
}
}
$elapsed = microtime(true) - $startedAt; $elapsed = microtime(true) - $startedAt;
$this->info(sprintf('Restore complete in %.1fs.', $elapsed)); $this->info(sprintf('Restore complete in %.1fs.', $elapsed));
if ($snapshotPath) {
$this->line('Pre-restore snapshot kept at: '.$snapshotPath);
}
return self::SUCCESS; return self::SUCCESS;
} }
@ -89,7 +214,8 @@ class RestoreCommand extends Command
* Resolve the file argument: * Resolve the file argument:
* - --file=path/to/X.sql.xz.enc use as-is if it exists * - --file=path/to/X.sql.xz.enc use as-is if it exists
* - --file=basename look up inside storage/backups * - --file=basename look up inside storage/backups
* - omitted pick newest in storage/backups * - omitted newest *restorable* file (pre-restore
* safety snapshots excluded)
*/ */
private function resolveFile(): ?string private function resolveFile(): ?string
{ {
@ -101,14 +227,152 @@ class RestoreCommand extends Command
return $opt; return $opt;
} }
$candidate = $base.'/'.ltrim((string) $opt, '/'); $candidate = $base.'/'.ltrim((string) $opt, '/');
return is_file($candidate) ? $candidate : null; return is_file($candidate) ? $candidate : null;
} }
$files = glob($base . '/*'); $candidates = BackupService::restoreCandidates();
if (! $files) { if (! $candidates) {
// Distinguish "directory empty" from "only safety snapshots present"
// so the operator isn't told "No backup found" while a recovery
// snapshot sits right there (it just has to be named explicitly).
if (BackupService::backupFiles()) {
$this->warn('Only pre-restore safety snapshots are present — pass one explicitly with --file=... to restore it.');
}
return null;
}
return $candidates[0];
}
/**
* Dump verify the current database into a distinctly-named pre-restore
* snapshot. Returns its absolute path. Throws if the dump or its
* verification fails, so the caller can abort before dropping anything.
*/
private function createSafetySnapshot(string $connection, array $cfg): string
{
$base = BackupService::backupDirectory();
$stamp = date('Y-m-d_H-i-s');
$path = sprintf(
'%s/db_%s_%s%s%s',
$base,
$connection,
$stamp,
BackupService::PRE_RESTORE_MARK,
BackupService::BACKUP_EXT,
);
if (file_exists($path)) {
throw new RuntimeException("Safety snapshot path already exists, refusing to overwrite: {$path}");
}
// Best-effort disk sanity check — warn, don't block (the estimate is
// rough: ~6× compression is conservative for SQL).
$free = @disk_free_space($base);
$dbBytes = $this->safeEstimateDbBytes($cfg);
if ($free !== false && $dbBytes !== null && $dbBytes > 0) {
$estimate = (int) ($dbBytes / 6);
if ($free < $estimate) {
$this->warn(sprintf(
'Low disk on backup volume: %s free, snapshot may need ~%s. Proceeding anyway.',
BackupService::humanBytes((int) $free),
BackupService::humanBytes($estimate),
));
}
}
$this->line('Creating pre-restore safety snapshot…');
BackupService::dumpCompressEncrypt($cfg, $path, (int) config('workkit.backup.xz_level', 3));
$sv = BackupService::verify($path);
if (! $sv['integrity_ok']) {
// Don't leave an unverifiable snapshot (and its sidecar) behind —
// it would clutter the dir and collide with a same-second retry.
@unlink($path);
@unlink(BackupService::metaPathFor($path));
throw new RuntimeException('Safety snapshot did not verify: '.$sv['message']);
}
$this->info(sprintf(
'Safety snapshot saved + verified: %s (%s)',
$path,
BackupService::humanBytes((int) $sv['bytes']),
));
return $path;
}
/**
* Translate an import failure into operator-actionable guidance, and
* when --fresh already dropped the schema make the recovery path
* impossible to miss.
*/
private function reportImportFailure(
RuntimeException $e,
string $connection,
array $cfg,
string $file,
bool $fresh,
?string $snapshotPath
): void {
$msg = $e->getMessage();
if (str_contains($msg, 'bad decrypt') || str_contains($msg, 'bad magic')) {
$this->error('Decryption failed — likely an APP_KEY mismatch between this host and the host that produced the backup.');
$this->line($msg);
} elseif (preg_match('/\b3780\b|\b1215\b|errno:?\s*150|incompatible/i', $msg)) {
// Match the error code, not the broad phrase "foreign key", to
// avoid false-positives on unrelated FK row failures.
$this->error('Import failed on a foreign-key / column-type incompatibility (errno 150 / error 3780).');
if (! $fresh) {
$this->line('The target has a conflicting/leftover schema. Re-run with --fresh to drop existing tables first:');
$this->line(sprintf(
' php artisan workkit:db:restore --connection=%s --file=%s --fresh',
$connection,
basename($file),
));
$this->line('A non-fresh import may have PARTIALLY applied before failing, so the database is now in a mixed state.');
} else {
$this->line($msg);
}
} else {
$this->error($msg);
}
if ($fresh) {
$this->newLine();
if ($snapshotPath) {
$this->error('THE TARGET DATABASE IS NOW EMPTY — --fresh dropped it before the failed import.');
$this->warn('Recover the pre-restore state with:');
$this->line(sprintf(
' php artisan workkit:db:restore --connection=%s --file=%s',
$connection,
$snapshotPath,
));
} else {
$this->error('THE TARGET DATABASE IS NOW EMPTY and no safety snapshot was taken (--no-safety-backup).');
$this->warn('Restore from your most recent good backup: workkit:db:restore --connection='.$connection.' --file=…');
}
}
}
private function safeCountTables(array $cfg): ?int
{
try {
return BackupService::countTables($cfg);
} catch (Throwable $e) {
return null;
}
}
private function safeEstimateDbBytes(array $cfg): ?int
{
try {
return BackupService::estimateDatabaseBytes($cfg);
} catch (Throwable $e) {
return null; return null;
} }
usort($files, fn($a, $b) => filemtime($b) <=> filemtime($a));
return $files[0];
} }
} }

View File

@ -0,0 +1,173 @@
<?php
declare(strict_types=1);
namespace Blax\Workkit\Commands\Database;
use Blax\Workkit\Services\BackupService;
use Blax\Workkit\Services\QueueHealthService;
use Blax\Workkit\Services\StatsService;
use Illuminate\Console\Command;
use Throwable;
/**
* An at-a-glance "admin overview" in the terminal: row count per Eloquent
* model, the largest tables by size, the total database footprint, and a
* compact queue + backup line. Read-only.
*
* Counts are estimated (information_schema, instant) by default on MySQL; pass
* --exact for a true COUNT(*) per table (heavier on big tables).
*/
class StatsCommand extends Command
{
protected $signature = 'workkit:stats
{--connection= : DB connection (defaults to config(database.default))}
{--exact : Exact COUNT(*) per table instead of the fast estimate}
{--limit=15 : How many of the largest tables to list}
{--models : Only show per-model counts}
{--tables : Only show per-table sizes}
{--json : Output machine-readable JSON}';
protected $description = 'Show a database overview: per-model row counts, largest tables, total size, queue + backup summary.';
public function handle(): int
{
$connection = $this->option('connection') ?: config('database.default');
if (! config("database.connections.{$connection}")) {
$this->error("Unknown database connection: {$connection}");
return self::FAILURE;
}
$exact = (bool) $this->option('exact');
$limit = max(1, (int) $this->option('limit'));
$onlyModels = (bool) $this->option('models');
$onlyTables = (bool) $this->option('tables');
$showAll = ! $onlyModels && ! $onlyTables;
$data = ['connection' => $connection];
if ($onlyModels || $showAll) {
$data['models'] = StatsService::modelCounts(null, $exact, $connection);
}
if ($onlyTables || $showAll) {
$data['tables'] = array_slice(StatsService::tableStats($connection), 0, $limit);
$data['database'] = StatsService::databaseSize($connection);
}
if ($showAll) {
$data['queue'] = $this->safe(fn () => $this->queueSummary());
$data['backup'] = $this->safe(fn () => $this->backupSummary());
}
if ($this->option('json')) {
$this->line((string) json_encode($data, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES));
return self::SUCCESS;
}
$this->render($data, $exact);
return self::SUCCESS;
}
private function render(array $data, bool $exact): void
{
if (isset($data['models'])) {
$this->info('Models'.($exact ? ' (exact)' : ' (estimated)'));
$rows = [];
foreach ($data['models'] as $m) {
$rows[] = [
$m['model'],
$m['table'],
($m['error'] !== null || $m['count'] === null) ? '—' : number_format((int) $m['count']).($m['estimated'] ? ' ~' : ''),
];
}
$rows === []
? $this->line(' (no models discovered — set config workkit.stats.models)')
: $this->table(['Model', 'Table', 'Rows'], $rows);
}
if (isset($data['tables'])) {
$this->info('Largest tables');
$rows = [];
foreach ($data['tables'] as $t) {
$rows[] = [
$t['table'],
$t['rows'] === null ? '—' : number_format((int) $t['rows']),
$t['bytes'] === null ? '—' : BackupService::humanBytes((int) $t['bytes']),
$t['engine'] ?? '—',
];
}
$rows === []
? $this->line(' (no tables found)')
: $this->table(['Table', 'Rows', 'Size', 'Engine'], $rows);
if (isset($data['database'])) {
$db = $data['database'];
$this->line(sprintf(
'Database: %d tables, %s total',
$db['tables'],
$db['bytes'] === null ? 'size n/a (non-MySQL)' : BackupService::humanBytes((int) $db['bytes']),
));
}
}
if (! empty($data['queue'])) {
$q = $data['queue'];
$this->line(sprintf('Queue: %s pending, %s failed (24h)', $q['pending'] ?? '?', $q['failed_recent'] ?? '?'));
}
if (! empty($data['backup'])) {
$b = $data['backup'];
$this->line(sprintf('Backups: %d on disk, newest %s', $b['count'] ?? 0, $b['newest_age'] ?? 'none'));
}
}
private function queueSummary(): array
{
$snap = QueueHealthService::inspect();
$pending = 0;
foreach ($snap['queues'] as $q) {
$pending += (int) $q['pending'];
}
return ['pending' => $pending, 'failed_recent' => $snap['failed_recent']];
}
private function backupSummary(): array
{
$files = BackupService::backupFiles();
$newest = $files[0] ?? null;
$age = null;
if ($newest !== null) {
$secs = max(0, time() - (int) filemtime($newest));
$age = $this->humanAge($secs);
}
return ['count' => count($files), 'newest_age' => $age];
}
private function humanAge(int $secs): string
{
if ($secs < 90) {
return $secs.'s ago';
}
if ($secs < 5400) {
return intdiv($secs, 60).'m ago';
}
if ($secs < 172800) {
return intdiv($secs, 3600).'h ago';
}
return intdiv($secs, 86400).'d ago';
}
/** Run a best-effort summary probe; return null if it throws. */
private function safe(callable $fn): ?array
{
try {
return $fn();
} catch (Throwable $e) {
return null;
}
}
}

View File

@ -0,0 +1,120 @@
<?php
declare(strict_types=1);
namespace Blax\Workkit\Commands\Database;
use Blax\Workkit\Services\BackupService;
use Illuminate\Console\Command;
/**
* Check that a backup is intact WITHOUT touching the database. Streams
* openssl decrypt xz integrity-test tail, so it proves the file
* decrypts with this host's APP_KEY and is a complete, non-truncated xz
* stream, and (for mysqldump output) that it ends with the
* "-- Dump completed" marker. When a .meta.json sidecar exists it also
* confirms the on-disk sha256 still matches what was recorded at backup
* time.
*
* This is exactly the check that distinguishes "the backup is corrupt"
* from "the import failed for schema reasons" the misdiagnosis that
* otherwise gets a perfectly good backup deleted.
*
* Caveat: integrity restorability. A valid backup of the wrong (or
* empty) database verifies "OK"; verify proves the bytes are sound, not
* that the SQL will apply cleanly against a given schema.
*/
class VerifyCommand extends Command
{
protected $signature = 'workkit:db:verify
{--file= : Specific backup filename inside the backups directory (default: newest)}
{--all : Verify every backup in the backups directory}';
protected $description = 'Verify a backup decrypts + is a valid (complete) xz stream, without importing it.';
public function handle(): int
{
$targets = $this->resolveTargets();
if ($targets === null) {
return self::FAILURE;
}
if ($targets === []) {
$this->warn('No backups found to verify.');
return self::SUCCESS;
}
$ok = 0;
$bad = 0;
$incomplete = 0;
foreach ($targets as $file) {
$r = BackupService::verify($file);
$label = basename($file).' ('.BackupService::humanBytes((int) $r['bytes']).')';
if (! $r['integrity_ok']) {
$bad++;
$this->error("{$label}");
$this->line(' '.$r['message']);
continue;
}
if ($r['checksum_matches'] === false) {
$bad++;
$this->error("{$label}");
$this->line(' '.$r['message']);
continue;
}
if (! $r['looks_complete']) {
$incomplete++;
$this->warn("{$label}");
$this->line(' '.$r['message']);
continue;
}
$ok++;
$suffix = $r['checksum_matches'] === true ? ' (checksum matches sidecar)' : '';
$this->info("{$label}{$suffix}");
}
$this->newLine();
$this->line(sprintf('Verified: %d OK, %d incomplete/uncertain, %d failed.', $ok, $incomplete, $bad));
return $bad > 0 ? self::FAILURE : self::SUCCESS;
}
/**
* @return string[]|null list of files to verify, or null on a hard error
*/
private function resolveTargets(): ?array
{
if ($this->option('all')) {
return BackupService::backupFiles();
}
$base = BackupService::backupDirectory();
$opt = $this->option('file');
if ($opt) {
if (is_file($opt)) {
return [$opt];
}
$candidate = $base.'/'.ltrim((string) $opt, '/');
if (! is_file($candidate)) {
$this->error("Backup file not found: {$opt}");
return null;
}
return [$candidate];
}
$files = BackupService::backupFiles();
return $files === [] ? [] : [$files[0]];
}
}

View File

@ -0,0 +1,103 @@
<?php
declare(strict_types=1);
namespace Blax\Workkit\Commands\Queue;
use Blax\Workkit\Services\QueueHealthService;
use Illuminate\Console\Command;
/**
* Report queue health and exit non-zero when a threshold is breached, so it
* drops straight into cron / a monitoring probe:
*
* * * * * * php artisan workkit:queue:health --json || alert
*
* For the database queue driver it shows per-queue depth, due/delayed/reserved
* counts and the oldest due job's age, and flags a STALLED queue (due work past
* the age limit with nothing reserved the worker is probably down). Failed
* jobs (total + last 24h) are reported on any driver.
*/
class QueueHealthCommand extends Command
{
protected $signature = 'workkit:queue:health
{--connection= : Queue connection (defaults to config(queue.default))}
{--max-age= : Stall threshold in minutes (default: workkit.queue.max_age_minutes)}
{--max-depth= : Per-queue backlog threshold (default: workkit.queue.max_depth)}
{--max-failed= : Max failed jobs in the window (default: workkit.queue.max_failed)}
{--window=24 : Failed-jobs lookback window in hours}
{--json : Output machine-readable JSON}';
protected $description = 'Report queue depth/age/failed-jobs health; exit non-zero on any threshold breach.';
public function handle(): int
{
$connection = $this->option('connection') ?: config('queue.default');
$window = max(1, (int) $this->option('window'));
$snapshot = QueueHealthService::inspect($connection, $window);
$thresholds = QueueHealthService::thresholds();
foreach (['max-age' => 'max_age_minutes', 'max-depth' => 'max_depth', 'max-failed' => 'max_failed'] as $opt => $key) {
$val = $this->option($opt);
if ($val !== null) {
$thresholds[$key] = (int) $val;
}
}
$result = QueueHealthService::evaluate($snapshot, $thresholds);
if ($this->option('json')) {
$this->line((string) json_encode(
['snapshot' => $snapshot, 'thresholds' => $thresholds] + $result,
JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES
));
return $result['ok'] ? self::SUCCESS : self::FAILURE;
}
$this->line(sprintf('Queue connection: %s (driver: %s)', $snapshot['connection'], $snapshot['driver'] ?? 'unknown'));
if ($snapshot['supported']) {
if ($snapshot['queues'] === []) {
$this->line(' (no jobs queued)');
} else {
$rows = [];
foreach ($snapshot['queues'] as $name => $q) {
$rows[] = [
$name,
number_format($q['pending']),
number_format($q['due']),
number_format($q['delayed']),
number_format($q['reserved']),
$q['oldest_due_age'] === null ? '—' : intdiv($q['oldest_due_age'], 60).'m',
];
}
$this->table(['Queue', 'Pending', 'Due', 'Delayed', 'Reserved', 'Oldest due'], $rows);
}
} else {
$this->warn(sprintf('Per-queue introspection unsupported for the "%s" driver — showing failed jobs only.', $snapshot['driver'] ?? 'unknown'));
}
$this->line(sprintf(
'Failed jobs: %s total, %s in last %dh',
$snapshot['failed_total'] ?? 'n/a',
$snapshot['failed_recent'] ?? 'n/a',
$snapshot['failed_window_hours'],
));
if ($result['ok']) {
$this->info('Queue health: OK');
return self::SUCCESS;
}
$this->newLine();
$this->error('Queue health: '.count($result['breaches']).' issue(s)');
foreach ($result['breaches'] as $b) {
$this->line(' • '.$b['message']);
}
return self::FAILURE;
}
}

View File

@ -28,12 +28,31 @@ use RuntimeException;
* Required system binaries: `mysqldump`, `mysql`, `xz`, `openssl`, * Required system binaries: `mysqldump`, `mysql`, `xz`, `openssl`,
* `bash` (for `set -o pipefail`). All are standard on every reasonable * `bash` (for `set -o pipefail`). All are standard on every reasonable
* Linux server. * Linux server.
*
* IMPORTANT single source of truth for the target server: every path
* that touches MySQL (dump, import, drop, table/size queries) builds its
* client invocation from the SAME `$cfg` array via {@see mysqlClient()}.
* Never mix in Laravel's PDO connection for one operation and the raw
* `$cfg` CLI for another a `url` DSN, `unix_socket`, or read/write split
* can make those two resolve to *different databases*, and a restore that
* drops one schema while importing into another is exactly the
* catastrophe these commands exist to prevent.
*/ */
class BackupService class BackupService
{ {
public const CIPHER = 'aes-256-cbc'; public const CIPHER = 'aes-256-cbc';
public const PBKDF2_ITER = 600000; public const PBKDF2_ITER = 600000;
/** Canonical extension every backup this package writes carries. */
public const BACKUP_EXT = '.sql.xz.enc';
/** Suffix marking a pre-restore safety snapshot (excluded from auto-restore). */
public const PRE_RESTORE_MARK = '.pre-restore';
/** Sidecar metadata suffix written next to each backup (best-effort). */
public const META_EXT = '.meta.json';
/** /**
* mysqldump xz openssl enc $outPath. One pipeline, zero PHP * mysqldump xz openssl enc $outPath. One pipeline, zero PHP
* memory pressure. Pipefail propagates a failure in any stage out * memory pressure. Pipefail propagates a failure in any stage out
@ -86,12 +105,30 @@ class BackupService
if (! file_exists($outPath) || filesize($outPath) === 0) { if (! file_exists($outPath) || filesize($outPath) === 0) {
throw new RuntimeException("Backup pipeline produced no output at {$outPath}"); throw new RuntimeException("Backup pipeline produced no output at {$outPath}");
} }
// Best-effort metadata sidecar — never fail a good backup over it.
self::writeMeta($outPath, $cfg, $level);
} }
/** /**
* openssl dec xz -d mysql. Streams through the same pipeline * openssl dec xz -d mysql. Streams through the same pipeline
* in reverse. Does no PHP-side decoding so the file size is bounded * in reverse. Does no PHP-side decoding so the file size is bounded
* only by disk I/O. * only by disk I/O.
*
* A `SET FOREIGN_KEY_CHECKS=0;` line is prepended to the SQL stream so
* that even a hand-rolled dump lacking mysqldump's own header imports
* regardless of table order (forward FK references resolve once every
* table exists). This is harmless for mysqldump output, which sets and
* restores its own session vars, and the whole import is a single
* short-lived mysql session so the relaxed checks never leak out.
*
* NOTE: this does NOT rescue a restore onto a *dirty* schema whose
* leftover tables have incompatible column types MySQL still raises
* errno 150 / error 3780 at CREATE TABLE because the referenced table
* already exists with the wrong type. For that case the target must be
* empty first (see RestoreCommand --fresh). The openssl|xz stages stay
* a real pipeline segment so `set -o pipefail` still surfaces a bad key
* or corrupt file as a non-zero exit.
*/ */
public static function decryptDecompressImport(string $inPath, array $cfg): void public static function decryptDecompressImport(string $inPath, array $cfg): void
{ {
@ -100,30 +137,16 @@ class BackupService
self::requireBinary('openssl'); self::requireBinary('openssl');
self::requireBinary('bash'); self::requireBinary('bash');
// Sanity check on the file's magic bytes. `openssl enc -salt` self::assertLooksEncrypted($inPath);
// output always starts with the literal "Salted__" header.
$head = (string) @file_get_contents($inPath, false, null, 0, 8);
if ($head !== 'Salted__') {
throw new RuntimeException(
"File at {$inPath} doesn't look like a streaming openssl backup. "
. 'Legacy backups (made with the pre-streaming Crypt envelope) need '
. 'a separate restore path; see workkit:db:restore-legacy or restore '
. 'manually with `Crypt::decryptString` after raising memory_limit.'
);
}
$openssl = 'openssl enc -d -' . self::CIPHER $openssl = self::opensslDecrypt($inPath);
. ' -pbkdf2 -iter ' . self::PBKDF2_ITER $mysql = self::mysqlClient($cfg);
. ' -pass env:WK_KEY '
. '-in ' . escapeshellarg($inPath);
$mysql = 'mysql ' // The brace group's exit status is that of its last command (the
. '--user=' . escapeshellarg((string) ($cfg['username'] ?? 'root')) . ' ' // openssl|xz pipeline), so under `set -o pipefail` a decrypt/decompress
. '--host=' . escapeshellarg((string) ($cfg['host'] ?? 'localhost')) . ' ' // failure still propagates out as the head of the outer pipeline.
. '--port=' . escapeshellarg((string) ($cfg['port'] ?? 3306)) . ' ' $payload = "{ printf 'SET FOREIGN_KEY_CHECKS=0;\\n'; {$openssl} | xz -d -T0; }";
. escapeshellarg((string) $cfg['database']); $pipeline = "{$payload} | {$mysql}";
$pipeline = "{$openssl} | xz -d -T0 | {$mysql}";
$cmd = '/bin/bash -c '.escapeshellarg('set -o pipefail; '.$pipeline); $cmd = '/bin/bash -c '.escapeshellarg('set -o pipefail; '.$pipeline);
self::run($cmd, [ self::run($cmd, [
@ -132,6 +155,217 @@ class BackupService
]); ]);
} }
/**
* Drop every base table and view in the target database, then leave
* the (empty) database in place. Built entirely from $cfg via the same
* mysql client the import uses, and scoped to `DATABASE()` (the schema
* the CLI connects to) so the wipe provably hits the same server +
* schema the subsequent import writes to.
*
* Implemented as in-SQL prepared statements rather than DROP DATABASE
* so it never needs CREATE privilege and preserves the database's
* charset/collation and any grants attached to it.
*/
public static function dropAllTables(array $cfg): void
{
self::requireBinary('mysql');
self::requireBinary('bash');
// Tables and views dropped in two single statements (GROUP_CONCAT
// builds `a`,`b`,`c`); FK checks off so table order is irrelevant.
$sql = <<<'SQL'
SET FOREIGN_KEY_CHECKS = 0;
SET SESSION group_concat_max_len = 1000000000;
SET @t := (SELECT GROUP_CONCAT(CONCAT('`', table_name, '`')) FROM information_schema.tables WHERE table_schema = DATABASE() AND table_type = 'BASE TABLE');
SET @v := (SELECT GROUP_CONCAT(CONCAT('`', table_name, '`')) FROM information_schema.tables WHERE table_schema = DATABASE() AND table_type = 'VIEW');
SET @dt := IF(@t IS NULL, 'SELECT 1', CONCAT('DROP TABLE IF EXISTS ', @t));
PREPARE s1 FROM @dt; EXECUTE s1; DEALLOCATE PREPARE s1;
SET @dv := IF(@v IS NULL, 'SELECT 1', CONCAT('DROP VIEW IF EXISTS ', @v));
PREPARE s2 FROM @dv; EXECUTE s2; DEALLOCATE PREPARE s2;
SET FOREIGN_KEY_CHECKS = 1;
SQL;
$pipeline = 'printf %s '.escapeshellarg($sql).' | '.self::mysqlClient($cfg);
$cmd = '/bin/bash -c '.escapeshellarg('set -o pipefail; '.$pipeline);
self::run($cmd, ['MYSQL_PWD' => (string) ($cfg['password'] ?? '')]);
}
/**
* Verify a backup without touching MySQL: decrypt + fully decompress and
* inspect the tail. Returns a structured result rather than throwing so
* callers (workkit:db:verify, the --fresh pre-drop gate) can branch on it.
*
* What it proves:
* - integrity_ok : decrypts with the current APP_KEY AND is a valid,
* non-truncated xz stream (xz -d fails otherwise).
* - looks_complete : the plaintext ends with mysqldump's
* "-- Dump completed" marker catches a dump that
* was truncated AT SOURCE (mysqldump crashed mid-run)
* yet compressed cleanly, which xz integrity can't.
* - checksum_matches: on-disk sha256 equals the value recorded in the
* .meta.json sidecar (proves the file wasn't altered
* on disk since it was written), or null if no sidecar.
*
* It does NOT prove the dump is restorable against a given schema a
* valid backup of the wrong/empty database still verifies "OK".
*
* @return array{file:string,bytes:int,integrity_ok:bool,looks_complete:bool,sha256:?string,recorded_sha256:?string,checksum_matches:?bool,message:string}
*/
public static function verify(string $path): array
{
self::requireBinary('openssl');
self::requireBinary('xz');
self::requireBinary('bash');
$result = [
'file' => $path,
'bytes' => is_file($path) ? (int) filesize($path) : 0,
'integrity_ok' => false,
'looks_complete' => false,
'sha256' => null,
'recorded_sha256' => null,
'checksum_matches' => null,
'message' => '',
];
if (! is_file($path)) {
$result['message'] = "Backup file not found: {$path}";
return $result;
}
$head = (string) @file_get_contents($path, false, null, 0, 8);
if ($head !== 'Salted__') {
$result['message'] = "Not a streaming openssl backup (missing 'Salted__' header). "
.'Legacy Crypt-envelope backups need a different restore path.';
return $result;
}
// Decrypt + decompress in one streaming pass, keeping only the tail
// for the completion-marker check. xz -d fails on truncation/bit-rot.
$pipeline = self::opensslDecrypt($path).' | xz -d -T0 | tail -c 512';
$cmd = '/bin/bash -c '.escapeshellarg('set -o pipefail; '.$pipeline);
[$exit, $stdout, $stderr] = self::runResult($cmd, ['WK_KEY' => self::passphrase()]);
if ($exit !== 0) {
if (str_contains($stderr, 'bad decrypt') || str_contains($stderr, 'bad magic')) {
$result['message'] = 'Decryption failed — APP_KEY likely does not match the host that produced this backup.';
} else {
// openssl can (rarely, ~1/256) yield valid PKCS#7 padding on a
// wrong key and exit 0, leaving xz to choke on garbage — so a
// post-decrypt xz failure is ambiguous between the two causes.
$result['message'] = 'Decrypted but the stream is not valid xz — the backup is corrupt OR '
.'(rarely) the APP_KEY is wrong and the padding happened to validate. Verify APP_KEY first. '
.'stderr: '.trim($stderr);
}
return $result;
}
$result['integrity_ok'] = true;
$result['looks_complete'] = str_contains($stdout, '-- Dump completed');
// sha256 + sidecar comparison (best-effort; streams the file).
$sha = @hash_file('sha256', $path);
$result['sha256'] = $sha === false ? null : $sha;
$meta = self::readMeta($path);
if ($meta !== null && isset($meta['sha256']) && is_string($meta['sha256'])) {
$result['recorded_sha256'] = $meta['sha256'];
if ($result['sha256'] !== null) {
$result['checksum_matches'] = hash_equals($meta['sha256'], $result['sha256']);
}
}
if ($result['checksum_matches'] === false) {
$result['message'] = 'CHECKSUM MISMATCH — the file changed on disk since it was written (integrity of the xz stream is still OK).';
} elseif (! $result['looks_complete']) {
$result['message'] = 'Integrity OK (decrypts + valid xz) but no mysqldump completion marker — '
.'may be truncated at source or a non-mysqldump dump.';
} else {
$result['message'] = 'OK — decrypts, valid xz, mysqldump completion marker present'
.($result['checksum_matches'] === true ? ', checksum matches sidecar.' : '.');
}
return $result;
}
/**
* Count base tables + views in the target database, via the same mysql
* client the import/drop use. Returns null if the query can't run (no
* connection, database missing) so callers can treat it as "unknown".
*/
public static function countTables(array $cfg): ?int
{
$val = self::scalarQuery(
$cfg,
'SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = DATABASE()'
);
return $val === null ? null : (int) $val;
}
/**
* Approximate on-disk size (bytes) of the target database from
* information_schema. Null if unavailable. Used only for a best-effort
* pre-snapshot disk-space sanity check.
*/
public static function estimateDatabaseBytes(array $cfg): ?int
{
$val = self::scalarQuery(
$cfg,
'SELECT IFNULL(SUM(data_length + index_length), 0) FROM information_schema.tables WHERE table_schema = DATABASE()'
);
return $val === null ? null : (int) $val;
}
/**
* Real backup files in the backup directory, newest first. Excludes
* `.meta.json` sidecars but KEEPS custom `--out`-named backups (we only
* deny the sidecar suffix rather than allow-listing one extension, so a
* consumer's nightly.enc / db.bak is still found and pruned).
*
* @return string[] absolute paths, newest (by mtime) first
*/
public static function backupFiles(): array
{
$base = self::backupDirectory();
$files = glob($base.'/*') ?: [];
$files = array_values(array_filter($files, function ($f) {
return is_file($f) && ! str_ends_with($f, self::META_EXT);
}));
usort($files, fn ($a, $b) => filemtime($b) <=> filemtime($a));
return $files;
}
/**
* Like {@see backupFiles()} but also drops pre-restore safety snapshots,
* so the "newest backup" auto-resolver never silently picks a snapshot of
* a known-bad database taken moments before a wipe.
*
* @return string[]
*/
public static function restoreCandidates(): array
{
return array_values(array_filter(
self::backupFiles(),
fn ($f) => ! str_contains(basename($f), self::PRE_RESTORE_MARK.'.')
));
}
/**
* Derive the metadata sidecar path for a backup file. Always
* backup sidecar (never the reverse), so a backup with no sidecar is
* still handled and a lone sidecar is recognisably an orphan.
*/
public static function metaPathFor(string $backupPath): string
{
return $backupPath.self::META_EXT;
}
/** /**
* Derive the openssl passphrase from APP_KEY. Laravel stores APP_KEY * Derive the openssl passphrase from APP_KEY. Laravel stores APP_KEY
* as "base64:<random-bytes>"; we strip the prefix and feed the rest * as "base64:<random-bytes>"; we strip the prefix and feed the rest
@ -147,6 +381,7 @@ class BackupService
if (str_starts_with($key, 'base64:')) { if (str_starts_with($key, 'base64:')) {
$key = substr($key, 7); $key = substr($key, 7);
} }
return $key; return $key;
} }
@ -191,7 +426,7 @@ class BackupService
throw new RuntimeException(sprintf( throw new RuntimeException(sprintf(
"Backup directory not writable: %s\n" "Backup directory not writable: %s\n"
." owner=%s, current user=%s\n" ." owner=%s, current user=%s\n"
. " Fix as root: chown -R %s %s && chmod 0775 %s", .' Fix as root: chown -R %s %s && chmod 0775 %s',
$path, $path,
$owner, $owner,
$current, $current,
@ -219,11 +454,34 @@ class BackupService
/** /**
* Run a shell command and throw on non-zero exit, capturing stderr * Run a shell command and throw on non-zero exit, capturing stderr
* for the error message. Env vars are passed via proc_open's env * for the error message.
* arg they're scoped to the child process and not visible in the
* host's `ps` listing.
*/ */
public static function run(string $command, array $env = []): void public static function run(string $command, array $env = []): void
{
[$exit, , $stderr] = self::runResult($command, $env);
if ($exit !== 0) {
throw new RuntimeException(sprintf(
"Command failed (exit %d):\n %s\nstderr:\n%s",
$exit,
self::redactCommand($command),
trim((string) $stderr) ?: '(empty)'
));
}
}
/**
* Run a shell command and return [exitCode, stdout, stderr] without
* throwing. Env vars are passed via proc_open's env arg scoped to the
* child process, never visible in the host's `ps` listing.
*
* stdout is read fully; in this package the only stdout producers are
* tiny (a `tail -c 512`, a scalar query, a row of SHOW output), so this
* stays well within the zero-DB-bytes-in-PHP invariant. stdout and stderr
* are drained concurrently (non-blocking + stream_select) so a child that
* fills both pipe buffers at once can never deadlock against a serial read.
*/
public static function runResult(string $command, array $env = []): array
{ {
$descriptors = [ $descriptors = [
0 => ['pipe', 'r'], 0 => ['pipe', 'r'],
@ -238,19 +496,161 @@ class BackupService
} }
fclose($pipes[0]); fclose($pipes[0]);
fclose($pipes[1]); stream_set_blocking($pipes[1], false);
$stderr = stream_get_contents($pipes[2]); stream_set_blocking($pipes[2], false);
fclose($pipes[2]);
$stdout = '';
$stderr = '';
$open = [1 => $pipes[1], 2 => $pipes[2]];
while ($open !== []) {
$read = array_values($open);
$write = null;
$except = null;
if (@stream_select($read, $write, $except, null) === false) {
break;
}
foreach ($read as $stream) {
$chunk = fread($stream, 8192);
if (($chunk === '' || $chunk === false) && feof($stream)) {
$key = array_search($stream, $open, true);
if ($key !== false) {
fclose($stream);
unset($open[$key]);
}
continue;
}
if ($chunk === false) {
continue;
}
if ($stream === $pipes[1]) {
$stdout .= $chunk;
} else {
$stderr .= $chunk;
}
}
}
// Close anything stream_select bailed on, then reap the process.
foreach ($open as $stream) {
fclose($stream);
}
$exit = proc_close($proc); $exit = proc_close($proc);
if ($exit !== 0) { return [$exit, $stdout, $stderr];
throw new RuntimeException(sprintf(
"Command failed (exit %d):\n %s\nstderr:\n%s",
$exit,
self::redactCommand($command),
trim((string) $stderr) ?: '(empty)'
));
} }
/** Human-readable byte size, e.g. 13631488 → "13.00 MB". */
public static function humanBytes(int $bytes): string
{
$units = ['B', 'KB', 'MB', 'GB', 'TB'];
$i = 0;
$value = (float) $bytes;
while ($value >= 1024 && $i < count($units) - 1) {
$value /= 1024;
$i++;
}
return sprintf('%.2f %s', $value, $units[$i]);
}
// ---- internals ---------------------------------------------------------
/**
* Build the mysql client invocation from $cfg. Single source of truth
* for the target server across import, drop and queries.
*/
private static function mysqlClient(array $cfg): string
{
return 'mysql '
.'--user='.escapeshellarg((string) ($cfg['username'] ?? 'root')).' '
.'--host='.escapeshellarg((string) ($cfg['host'] ?? 'localhost')).' '
.'--port='.escapeshellarg((string) ($cfg['port'] ?? 3306)).' '
.escapeshellarg((string) $cfg['database']);
}
/** The openssl decrypt stage, reading from $inPath. */
private static function opensslDecrypt(string $inPath): string
{
return 'openssl enc -d -'.self::CIPHER
.' -pbkdf2 -iter '.self::PBKDF2_ITER
.' -pass env:WK_KEY '
.'-in '.escapeshellarg($inPath);
}
/** Sanity-check the openssl `-salt` magic header. */
private static function assertLooksEncrypted(string $inPath): void
{
$head = (string) @file_get_contents($inPath, false, null, 0, 8);
if ($head !== 'Salted__') {
throw new RuntimeException(
"File at {$inPath} doesn't look like a streaming openssl backup. "
.'Legacy backups (made with the pre-streaming Crypt envelope) need '
.'a separate restore path; see workkit:db:restore-legacy or restore '
.'manually with `Crypt::decryptString` after raising memory_limit.'
);
}
}
/**
* Run a single-row, single-column query against $cfg and return the
* trimmed scalar, or null if the query fails (no connection / missing db).
*/
private static function scalarQuery(array $cfg, string $sql): ?string
{
self::requireBinary('mysql');
self::requireBinary('bash');
$pipeline = self::mysqlClient($cfg).' -N -e '.escapeshellarg($sql);
$cmd = '/bin/bash -c '.escapeshellarg('set -o pipefail; '.$pipeline);
[$exit, $stdout] = self::runResult($cmd, ['MYSQL_PWD' => (string) ($cfg['password'] ?? '')]);
if ($exit !== 0) {
return null;
}
return trim($stdout);
}
/**
* Write a best-effort metadata sidecar next to a freshly-created backup.
* Never throws: a backup is valid with or without its sidecar.
*/
private static function writeMeta(string $backupPath, array $cfg, int $xzLevel): void
{
try {
$sha = @hash_file('sha256', $backupPath);
$meta = [
'file' => basename($backupPath),
'database' => $cfg['database'] ?? null,
'host' => $cfg['host'] ?? null,
'created_at' => date('c'),
'bytes' => (int) @filesize($backupPath),
'sha256' => $sha === false ? null : $sha,
'cipher' => self::CIPHER,
'pbkdf2_iter' => self::PBKDF2_ITER,
'xz_level' => $xzLevel,
];
@file_put_contents(
self::metaPathFor($backupPath),
json_encode($meta, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES)
);
} catch (\Throwable $e) {
// Sidecar is a nicety, not a requirement.
}
}
/** Read a backup's metadata sidecar, or null if absent/unreadable. */
private static function readMeta(string $backupPath): ?array
{
$metaPath = self::metaPathFor($backupPath);
if (! is_file($metaPath)) {
return null;
}
$decoded = json_decode((string) @file_get_contents($metaPath), true);
return is_array($decoded) ? $decoded : null;
} }
/** /**

View File

@ -0,0 +1,211 @@
<?php
declare(strict_types=1);
namespace Blax\Workkit\Services;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Schema;
use Throwable;
/**
* Driver-agnostic queue health introspection, generalised from the learn-atc
* admin "Doctor" queue checks but stripped of app-specific bits (no Activity
* notices, no Redis heartbeat dependency, no job-payload decoding).
*
* For the `database` queue driver it reads the jobs table directly and reports,
* per queue: total pending, due (ready), delayed, reserved (in-flight) and the
* age of the oldest due job. From those it infers a STALLED queue due work
* piled up past the age threshold with nothing reserved which is the
* driver-agnostic proxy for "the worker is down" that needs no heartbeat.
*
* For other drivers per-queue introspection isn't available, so it degrades to
* the failed-jobs table only and marks per-queue stats unsupported.
*
* {@see evaluate()} is a pure function of (snapshot, thresholds) breaches, so
* the alarming logic is unit-testable without a database.
*/
class QueueHealthService
{
/**
* Inspect a queue connection.
*
* @return array{
* connection:string, driver:?string, supported:bool,
* queues:array<string, array{pending:int, due:int, delayed:int, reserved:int, oldest_due_age:?int}>,
* failed_total:?int, failed_recent:?int, failed_window_hours:int, now:int
* }
*/
public static function inspect(?string $connection = null, int $failedWindowHours = 24): array
{
$connection = $connection ?: config('queue.default');
$cfg = config("queue.connections.{$connection}", []);
$driver = $cfg['driver'] ?? null;
$now = time();
$snapshot = [
'connection' => (string) $connection,
'driver' => $driver,
'supported' => false,
'queues' => [],
'failed_total' => null,
'failed_recent' => null,
'failed_window_hours' => $failedWindowHours,
'errors' => [],
'now' => $now,
];
if ($driver === 'database') {
$dbConn = $cfg['connection'] ?? config('database.default');
$table = $cfg['table'] ?? 'jobs';
try {
// hasTable() === false is fine (a fresh app with no jobs table);
// only a thrown error means the database is actually unreachable,
// which must surface as a breach so cron doesn't read OK.
if (Schema::connection($dbConn)->hasTable($table)) {
$snapshot['supported'] = true;
$snapshot['queues'] = self::queueStats($dbConn, $table, $now);
}
} catch (Throwable $e) {
$snapshot['errors'][] = 'queue introspection failed: '.$e->getMessage();
}
}
// Failed jobs live in the DB regardless of the queue driver.
$failed = self::failedJobStats($failedWindowHours);
$snapshot['failed_total'] = $failed['total'];
$snapshot['failed_recent'] = $failed['recent'];
if ($failed['error'] !== null) {
$snapshot['errors'][] = $failed['error'];
}
return $snapshot;
}
/**
* Evaluate a snapshot against thresholds. Pure no I/O.
*
* @param array $snapshot output of {@see inspect()}
* @param array{max_depth:int, max_depth_per_queue:array<string,int>, max_age_minutes:int, max_failed:int} $thresholds
* @return array{ok:bool, breaches:array<int, array{type:string, queue:?string, message:string}>}
*/
public static function evaluate(array $snapshot, array $thresholds): array
{
$maxDepth = (int) ($thresholds['max_depth'] ?? 250);
$perQueue = (array) ($thresholds['max_depth_per_queue'] ?? []);
$maxAgeSec = (int) ($thresholds['max_age_minutes'] ?? 15) * 60;
$maxFailed = (int) ($thresholds['max_failed'] ?? 25);
$breaches = [];
foreach (($snapshot['queues'] ?? []) as $name => $q) {
$limit = array_key_exists($name, $perQueue) ? (int) $perQueue[$name] : $maxDepth;
if (($q['pending'] ?? 0) > $limit) {
$breaches[] = [
'type' => 'depth',
'queue' => $name,
'message' => sprintf("queue '%s' depth %d exceeds %d", $name, $q['pending'], $limit),
];
}
// Stalled: due work older than the age limit, nothing in flight.
$oldest = $q['oldest_due_age'] ?? null;
if (($q['due'] ?? 0) > 0 && (int) ($q['reserved'] ?? 0) === 0 && $oldest !== null && $oldest > $maxAgeSec) {
$breaches[] = [
'type' => 'stalled',
'queue' => $name,
'message' => sprintf(
"queue '%s' looks STALLED: oldest due job is %dm old with 0 reserved (worker down?)",
$name,
intdiv($oldest, 60),
),
];
}
}
$recent = $snapshot['failed_recent'] ?? null;
if ($recent !== null && $recent > $maxFailed) {
$breaches[] = [
'type' => 'failed',
'queue' => null,
'message' => sprintf('%d jobs failed in the last %dh (max %d)', $recent, (int) ($snapshot['failed_window_hours'] ?? 24), $maxFailed),
];
}
// A DB/connection error (not a merely-absent table) must surface as a
// breach so a monitoring probe exits non-zero instead of reading OK.
foreach (($snapshot['errors'] ?? []) as $err) {
$breaches[] = ['type' => 'error', 'queue' => null, 'message' => $err];
}
return ['ok' => $breaches === [], 'breaches' => $breaches];
}
/** Thresholds from config, each read with an explicit code-side default. */
public static function thresholds(): array
{
return [
'max_depth' => (int) config('workkit.queue.max_depth', 250),
'max_depth_per_queue' => (array) config('workkit.queue.max_depth_per_queue', []),
'max_age_minutes' => (int) config('workkit.queue.max_age_minutes', 15),
'max_failed' => (int) config('workkit.queue.max_failed', 25),
];
}
// ---- internals ---------------------------------------------------------
/**
* Per-queue conditional aggregates in a single grouped query (works on
* MySQL + SQLite + Postgres; jobs timestamps are unix ints).
*
* @return array<string, array{pending:int, due:int, delayed:int, reserved:int, oldest_due_age:?int}>
*/
private static function queueStats(string $dbConn, string $table, int $now): array
{
$rows = DB::connection($dbConn)->table($table)
->selectRaw('queue')
->selectRaw('COUNT(*) as pending')
->selectRaw('SUM(CASE WHEN reserved_at IS NOT NULL THEN 1 ELSE 0 END) as reserved')
->selectRaw('SUM(CASE WHEN reserved_at IS NULL AND available_at <= ? THEN 1 ELSE 0 END) as due', [$now])
->selectRaw('SUM(CASE WHEN reserved_at IS NULL AND available_at > ? THEN 1 ELSE 0 END) as delayed', [$now])
->selectRaw('MIN(CASE WHEN reserved_at IS NULL AND available_at <= ? THEN available_at END) as oldest_due', [$now])
->groupBy('queue')
->get();
$out = [];
foreach ($rows as $r) {
$oldest = $r->oldest_due;
$out[(string) $r->queue] = [
'pending' => (int) $r->pending,
'due' => (int) $r->due,
'delayed' => (int) $r->delayed,
'reserved' => (int) $r->reserved,
'oldest_due_age' => $oldest === null ? null : max(0, $now - (int) $oldest),
];
}
return $out;
}
/**
* @return array{total:?int, recent:?int, error:?string}
*/
private static function failedJobStats(int $windowHours): array
{
$dbConn = config('queue.failed.database') ?: config('database.default');
$table = config('queue.failed.table', 'failed_jobs');
try {
if (! Schema::connection($dbConn)->hasTable($table)) {
return ['total' => null, 'recent' => null, 'error' => null];
}
$total = DB::connection($dbConn)->table($table)->count();
$since = date('Y-m-d H:i:s', time() - $windowHours * 3600);
$recent = DB::connection($dbConn)->table($table)->where('failed_at', '>=', $since)->count();
return ['total' => $total, 'recent' => $recent, 'error' => null];
} catch (Throwable $e) {
return ['total' => null, 'recent' => null, 'error' => 'failed-jobs query failed: '.$e->getMessage()];
}
}
}

View File

@ -0,0 +1,292 @@
<?php
declare(strict_types=1);
namespace Blax\Workkit\Services;
use Illuminate\Database\Eloquent\Model;
use Illuminate\Support\Facades\DB;
use ReflectionClass;
use Throwable;
/**
* Read-only database/system stats for an at-a-glance "admin overview" in the
* terminal: per-model row counts, per-table sizes, and the total database
* footprint. Everything here is best-effort and never mutates data.
*
* Row counts come in two flavours:
* - estimate (default, MySQL): information_schema.TABLE_ROWS instant, but
* for InnoDB it's an approximation that drifts from the true count.
* - exact: COUNT(*) per table accurate but a full scan on huge tables.
*
* Table/database SIZE figures are MySQL-only (information_schema). On other
* drivers the size columns come back null and counts fall back to exact.
*/
class StatsService
{
/** Per-process memo so the information_schema scan runs once per connection. */
private static array $tableStatsCache = [];
/**
* Per-table stats for a connection's database, largest first. Memoised per
* connection for the life of the process (these commands take a single
* snapshot, and databaseSize/estimateMap both reuse it).
*
* @return array<int, array{table:string, rows:?int, bytes:?int, engine:?string}>
*/
public static function tableStats(?string $connection = null): array
{
$connection = $connection ?: config('database.default');
if (array_key_exists($connection, self::$tableStatsCache)) {
return self::$tableStatsCache[$connection];
}
if (! self::usesInformationSchema($connection)) {
// No information_schema equivalent we rely on — fall back to exact
// counts per known table, no size data.
return self::$tableStatsCache[$connection] = self::tableStatsFallback($connection);
}
$rows = DB::connection($connection)->select(
'SELECT table_name AS t, table_rows AS r, '
.'(data_length + index_length) AS b, engine AS e '
.'FROM information_schema.tables '
.'WHERE table_schema = DATABASE() AND table_type = ? '
.'ORDER BY b DESC',
['BASE TABLE']
);
return self::$tableStatsCache[$connection] = array_map(fn ($row) => [
'table' => (string) $row->t,
'rows' => $row->r === null ? null : (int) $row->r,
'bytes' => $row->b === null ? null : (int) $row->b,
'engine' => $row->e === null ? null : (string) $row->e,
], $rows);
}
/**
* Total database size + table count for a connection. On MySQL/MariaDB
* `tables` is the true BASE TABLE count and `bytes` the real total; on
* other drivers `bytes` is null and `tables` reflects only the discovered
* model tables (the fallback can't enumerate the full schema).
*
* @return array{bytes:?int, tables:int}
*/
public static function databaseSize(?string $connection = null): array
{
$connection = $connection ?: config('database.default');
$stats = self::tableStats($connection);
$bytes = 0;
$haveBytes = false;
foreach ($stats as $s) {
if ($s['bytes'] !== null) {
$bytes += $s['bytes'];
$haveBytes = true;
}
}
return [
'bytes' => $haveBytes ? $bytes : null,
'tables' => count($stats),
];
}
/**
* Row count per Eloquent model.
*
* Models are taken from config('workkit.stats.models') when set (an
* explicit FQCN list), otherwise auto-discovered by scanning
* config('workkit.stats.models_path') (default: app_path('Models')).
*
* @param string[]|null $models explicit class list (overrides discovery)
* @return array<int, array{model:string, table:string, count:?int, estimated:bool, error:?string}>
*/
public static function modelCounts(?array $models = null, bool $exact = false, ?string $connection = null): array
{
$models = $models ?? self::discoverModels();
$mapConn = $connection ?: config('database.default');
$estimates = (! $exact && self::usesInformationSchema($mapConn))
? self::estimateMap($mapConn)
: [];
$out = [];
foreach ($models as $class) {
$row = ['model' => self::shortName($class), 'table' => '', 'count' => null, 'estimated' => false, 'error' => null];
try {
/** @var Model $instance */
$instance = new $class;
$conn = $connection ?: $instance->getConnectionName() ?: config('database.default');
$table = $instance->getTable();
$row['table'] = $table;
// Only trust the estimate map for the SAME connection it was
// built from, key it with that connection's table prefix, and
// skip a null TABLE_ROWS (empty / never-analysed table) so it
// falls through to an exact count instead of reporting a bogus
// (or cross-database) value.
$prefixed = DB::connection($conn)->getTablePrefix().$table;
if (! $exact && $conn === $mapConn
&& array_key_exists($prefixed, $estimates)
&& $estimates[$prefixed] !== null) {
$row['count'] = $estimates[$prefixed];
$row['estimated'] = true;
} else {
$row['count'] = DB::connection($conn)->table($table)->count();
}
} catch (Throwable $e) {
$row['error'] = $e->getMessage();
}
$out[] = $row;
}
// Largest first; un-countable (error) rows sink to the bottom.
usort($out, fn ($a, $b) => ($b['count'] ?? -1) <=> ($a['count'] ?? -1));
return $out;
}
/**
* Discover instantiable Eloquent model classes under the models path.
*
* @return string[] fully-qualified class names
*/
public static function discoverModels(): array
{
$explicit = config('workkit.stats.models');
if (is_array($explicit) && $explicit !== []) {
return array_values(array_filter($explicit, 'is_string'));
}
$path = config('workkit.stats.models_path');
if (! $path && function_exists('app_path')) {
$path = app_path('Models');
}
if (! $path || ! is_dir($path)) {
return [];
}
$found = [];
$iterator = new \RecursiveIteratorIterator(
new \RecursiveDirectoryIterator($path, \FilesystemIterator::SKIP_DOTS)
);
foreach ($iterator as $file) {
if (! $file->isFile() || $file->getExtension() !== 'php') {
continue;
}
$class = self::classFromFile((string) $file->getPathname());
if ($class === null || ! class_exists($class)) {
continue;
}
try {
$ref = new ReflectionClass($class);
} catch (Throwable $e) {
continue;
}
if ($ref->isAbstract() || ! $ref->isInstantiable() || ! $ref->isSubclassOf(Model::class)) {
continue;
}
$found[$class] = true;
}
$classes = array_keys($found);
sort($classes);
return $classes;
}
/** Human byte size — delegates to BackupService for one implementation. */
public static function humanBytes(int $bytes): string
{
return BackupService::humanBytes($bytes);
}
// ---- internals ---------------------------------------------------------
/**
* MySQL and MariaDB expose the same information_schema.tables columns we
* query (table_rows, data_length, index_length, engine), so both take the
* fast/size-aware path. Laravel 11+ ships 'mariadb' as a distinct driver.
*/
private static function usesInformationSchema(string $connection): bool
{
return in_array(
config("database.connections.{$connection}.driver"),
['mysql', 'mariadb'],
true
);
}
/** table => estimated row count, from information_schema (MySQL). */
private static function estimateMap(string $connection): array
{
$map = [];
foreach (self::tableStats($connection) as $s) {
$map[$s['table']] = $s['rows'];
}
return $map;
}
/**
* Non-MySQL fallback: exact COUNT per discovered model's table (no sizes).
*
* @return array<int, array{table:string, rows:?int, bytes:?int, engine:?string}>
*/
private static function tableStatsFallback(string $connection): array
{
$seen = [];
$out = [];
foreach (self::discoverModels() as $class) {
try {
$instance = new $class;
$table = $instance->getTable();
if (isset($seen[$table])) {
continue;
}
$seen[$table] = true;
$out[] = [
'table' => $table,
'rows' => DB::connection($connection)->table($table)->count(),
'bytes' => null,
'engine' => null,
];
} catch (Throwable $e) {
// skip un-countable model
}
}
usort($out, fn ($a, $b) => ($b['rows'] ?? -1) <=> ($a['rows'] ?? -1));
return $out;
}
/** Parse the FQCN from a PHP file via its namespace + class declaration. */
public static function classFromFile(string $path): ?string
{
$src = @file_get_contents($path);
if ($src === false) {
return null;
}
if (! preg_match('/^\s*namespace\s+([^;]+);/m', $src, $ns)) {
return null;
}
if (! preg_match_all('/^\s*(?:final\s+|abstract\s+)*class\s+(\w+)/m', $src, $all)) {
return null;
}
// Prefer the class whose name matches the file (PSR-4 expectation), so
// a file with a base/abstract class declared before the real model
// doesn't hide it; otherwise fall back to the first declared class.
$expected = pathinfo($path, PATHINFO_FILENAME);
$chosen = in_array($expected, $all[1], true) ? $expected : $all[1][0];
return trim($ns[1]).'\\'.$chosen;
}
private static function shortName(string $class): string
{
$pos = strrpos($class, '\\');
return $pos === false ? $class : substr($class, $pos + 1);
}
}

View File

@ -6,7 +6,10 @@ use Blax\Workkit\Attributes\VariablePaginatable;
use Blax\Workkit\Commands\Database\BackupCommand; use Blax\Workkit\Commands\Database\BackupCommand;
use Blax\Workkit\Commands\Database\PruneBackupsCommand; use Blax\Workkit\Commands\Database\PruneBackupsCommand;
use Blax\Workkit\Commands\Database\RestoreCommand; use Blax\Workkit\Commands\Database\RestoreCommand;
use Blax\Workkit\Commands\Database\StatsCommand;
use Blax\Workkit\Commands\Database\VerifyCommand;
use Blax\Workkit\Commands\PlugNPrayCommand; use Blax\Workkit\Commands\PlugNPrayCommand;
use Blax\Workkit\Commands\Queue\QueueHealthCommand;
use Illuminate\Http\Request; use Illuminate\Http\Request;
use ReflectionException; use ReflectionException;
use ReflectionMethod; use ReflectionMethod;
@ -37,7 +40,10 @@ class WorkkitServiceProvider extends \Illuminate\Support\ServiceProvider
PlugNPrayCommand::class, PlugNPrayCommand::class,
BackupCommand::class, BackupCommand::class,
RestoreCommand::class, RestoreCommand::class,
VerifyCommand::class,
PruneBackupsCommand::class, PruneBackupsCommand::class,
StatsCommand::class,
QueueHealthCommand::class,
]); ]);
// Hosts that want to override path / retention publish the // Hosts that want to override path / retention publish the