diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..4579ba5 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,85 @@ +# Changelog + +All notable changes to `laravel-workkit` are documented here. This project +follows [Keep a Changelog](https://keepachangelog.com/) and keeps every release +backward-compatible with the previous minor (see +`PRINCIPLES/laravel-composer-packages.md` § Backward compatibility). + +## Unreleased + +### Added + +- **`workkit:stats`** (+ `StatsService`) — terminal "admin overview": per-Eloquent-model + row counts (fast `information_schema` estimate on MySQL, `--exact` for `COUNT(*)`), + largest tables by size, total database size, and a compact queue + backup summary. + Models are auto-discovered from `app/Models` or `config('workkit.stats.models')`. + `--json` for machine consumption; degrades gracefully on non-MySQL drivers. +- **`workkit:queue:health`** (+ `QueueHealthService`) — per-queue depth / due / delayed / + reserved counts and oldest-due-job age for the database queue driver, failed-jobs + totals (+ last *N* hours), and STALLED-queue detection (due work past + `max_age_minutes` with nothing reserved → worker likely down). Exits non-zero on any + breach for cron/monitoring; thresholds under `config('workkit.queue.*')` with per-queue + depth overrides. `QueueHealthService::evaluate()` is a pure, unit-testable function. +- New config blocks `workkit.stats.*` and `workkit.queue.*` (read with explicit + code-side defaults, so consumers who published an older config are unaffected). + +### Added — backups + +- **`workkit:db:restore --fresh`** — drops every table + view in the target + before importing, the only reliable fix for a restore failing with + `errno 150` / error `3780` against a dirty or partially-migrated schema. The + wipe is performed via the **same** `mysql` client and database the import + uses (built from the connection's `$cfg`, scoped to `DATABASE()`), so a + `url` DSN / `unix_socket` / read-write split can never make the wipe and the + import target different databases. +- **Pre-restore safety snapshot** — `--fresh` first takes *and verifies* a + snapshot of the current database (named `*.pre-restore.sql.xz.enc`) and + aborts before dropping anything if it can't be written; opt out with + `--no-safety-backup`. On a post-wipe import failure the command prints the + exact `workkit:db:restore --file=` recovery command. +- **`workkit:db:verify`** (+ `workkit:db:backup --verify`) — proves a backup + decrypts with this host's `APP_KEY` and is a complete, non-truncated xz + stream, without touching the database; flags a missing mysqldump completion + marker and compares the on-disk sha256 against the metadata sidecar. + `--all` checks every backup. +- **Metadata sidecar** — `workkit:db:backup` writes a best-effort + `.meta.json` (size, sha256, created_at, cipher params) next to each + backup. Never fails a good backup if it can't be written. +- **`workkit:db:prune-backups --keep-min`** (config + `backup.retention_min_keep`, default 5) — always keeps the N newest backups + regardless of age, so an aggressive `--days` can never leave zero recovery + points. Prune now also removes a backup's sidecar alongside it and sweeps + orphaned sidecars. +- **Restore safeguards** — the source backup is verified before any + destructive step; a dirty (non-empty) target is reported with a `--fresh` + hint; an import that incompatibly fails prints actionable guidance; a + non-trivial backup that imports zero tables is flagged as a possible silent + failure. The confirm line now shows the backup size. +- New public `BackupService` helpers (all additive): `verify()`, + `dropAllTables()`, `countTables()`, `estimateDatabaseBytes()`, + `backupFiles()`, `restoreCandidates()`, `metaPathFor()`, `humanBytes()`, + `runResult()`. + +### Changed + +- **`BackupService::decryptDecompressImport()` now prepends + `SET FOREIGN_KEY_CHECKS=0;`** to the import stream so a hand-rolled dump + lacking mysqldump's own header still imports into an empty database + regardless of table order. Harmless for mysqldump output (which sets/restores + its own session vars), and the whole import is one short-lived session so the + relaxed check never leaks. This does **not** rescue a restore onto a dirty + schema with incompatible column types — use `--fresh` for that. +- File selection in `restore`/`prune` now ignores `*.meta.json` sidecars while + still finding custom `--out`-named backups (it denies the sidecar suffix + rather than allow-listing one extension). The `restore` auto-pick also skips + `*.pre-restore.*` snapshots so a recovery snapshot of a known-bad database is + never restored by accident. + +### Notes + +- Existing `BackupService` method signatures and all config keys are unchanged; + new behavior is additive or confined to the new `--fresh` path. New nested + config keys are read with explicit code-side defaults, so a consumer who + published an older `config/workkit.php` is unaffected. +- The README is now backup-documented and brought up to the standard Blax OSS + package skeleton (badges, feature list, Star History). diff --git a/README.md b/README.md index 25e82e4..bd9d1cc 100644 --- a/README.md +++ b/README.md @@ -2,4 +2,203 @@ # Laravel Workkit +[![PHP Version](https://img.shields.io/badge/php-%3E%3D8.0-blue)](https://php.net) +[![Laravel](https://img.shields.io/badge/laravel-10.x--13.x-orange)](https://laravel.com) +[![License](https://img.shields.io/badge/license-MIT-green)](LICENSE) + A Laravel collection of helpers and utilities to reduce redundant code over multiple projects. + +## Features + +- 💾 **Streaming DB backups** — `mysqldump | xz | openssl` in one pipe, APP_KEY-encrypted, zero DB bytes held in PHP memory (multi-GB safe) +- ♻️ **Safe restores** — `--fresh` wipes a dirty/partial schema before importing, after taking + verifying a recovery snapshot; failed imports never leave you stranded +- 🔎 **Backup verification** — `workkit:db:verify` proves a backup decrypts + is a complete xz stream *without* a database, so "is it corrupt?" gets a real answer +- 🧹 **Retention pruning** — age-based cleanup that always keeps the N newest, so prune can never delete your last recovery point +- 📊 **Stats overview** — `workkit:stats` prints per-model row counts, the largest tables, total DB size + a queue/backup summary +- 🩺 **Queue health** — `workkit:queue:health` reports per-queue depth/age/failed jobs and exits non-zero on breach (cron-friendly) +- 📄 **Variable pagination** — a `#[VariablePaginatable]` attribute + `request()->perPage()` macro for per-route, user-overridable page sizes +- 🧩 **Reusable middleware & traits** — bearer-token auth, force-JSON responses, `HasExpiration`, `HasMeta`, `HasMetaTranslation`, and small service helpers + +## Quick Start + +```bash +composer require blax-software/laravel-workkit +``` + +```bash +php artisan workkit:db:backup # storage/backups/db_mysql_.sql.xz.enc +php artisan workkit:db:verify # is the newest backup intact? +php artisan workkit:db:restore --fresh # wipe + restore the newest backup +``` + +Backups are encrypted with a key derived from your app's `APP_KEY`; a backup is +restorable only by a deployment that knows the same `APP_KEY`. + +## Database Backups + +A streaming, compressed, encrypted MySQL backup/restore suite. The dump, +compression and encryption happen in a single shell pipe, so PHP never holds +the database content in memory regardless of dump size. + +### Commands + +| Command | What it does | +|---|---| +| `workkit:db:backup` | Stream `mysqldump → xz → openssl` into `storage/backups`, write a `.meta.json` sidecar (size, sha256, params). | +| `workkit:db:restore` | Stream `openssl → xz → mysql`. Verifies the source first; `--fresh` wipes the target after a verified safety snapshot. | +| `workkit:db:verify` | Prove a backup decrypts and is a complete xz stream — no database touched. | +| `workkit:db:prune-backups` | Delete backups older than the retention window, always keeping the N newest. | + +### Backup + +```bash +php artisan workkit:db:backup # default connection +php artisan workkit:db:backup --connection=mysql --xz-level=6 +php artisan workkit:db:backup --verify # confirm the backup right after writing +``` + +### Restore + +```bash +# Newest restorable backup into the default connection (prompts unless --force) +php artisan workkit:db:restore + +# A specific file +php artisan workkit:db:restore --file=db_mysql_2026-06-28_09-21-49.sql.xz.enc + +# Wipe the target first — the ONLY thing that fixes a restore failing with +# errno 150 / error 3780 against a dirty or partially-migrated schema. +php artisan workkit:db:restore --fresh +``` + +`--fresh` is deliberately careful, because dropping every table is irreversible: + +1. The **source backup is verified** before anything is touched — a file that + can't be decrypted never triggers a wipe. +2. A **pre-restore safety snapshot** of the current database is taken *and + verified* (skip with `--no-safety-backup`; the restore aborts if the + snapshot can't be written, rather than wiping with no recovery point). +3. Only then are all tables + views dropped — via the **same** `mysql` client + and database the import uses, so the wipe and the import provably hit the + same server (a `url` DSN / `unix_socket` / read-write split can't make them + diverge). +4. If the import fails *after* the wipe, the command prints the exact + `workkit:db:restore --file=` command to recover. + +Interactively, `--fresh` asks you to type the database name to confirm; under +`--force` it proceeds non-interactively for deploy scripts. + +> **Why `--fresh` and not just FK checks?** A leftover table with an +> incompatible column type makes MySQL raise `errno 150` / error `3780` at +> `CREATE TABLE` even with `FOREIGN_KEY_CHECKS=0` (which restores already set). +> The only fix is an empty target. If a restore fails this way, the command +> tells you to re-run with `--fresh`. + +### Verify + +```bash +php artisan workkit:db:verify # newest backup +php artisan workkit:db:verify --file=db_mysql_2026-06-28_09-21-49.sql.xz.enc +php artisan workkit:db:verify --all # every backup in the directory +``` + +Verify proves **integrity** (decrypts with this host's `APP_KEY` + a complete, +non-truncated xz stream), flags a missing mysqldump completion marker +(source-truncation), and compares the on-disk sha256 against the `.meta.json` +sidecar. It does **not** prove restorability — a valid backup of the wrong or +empty database still verifies "OK". + +### Prune + +```bash +php artisan workkit:db:prune-backups --dry-run +php artisan workkit:db:prune-backups --days=30 --keep-min=5 +``` + +The `--keep-min` floor (default 5) is always kept regardless of age, so an +aggressive `--days` can never leave you with zero backups. Schedule it: + +```php +Schedule::command('workkit:db:prune-backups')->daily(); +``` + +### Configuration + +```bash +php artisan vendor:publish --tag=workkit-config +``` + +| Key | Env | Default | Purpose | +|---|---|---|---| +| `backup.path` | `WORKKIT_BACKUP_PATH` | `storage/backups` | Where backups live | +| `backup.retention_days` | `WORKKIT_BACKUP_RETENTION_DAYS` | `30` | Age cutoff for prune | +| `backup.retention_min_keep` | `WORKKIT_BACKUP_RETENTION_MIN_KEEP` | `5` | Newest backups always kept | +| `backup.xz_level` | `WORKKIT_BACKUP_XZ_LEVEL` | `3` | xz compression level (0–9) | + +### Requirements + +The host needs `mysqldump`, `mysql`, `xz`, `openssl` and `bash` on `PATH` — +standard on any reasonable Linux server. A non-empty `APP_KEY` is required +(backups are unrecoverable without the key that produced them). + +### Restoring without the package + +The output is plain `openssl enc -salt` format, so any host with the same +`APP_KEY` can restore it directly: + +```bash +openssl enc -d -aes-256-cbc -pbkdf2 -iter 600000 -pass env:WK_KEY \ + -in db_mysql_2026-06-28_09-21-49.sql.xz.enc | xz -d | mysql +# WK_KEY = your APP_KEY with the "base64:" prefix stripped +``` + +## Ops & Observability + +### Stats overview + +```bash +php artisan workkit:stats # models + largest tables + DB size + queue/backup summary +php artisan workkit:stats --exact # true COUNT(*) per table (heavier) instead of the estimate +php artisan workkit:stats --models --json # just per-model counts, machine-readable +php artisan workkit:stats --tables --limit=30 +``` + +Model row counts are **estimated** from `information_schema` on MySQL (instant); +pass `--exact` for a real `COUNT(*)`. Models are auto-discovered from +`app/Models` — override with `config('workkit.stats.models')` (an explicit FQCN +list) or `models_path`. Table/DB **sizes** are MySQL-only and degrade to "n/a" +elsewhere. + +### Queue health + +```bash +php artisan workkit:queue:health # human table; exits 0 / non-zero +php artisan workkit:queue:health --json || notify-ops # cron / monitoring probe +php artisan workkit:queue:health --max-age=10 --max-depth=500 --max-failed=50 +``` + +For the `database` queue driver it reports per-queue **pending / due / delayed / +reserved** counts and the **oldest due job's age**, plus failed jobs (total + +last *N* hours). It flags a queue **STALLED** — due work older than +`max_age_minutes` with nothing reserved (the worker is probably down) — and +**exits non-zero on any breach** so it slots straight into cron or a health +probe. Thresholds live under `config('workkit.queue.*')`, with per-queue depth +overrides for intentionally-slow queues. Schedule it: + +```php +Schedule::command('workkit:queue:health')->everyFiveMinutes(); +``` + +## Changelog + +See [CHANGELOG.md](CHANGELOG.md). + +## Star History + + + + + + Star History Chart + + diff --git a/config/workkit.php b/config/workkit.php index 72a74bf..9820771 100644 --- a/config/workkit.php +++ b/config/workkit.php @@ -6,18 +6,63 @@ return [ | Backup Settings |-------------------------------------------------------------------------- | - | Used by workkit:db:backup, workkit:db:restore and + | Used by workkit:db:backup, workkit:db:restore, workkit:db:verify and | workkit:db:prune-backups. The default `path` is storage_path('backups') | so backups live alongside the rest of the app's storage. `retention_days` | is the threshold workkit:db:prune-backups uses by default — anything - | older than that gets deleted on the next prune run. + | older than that gets deleted on the next prune run, EXCEPT the + | `retention_min_keep` newest, which are kept regardless of age so prune + | can never leave you with zero recovery points. + | + | NOTE: every command reads these nested keys with an explicit code-side + | default (e.g. config('workkit.backup.retention_min_keep', 5)) because + | mergeConfigFrom does not deep-merge nested arrays — a consumer who + | published an older config without a new key still gets a sane value. | */ 'backup' => [ 'path' => env('WORKKIT_BACKUP_PATH'), // null → storage_path('backups') 'retention_days' => (int) env('WORKKIT_BACKUP_RETENTION_DAYS', 30), + // Always keep at least this many newest backups regardless of age. + // 0 disables the floor (prune may then leave zero backups). + 'retention_min_keep' => (int) env('WORKKIT_BACKUP_RETENTION_MIN_KEEP', 5), // xz compression level. Lower = faster + larger output. 3 is a // good default for SQL dumps (~10× ratio at ~3× the speed of -9). 'xz_level' => (int) env('WORKKIT_BACKUP_XZ_LEVEL', 3), ], + + /* + |-------------------------------------------------------------------------- + | Stats overview (workkit:stats) + |-------------------------------------------------------------------------- + | + | `models` is an explicit list of Eloquent model FQCNs to count. When null + | the command auto-discovers models by scanning `models_path` (default: + | app_path('Models')). Set an explicit list to avoid scanning, or to count + | models that live outside app/Models. + | + */ + 'stats' => [ + 'models' => null, // e.g. [\App\Models\User::class, \App\Models\Order::class] + 'models_path' => null, // null → app_path('Models') + ], + + /* + |-------------------------------------------------------------------------- + | Queue health thresholds (workkit:queue:health) + |-------------------------------------------------------------------------- + | + | The command exits non-zero when any of these is breached. `max_depth` + | applies per queue; `max_depth_per_queue` overrides it for named queues + | (e.g. an intentionally-slow, rate-limited queue allowed a deeper backlog). + | A queue is flagged STALLED when its oldest *due* job is older than + | `max_age_minutes` while nothing is reserved (worker likely down). + | + */ + 'queue' => [ + 'max_depth' => (int) env('WORKKIT_QUEUE_MAX_DEPTH', 250), + 'max_depth_per_queue' => [], // e.g. ['geoip' => 5000] + 'max_age_minutes' => (int) env('WORKKIT_QUEUE_MAX_AGE_MINUTES', 15), + 'max_failed' => (int) env('WORKKIT_QUEUE_MAX_FAILED', 25), + ], ]; diff --git a/src/Commands/Database/BackupCommand.php b/src/Commands/Database/BackupCommand.php index 3aa77ce..4b65718 100644 --- a/src/Commands/Database/BackupCommand.php +++ b/src/Commands/Database/BackupCommand.php @@ -14,6 +14,10 @@ use RuntimeException; * is one shell pipe (mysqldump | xz | openssl); PHP holds zero bytes * of database content in memory regardless of dump size. * + * A best-effort .meta.json sidecar (size, sha256, created_at, + * cipher params) is written alongside each backup so its integrity can + * later be confirmed and a suspiciously-small dump spotted at a glance. + * * Output filename: * storage/backups/db__.sql.xz.enc */ @@ -22,7 +26,8 @@ class BackupCommand extends Command protected $signature = 'workkit:db:backup {--connection= : DB connection to back up (defaults to config(database.default))} {--out= : Custom output path (overrides storage/backups default)} - {--xz-level= : xz compression level 0–9 (default: 3 — fast, ~10× ratio for SQL)}'; + {--xz-level= : xz compression level 0–9 (default: 3 — fast, ~10× ratio for SQL)} + {--verify : After writing, verify the backup decrypts + is a valid xz stream}'; protected $description = 'Create a streamed, compressed + APP_KEY-encrypted backup of the configured MySQL database.'; @@ -33,17 +38,19 @@ class BackupCommand extends Command if (! $cfg) { $this->error("Unknown database connection: {$connection}"); + return self::FAILURE; } if (($cfg['driver'] ?? null) !== 'mysql') { $this->error("workkit:db:backup currently supports only MySQL connections (got: {$cfg['driver']})."); + return self::FAILURE; } $stamp = date('Y-m-d_H-i-s'); $base = BackupService::backupDirectory(); $outPath = $this->option('out') - ?: "{$base}/db_{$connection}_{$stamp}.sql.xz.enc"; + ?: "{$base}/db_{$connection}_{$stamp}".BackupService::BACKUP_EXT; $xzLevel = (int) ($this->option('xz-level') ?? config('workkit.backup.xz_level', 3)); @@ -58,28 +65,33 @@ class BackupCommand extends Command BackupService::dumpCompressEncrypt($cfg, $outPath, $xzLevel); } catch (RuntimeException $e) { $this->error($e->getMessage()); + return self::FAILURE; } $elapsed = microtime(true) - $startedAt; - $size = filesize($outPath); + $size = (int) filesize($outPath); $this->info(sprintf( 'Backup complete in %.1fs: %s (%s)', $elapsed, $outPath, - self::humanBytes((int) $size), + BackupService::humanBytes($size), )); + + if ($this->option('verify')) { + $r = BackupService::verify($outPath); + if (! $r['integrity_ok']) { + $this->error('Verification FAILED: '.$r['message']); + + return self::FAILURE; + } + if (! $r['looks_complete']) { + $this->warn('Verification: '.$r['message']); + } else { + $this->info('Verified: decrypts + valid xz + completion marker present.'); + } + } + return self::SUCCESS; } - - private static function humanBytes(int $bytes): string - { - $units = ['B', 'KB', 'MB', 'GB', 'TB']; - $i = 0; - while ($bytes >= 1024 && $i < count($units) - 1) { - $bytes /= 1024; - $i++; - } - return sprintf('%.2f %s', $bytes, $units[$i]); - } } diff --git a/src/Commands/Database/PruneBackupsCommand.php b/src/Commands/Database/PruneBackupsCommand.php index 7c1ed15..f151f40 100644 --- a/src/Commands/Database/PruneBackupsCommand.php +++ b/src/Commands/Database/PruneBackupsCommand.php @@ -8,9 +8,15 @@ use Blax\Workkit\Services\BackupService; use Illuminate\Console\Command; /** - * Drop backup files older than the retention window. Defaults to 30 - * days; the host can override via --days or by setting - * `workkit.backup.retention_days` in the published config. + * Drop backup files older than the retention window, while ALWAYS keeping + * the N newest regardless of age (retention_min_keep, default 5) so an + * aggressive --days can never leave you with zero recovery points — the + * floor that turns "I deleted my only good backup" into a non-event. + * + * Each backup's .meta.json sidecar is removed alongside it, and orphaned + * sidecars (whose backup is already gone) are swept up too. Pre-restore + * safety snapshots are pruned by the same rules (they're emergency + * artifacts, not long-term backups) but still count toward the keep floor. * * Designed to be wired into the scheduler so storage doesn't fill up * with stale dumps, e.g. Schedule::command('workkit:db:prune-backups')->daily(); @@ -19,34 +25,63 @@ class PruneBackupsCommand extends Command { protected $signature = 'workkit:db:prune-backups {--days= : Retention window in days (default: workkit.backup.retention_days, falling back to 30)} + {--keep-min= : Always keep at least this many newest backups regardless of age (default: workkit.backup.retention_min_keep, falling back to 5; 0 disables the floor)} {--dry-run : Show what would be deleted without removing anything}'; - protected $description = 'Delete backup files older than the retention window.'; + protected $description = 'Delete backup files older than the retention window, always keeping the N newest.'; public function handle(): int { - $days = (int) ($this->option('days') ?: config('workkit.backup.retention_days', 30)); + // Distinguish "not passed" (null) from an explicit --days=0, which must + // error via the < 1 guard rather than being silently swallowed by ?:. + $daysOpt = $this->option('days'); + $days = (int) ($daysOpt !== null ? $daysOpt : config('workkit.backup.retention_days', 30)); if ($days < 1) { $this->error('--days must be >= 1'); + return self::FAILURE; } - $base = BackupService::backupDirectory(); - $files = glob($base . '/*') ?: []; + // Read with an explicit code-side default: a consumer who published an + // OLD config (a `backup` array without this key) would otherwise get + // null here, because mergeConfigFrom does not deep-merge nested keys. + $keepMinOpt = $this->option('keep-min'); + $keepMin = (int) ($keepMinOpt !== null ? $keepMinOpt : config('workkit.backup.retention_min_keep', 5)); + if ($keepMin < 0) { + $keepMin = 0; + } + if ($keepMin === 0) { + $this->warn('keep-min is 0 — prune may delete every aged backup, leaving zero recovery points.'); + } + $dry = (bool) $this->option('dry-run'); + + // Newest-first, sidecars excluded. Same ordering the restore resolver + // uses, so the always-kept set and the restore candidate agree. + $files = BackupService::backupFiles(); $cutoff = time() - $days * 86400; + + $protected = array_slice($files, 0, $keepMin); + $protectedSet = array_flip($protected); + $removed = 0; $kept = 0; foreach ($files as $f) { - if (! is_file($f)) { + if (isset($protectedSet[$f])) { + $kept++; + continue; } if (filemtime($f) < $cutoff) { - if ($this->option('dry-run')) { + $meta = BackupService::metaPathFor($f); + if ($dry) { $this->line("would remove: {$f}"); } else { @unlink($f); + if (is_file($meta)) { + @unlink($meta); + } $this->line("removed: {$f}"); } $removed++; @@ -55,12 +90,32 @@ class PruneBackupsCommand extends Command } } + // Sweep orphaned sidecars (backup already gone, e.g. removed by hand). + $base = BackupService::backupDirectory(); + $orphans = 0; + foreach (glob($base.'/*'.BackupService::META_EXT) ?: [] as $meta) { + $backup = substr($meta, 0, -strlen(BackupService::META_EXT)); + if (! is_file($backup)) { + if ($dry) { + $this->line("would remove orphan sidecar: {$meta}"); + } else { + @unlink($meta); + $this->line("removed orphan sidecar: {$meta}"); + } + $orphans++; + } + } + + $verb = $dry ? 'would-remove' : 'removed'; $this->info(sprintf( - 'Backups: %d kept, %d %s (cutoff: %d days).', + 'Backups: %d kept, %d %s, %d orphan sidecar(s) %s (cutoff: %d days, keep-min: %d).', $kept, $removed, - $this->option('dry-run') ? 'would-remove' : 'removed', + $verb, + $orphans, + $verb, $days, + $keepMin, )); return self::SUCCESS; diff --git a/src/Commands/Database/RestoreCommand.php b/src/Commands/Database/RestoreCommand.php index 2fe369d..2f0db8d 100644 --- a/src/Commands/Database/RestoreCommand.php +++ b/src/Commands/Database/RestoreCommand.php @@ -7,6 +7,7 @@ namespace Blax\Workkit\Commands\Database; use Blax\Workkit\Services\BackupService; use Illuminate\Console\Command; use RuntimeException; +use Throwable; /** * Restore a backup produced by workkit:db:backup. Streams the file @@ -14,18 +15,29 @@ use RuntimeException; * matching the backup pipeline exactly. PHP allocates nothing for the * payload, so even multi-GB backups restore without bumping memory_limit. * - * Without --file, picks the newest backup in storage/backups by mtime. - * Refuses to run unless --force is passed: a restore overwrites whatever - * is currently in the target database. + * Without --file, picks the newest *restorable* backup in storage/backups + * by mtime (pre-restore safety snapshots are excluded from auto-pick). + * Refuses to run unless --force is passed or the operator confirms: a + * restore overwrites whatever is currently in the target database. + * + * --fresh drops every table + view in the target first, which is the only + * thing that fixes a restore failing with errno 150 / error 3780 against a + * dirty or partially-migrated schema (incompatible leftover column types). + * Before it drops anything it (a) verifies the source backup decrypts and + * is a valid xz stream and (b) takes + verifies a pre-restore safety + * snapshot, so a failed import never leaves you with an empty database and + * no recovery point — the exact trap that loses data. */ class RestoreCommand extends Command { protected $signature = 'workkit:db:restore {--connection= : DB connection to restore into (defaults to config(database.default))} - {--file= : Specific backup filename inside the backups directory (default: newest by mtime)} - {--force : Skip the confirmation prompt}'; + {--file= : Specific backup filename inside the backups directory (default: newest restorable file by mtime)} + {--fresh : Drop ALL tables and views in the target before importing. Fixes restore-onto-dirty-schema failures (errno 150 / error 3780). Takes a verified pre-restore safety snapshot first.} + {--no-safety-backup : With --fresh, skip the automatic pre-restore safety snapshot. DANGEROUS: no recovery point if the import fails after the wipe.} + {--force : Skip confirmation prompts (non-interactive).}'; - protected $description = 'Restore a streaming, APP_KEY-encrypted database backup. Defaults to the newest file in storage/backups.'; + protected $description = 'Restore a streaming, APP_KEY-encrypted database backup. Defaults to the newest restorable file in storage/backups.'; public function handle(): int { @@ -34,54 +46,167 @@ class RestoreCommand extends Command if (! $cfg) { $this->error("Unknown database connection: {$connection}"); + return self::FAILURE; } if (($cfg['driver'] ?? null) !== 'mysql') { $this->error("workkit:db:restore currently supports only MySQL connections (got: {$cfg['driver']})."); + return self::FAILURE; } $file = $this->resolveFile(); if (! $file) { $this->error('No backup found.'); + return self::FAILURE; } if (! file_exists($file)) { $this->error("Backup file not found: {$file}"); + return self::FAILURE; } + $fresh = (bool) $this->option('fresh'); + + // 1) Verify the SOURCE before doing anything destructive. Never drop + // tables on the basis of a file that can't even be decrypted. + $v = BackupService::verify($file); + if (! $v['integrity_ok']) { + $this->error('Backup failed verification — refusing to restore.'); + $this->line($v['message']); + + return self::FAILURE; + } + if (! $v['looks_complete']) { + $this->warn('Note: '.$v['message']); + // A source with no completion marker may be truncated-at-source. + // Importing it is recoverable when a safety snapshot is taken, but + // --fresh --no-safety-backup would wipe the target and repopulate + // it from a partial dump with no way back — refuse that outright. + if ($fresh && $this->option('no-safety-backup')) { + $this->error('Refusing: --fresh --no-safety-backup on a source that looks incomplete (no mysqldump completion marker).'); + $this->line('This would wipe the database and import a possibly-truncated dump with no recovery point.'); + $this->line('Run workkit:db:verify on the backup, or drop --no-safety-backup so a recovery snapshot is taken first.'); + + return self::FAILURE; + } + } + if ($v['checksum_matches'] === false) { + $this->warn($v['message']); + } + $this->warn(sprintf( - 'About to restore `%s`@%s from: %s', + 'About to restore `%s`@%s from: %s (%s)', $cfg['database'], - $cfg['host'], + $cfg['host'] ?? 'localhost', $file, + BackupService::humanBytes((int) $v['bytes']), )); $this->warn('This will OVERWRITE any data that conflicts with the dump.'); - if (! $this->option('force') && ! $this->confirm('Proceed?', false)) { - $this->info('Aborted.'); - return self::SUCCESS; + // 2) Dirty-target detection — purely informational; never blocks or + // prompts under --force (existing automation restores over data). + $tableCount = $this->safeCountTables($cfg); + if ($tableCount === null) { + $this->line('Target table count: unknown (could not query the database).'); + } elseif ($tableCount > 0 && ! $fresh) { + $this->warn(sprintf( + 'Target is not empty (%d tables/views). Restoring onto an existing schema can fail with ' + .'errno 150 / error 3780 when column types differ — re-run with --fresh if that happens.', + $tableCount, + )); + } + if ($fresh) { + $this->warn(sprintf( + '--fresh will DROP ALL %stables/views in `%s`@%s before importing.', + $tableCount === null ? '' : $tableCount.' ', + $cfg['database'], + $cfg['host'] ?? 'localhost', + )); } + // 3) Confirmation. --fresh is a full wipe, so require typed name + // confirmation interactively; --force is the non-interactive gate. + if (! $this->option('force')) { + if ($fresh) { + $typed = (string) $this->ask(sprintf( + "Type the database name '%s' to confirm dropping ALL its tables", + $cfg['database'], + )); + if ($typed !== $cfg['database']) { + $this->info('Aborted (name did not match).'); + + return self::SUCCESS; + } + if (! $v['looks_complete'] && ! $this->confirm('The source dump looks incomplete (no completion marker). Wipe and import it anyway?', false)) { + $this->info('Aborted.'); + + return self::SUCCESS; + } + } elseif (! $this->confirm('Proceed?', false)) { + $this->info('Aborted.'); + + return self::SUCCESS; + } + } + + // 4) --fresh: snapshot → verify snapshot → drop. Capture the snapshot + // path BEFORE dropping so every post-drop error can point at it. + $snapshotPath = null; + if ($fresh) { + if (! $this->option('no-safety-backup')) { + try { + $snapshotPath = $this->createSafetySnapshot($connection, $cfg); + } catch (Throwable $e) { + $this->error('Pre-restore safety snapshot failed — aborting before any data is dropped.'); + $this->line($e->getMessage()); + $this->line('Fix the cause (often disk space or a missing mysqldump), or re-run with ' + .'--no-safety-backup to proceed WITHOUT a recovery point (dangerous).'); + + return self::FAILURE; + } + } else { + $this->warn('Skipping pre-restore safety snapshot (--no-safety-backup). No recovery point if the import fails.'); + } + + try { + BackupService::dropAllTables($cfg); + $this->line('Dropped all tables/views in the target database.'); + } catch (Throwable $e) { + $this->error('Failed to drop existing tables — aborting (no import attempted; your data is intact).'); + $this->line($e->getMessage()); + + return self::FAILURE; + } + } + + // 5) Import. $startedAt = microtime(true); try { BackupService::decryptDecompressImport($file, $cfg); } catch (RuntimeException $e) { - $msg = $e->getMessage(); - // openssl's password mismatch error has a recognisable shape; - // translate it into something an operator can act on. - if (str_contains($msg, 'bad decrypt') || str_contains($msg, 'bad magic')) { - $this->error('Decryption failed — likely an APP_KEY mismatch between this host and the host that produced the backup.'); - $this->line($msg); - } else { - $this->error($msg); - } + $this->reportImportFailure($e, $connection, $cfg, $file, $fresh, $snapshotPath); + return self::FAILURE; } + // 6) Post-import sanity: a non-trivial backup that imported nothing is + // a silent-failure signal (swallowed pipe error / empty stream). + if ((int) $v['bytes'] > 1024) { + $after = $this->safeCountTables($cfg); + if ($after === 0) { + $this->warn('Restore reported success but the database has 0 tables — the import may have applied ' + .'nothing. Verify the backup (workkit:db:verify) and the APP_KEY.'); + } + } + $elapsed = microtime(true) - $startedAt; $this->info(sprintf('Restore complete in %.1fs.', $elapsed)); + if ($snapshotPath) { + $this->line('Pre-restore snapshot kept at: '.$snapshotPath); + } + return self::SUCCESS; } @@ -89,7 +214,8 @@ class RestoreCommand extends Command * Resolve the file argument: * - --file=path/to/X.sql.xz.enc → use as-is if it exists * - --file=basename → look up inside storage/backups - * - omitted → pick newest in storage/backups + * - omitted → newest *restorable* file (pre-restore + * safety snapshots excluded) */ private function resolveFile(): ?string { @@ -100,15 +226,153 @@ class RestoreCommand extends Command if (is_file($opt)) { return $opt; } - $candidate = $base . '/' . ltrim((string) $opt, '/'); + $candidate = $base.'/'.ltrim((string) $opt, '/'); + return is_file($candidate) ? $candidate : null; } - $files = glob($base . '/*'); - if (! $files) { + $candidates = BackupService::restoreCandidates(); + if (! $candidates) { + // Distinguish "directory empty" from "only safety snapshots present" + // so the operator isn't told "No backup found" while a recovery + // snapshot sits right there (it just has to be named explicitly). + if (BackupService::backupFiles()) { + $this->warn('Only pre-restore safety snapshots are present — pass one explicitly with --file=... to restore it.'); + } + + return null; + } + + return $candidates[0]; + } + + /** + * Dump → verify the current database into a distinctly-named pre-restore + * snapshot. Returns its absolute path. Throws if the dump or its + * verification fails, so the caller can abort before dropping anything. + */ + private function createSafetySnapshot(string $connection, array $cfg): string + { + $base = BackupService::backupDirectory(); + $stamp = date('Y-m-d_H-i-s'); + $path = sprintf( + '%s/db_%s_%s%s%s', + $base, + $connection, + $stamp, + BackupService::PRE_RESTORE_MARK, + BackupService::BACKUP_EXT, + ); + + if (file_exists($path)) { + throw new RuntimeException("Safety snapshot path already exists, refusing to overwrite: {$path}"); + } + + // Best-effort disk sanity check — warn, don't block (the estimate is + // rough: ~6× compression is conservative for SQL). + $free = @disk_free_space($base); + $dbBytes = $this->safeEstimateDbBytes($cfg); + if ($free !== false && $dbBytes !== null && $dbBytes > 0) { + $estimate = (int) ($dbBytes / 6); + if ($free < $estimate) { + $this->warn(sprintf( + 'Low disk on backup volume: %s free, snapshot may need ~%s. Proceeding anyway.', + BackupService::humanBytes((int) $free), + BackupService::humanBytes($estimate), + )); + } + } + + $this->line('Creating pre-restore safety snapshot…'); + BackupService::dumpCompressEncrypt($cfg, $path, (int) config('workkit.backup.xz_level', 3)); + + $sv = BackupService::verify($path); + if (! $sv['integrity_ok']) { + // Don't leave an unverifiable snapshot (and its sidecar) behind — + // it would clutter the dir and collide with a same-second retry. + @unlink($path); + @unlink(BackupService::metaPathFor($path)); + throw new RuntimeException('Safety snapshot did not verify: '.$sv['message']); + } + + $this->info(sprintf( + 'Safety snapshot saved + verified: %s (%s)', + $path, + BackupService::humanBytes((int) $sv['bytes']), + )); + + return $path; + } + + /** + * Translate an import failure into operator-actionable guidance, and — + * when --fresh already dropped the schema — make the recovery path + * impossible to miss. + */ + private function reportImportFailure( + RuntimeException $e, + string $connection, + array $cfg, + string $file, + bool $fresh, + ?string $snapshotPath + ): void { + $msg = $e->getMessage(); + + if (str_contains($msg, 'bad decrypt') || str_contains($msg, 'bad magic')) { + $this->error('Decryption failed — likely an APP_KEY mismatch between this host and the host that produced the backup.'); + $this->line($msg); + } elseif (preg_match('/\b3780\b|\b1215\b|errno:?\s*150|incompatible/i', $msg)) { + // Match the error code, not the broad phrase "foreign key", to + // avoid false-positives on unrelated FK row failures. + $this->error('Import failed on a foreign-key / column-type incompatibility (errno 150 / error 3780).'); + if (! $fresh) { + $this->line('The target has a conflicting/leftover schema. Re-run with --fresh to drop existing tables first:'); + $this->line(sprintf( + ' php artisan workkit:db:restore --connection=%s --file=%s --fresh', + $connection, + basename($file), + )); + $this->line('A non-fresh import may have PARTIALLY applied before failing, so the database is now in a mixed state.'); + } else { + $this->line($msg); + } + } else { + $this->error($msg); + } + + if ($fresh) { + $this->newLine(); + if ($snapshotPath) { + $this->error('THE TARGET DATABASE IS NOW EMPTY — --fresh dropped it before the failed import.'); + $this->warn('Recover the pre-restore state with:'); + $this->line(sprintf( + ' php artisan workkit:db:restore --connection=%s --file=%s', + $connection, + $snapshotPath, + )); + } else { + $this->error('THE TARGET DATABASE IS NOW EMPTY and no safety snapshot was taken (--no-safety-backup).'); + $this->warn('Restore from your most recent good backup: workkit:db:restore --connection='.$connection.' --file=…'); + } + } + } + + private function safeCountTables(array $cfg): ?int + { + try { + return BackupService::countTables($cfg); + } catch (Throwable $e) { + return null; + } + } + + private function safeEstimateDbBytes(array $cfg): ?int + { + try { + return BackupService::estimateDatabaseBytes($cfg); + } catch (Throwable $e) { return null; } - usort($files, fn($a, $b) => filemtime($b) <=> filemtime($a)); - return $files[0]; } } diff --git a/src/Commands/Database/StatsCommand.php b/src/Commands/Database/StatsCommand.php new file mode 100644 index 0000000..b0b2de0 --- /dev/null +++ b/src/Commands/Database/StatsCommand.php @@ -0,0 +1,173 @@ +option('connection') ?: config('database.default'); + if (! config("database.connections.{$connection}")) { + $this->error("Unknown database connection: {$connection}"); + + return self::FAILURE; + } + + $exact = (bool) $this->option('exact'); + $limit = max(1, (int) $this->option('limit')); + $onlyModels = (bool) $this->option('models'); + $onlyTables = (bool) $this->option('tables'); + $showAll = ! $onlyModels && ! $onlyTables; + + $data = ['connection' => $connection]; + + if ($onlyModels || $showAll) { + $data['models'] = StatsService::modelCounts(null, $exact, $connection); + } + if ($onlyTables || $showAll) { + $data['tables'] = array_slice(StatsService::tableStats($connection), 0, $limit); + $data['database'] = StatsService::databaseSize($connection); + } + if ($showAll) { + $data['queue'] = $this->safe(fn () => $this->queueSummary()); + $data['backup'] = $this->safe(fn () => $this->backupSummary()); + } + + if ($this->option('json')) { + $this->line((string) json_encode($data, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES)); + + return self::SUCCESS; + } + + $this->render($data, $exact); + + return self::SUCCESS; + } + + private function render(array $data, bool $exact): void + { + if (isset($data['models'])) { + $this->info('Models'.($exact ? ' (exact)' : ' (estimated)')); + $rows = []; + foreach ($data['models'] as $m) { + $rows[] = [ + $m['model'], + $m['table'], + ($m['error'] !== null || $m['count'] === null) ? '—' : number_format((int) $m['count']).($m['estimated'] ? ' ~' : ''), + ]; + } + $rows === [] + ? $this->line(' (no models discovered — set config workkit.stats.models)') + : $this->table(['Model', 'Table', 'Rows'], $rows); + } + + if (isset($data['tables'])) { + $this->info('Largest tables'); + $rows = []; + foreach ($data['tables'] as $t) { + $rows[] = [ + $t['table'], + $t['rows'] === null ? '—' : number_format((int) $t['rows']), + $t['bytes'] === null ? '—' : BackupService::humanBytes((int) $t['bytes']), + $t['engine'] ?? '—', + ]; + } + $rows === [] + ? $this->line(' (no tables found)') + : $this->table(['Table', 'Rows', 'Size', 'Engine'], $rows); + + if (isset($data['database'])) { + $db = $data['database']; + $this->line(sprintf( + 'Database: %d tables, %s total', + $db['tables'], + $db['bytes'] === null ? 'size n/a (non-MySQL)' : BackupService::humanBytes((int) $db['bytes']), + )); + } + } + + if (! empty($data['queue'])) { + $q = $data['queue']; + $this->line(sprintf('Queue: %s pending, %s failed (24h)', $q['pending'] ?? '?', $q['failed_recent'] ?? '?')); + } + if (! empty($data['backup'])) { + $b = $data['backup']; + $this->line(sprintf('Backups: %d on disk, newest %s', $b['count'] ?? 0, $b['newest_age'] ?? 'none')); + } + } + + private function queueSummary(): array + { + $snap = QueueHealthService::inspect(); + $pending = 0; + foreach ($snap['queues'] as $q) { + $pending += (int) $q['pending']; + } + + return ['pending' => $pending, 'failed_recent' => $snap['failed_recent']]; + } + + private function backupSummary(): array + { + $files = BackupService::backupFiles(); + $newest = $files[0] ?? null; + $age = null; + if ($newest !== null) { + $secs = max(0, time() - (int) filemtime($newest)); + $age = $this->humanAge($secs); + } + + return ['count' => count($files), 'newest_age' => $age]; + } + + private function humanAge(int $secs): string + { + if ($secs < 90) { + return $secs.'s ago'; + } + if ($secs < 5400) { + return intdiv($secs, 60).'m ago'; + } + if ($secs < 172800) { + return intdiv($secs, 3600).'h ago'; + } + + return intdiv($secs, 86400).'d ago'; + } + + /** Run a best-effort summary probe; return null if it throws. */ + private function safe(callable $fn): ?array + { + try { + return $fn(); + } catch (Throwable $e) { + return null; + } + } +} diff --git a/src/Commands/Database/VerifyCommand.php b/src/Commands/Database/VerifyCommand.php new file mode 100644 index 0000000..b935f87 --- /dev/null +++ b/src/Commands/Database/VerifyCommand.php @@ -0,0 +1,120 @@ +resolveTargets(); + if ($targets === null) { + return self::FAILURE; + } + if ($targets === []) { + $this->warn('No backups found to verify.'); + + return self::SUCCESS; + } + + $ok = 0; + $bad = 0; + $incomplete = 0; + + foreach ($targets as $file) { + $r = BackupService::verify($file); + $label = basename($file).' ('.BackupService::humanBytes((int) $r['bytes']).')'; + + if (! $r['integrity_ok']) { + $bad++; + $this->error("✗ {$label}"); + $this->line(' '.$r['message']); + + continue; + } + + if ($r['checksum_matches'] === false) { + $bad++; + $this->error("✗ {$label}"); + $this->line(' '.$r['message']); + + continue; + } + + if (! $r['looks_complete']) { + $incomplete++; + $this->warn("⚠ {$label}"); + $this->line(' '.$r['message']); + + continue; + } + + $ok++; + $suffix = $r['checksum_matches'] === true ? ' (checksum matches sidecar)' : ''; + $this->info("✓ {$label}{$suffix}"); + } + + $this->newLine(); + $this->line(sprintf('Verified: %d OK, %d incomplete/uncertain, %d failed.', $ok, $incomplete, $bad)); + + return $bad > 0 ? self::FAILURE : self::SUCCESS; + } + + /** + * @return string[]|null list of files to verify, or null on a hard error + */ + private function resolveTargets(): ?array + { + if ($this->option('all')) { + return BackupService::backupFiles(); + } + + $base = BackupService::backupDirectory(); + $opt = $this->option('file'); + + if ($opt) { + if (is_file($opt)) { + return [$opt]; + } + $candidate = $base.'/'.ltrim((string) $opt, '/'); + if (! is_file($candidate)) { + $this->error("Backup file not found: {$opt}"); + + return null; + } + + return [$candidate]; + } + + $files = BackupService::backupFiles(); + + return $files === [] ? [] : [$files[0]]; + } +} diff --git a/src/Commands/Queue/QueueHealthCommand.php b/src/Commands/Queue/QueueHealthCommand.php new file mode 100644 index 0000000..51a0d5c --- /dev/null +++ b/src/Commands/Queue/QueueHealthCommand.php @@ -0,0 +1,103 @@ +option('connection') ?: config('queue.default'); + $window = max(1, (int) $this->option('window')); + + $snapshot = QueueHealthService::inspect($connection, $window); + + $thresholds = QueueHealthService::thresholds(); + foreach (['max-age' => 'max_age_minutes', 'max-depth' => 'max_depth', 'max-failed' => 'max_failed'] as $opt => $key) { + $val = $this->option($opt); + if ($val !== null) { + $thresholds[$key] = (int) $val; + } + } + + $result = QueueHealthService::evaluate($snapshot, $thresholds); + + if ($this->option('json')) { + $this->line((string) json_encode( + ['snapshot' => $snapshot, 'thresholds' => $thresholds] + $result, + JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES + )); + + return $result['ok'] ? self::SUCCESS : self::FAILURE; + } + + $this->line(sprintf('Queue connection: %s (driver: %s)', $snapshot['connection'], $snapshot['driver'] ?? 'unknown')); + + if ($snapshot['supported']) { + if ($snapshot['queues'] === []) { + $this->line(' (no jobs queued)'); + } else { + $rows = []; + foreach ($snapshot['queues'] as $name => $q) { + $rows[] = [ + $name, + number_format($q['pending']), + number_format($q['due']), + number_format($q['delayed']), + number_format($q['reserved']), + $q['oldest_due_age'] === null ? '—' : intdiv($q['oldest_due_age'], 60).'m', + ]; + } + $this->table(['Queue', 'Pending', 'Due', 'Delayed', 'Reserved', 'Oldest due'], $rows); + } + } else { + $this->warn(sprintf('Per-queue introspection unsupported for the "%s" driver — showing failed jobs only.', $snapshot['driver'] ?? 'unknown')); + } + + $this->line(sprintf( + 'Failed jobs: %s total, %s in last %dh', + $snapshot['failed_total'] ?? 'n/a', + $snapshot['failed_recent'] ?? 'n/a', + $snapshot['failed_window_hours'], + )); + + if ($result['ok']) { + $this->info('Queue health: OK'); + + return self::SUCCESS; + } + + $this->newLine(); + $this->error('Queue health: '.count($result['breaches']).' issue(s)'); + foreach ($result['breaches'] as $b) { + $this->line(' • '.$b['message']); + } + + return self::FAILURE; + } +} diff --git a/src/Services/BackupService.php b/src/Services/BackupService.php index 0960ed2..21644ce 100644 --- a/src/Services/BackupService.php +++ b/src/Services/BackupService.php @@ -28,12 +28,31 @@ use RuntimeException; * Required system binaries: `mysqldump`, `mysql`, `xz`, `openssl`, * `bash` (for `set -o pipefail`). All are standard on every reasonable * Linux server. + * + * IMPORTANT — single source of truth for the target server: every path + * that touches MySQL (dump, import, drop, table/size queries) builds its + * client invocation from the SAME `$cfg` array via {@see mysqlClient()}. + * Never mix in Laravel's PDO connection for one operation and the raw + * `$cfg` CLI for another — a `url` DSN, `unix_socket`, or read/write split + * can make those two resolve to *different databases*, and a restore that + * drops one schema while importing into another is exactly the + * catastrophe these commands exist to prevent. */ class BackupService { public const CIPHER = 'aes-256-cbc'; + public const PBKDF2_ITER = 600000; + /** Canonical extension every backup this package writes carries. */ + public const BACKUP_EXT = '.sql.xz.enc'; + + /** Suffix marking a pre-restore safety snapshot (excluded from auto-restore). */ + public const PRE_RESTORE_MARK = '.pre-restore'; + + /** Sidecar metadata suffix written next to each backup (best-effort). */ + public const META_EXT = '.meta.json'; + /** * mysqldump → xz → openssl enc → $outPath. One pipeline, zero PHP * memory pressure. Pipefail propagates a failure in any stage out @@ -52,26 +71,26 @@ class BackupService $level = max(0, min(9, $xzLevel)); $mysqldump = 'mysqldump ' - . '--single-transaction --quick --skip-lock-tables ' - . '--user=' . escapeshellarg((string) ($cfg['username'] ?? 'root')) . ' ' - . '--host=' . escapeshellarg((string) ($cfg['host'] ?? 'localhost')) . ' ' - . '--port=' . escapeshellarg((string) ($cfg['port'] ?? 3306)) . ' ' - . escapeshellarg((string) $cfg['database']); + .'--single-transaction --quick --skip-lock-tables ' + .'--user='.escapeshellarg((string) ($cfg['username'] ?? 'root')).' ' + .'--host='.escapeshellarg((string) ($cfg['host'] ?? 'localhost')).' ' + .'--port='.escapeshellarg((string) ($cfg['port'] ?? 3306)).' ' + .escapeshellarg((string) $cfg['database']); - $xz = "xz -{$level} -T0"; - $openssl = 'openssl enc -' . self::CIPHER . ' -pbkdf2 -iter ' . self::PBKDF2_ITER . ' -salt -pass env:WK_KEY'; + $xz = "xz -{$level} -T0"; + $openssl = 'openssl enc -'.self::CIPHER.' -pbkdf2 -iter '.self::PBKDF2_ITER.' -salt -pass env:WK_KEY'; // bash -c with pipefail: any stage failing trips the whole // pipeline. Without pipefail, a mysqldump crash with xz still // running would leave the output file 0-byte and the exit // code 0 — silent corruption. - $pipeline = "{$mysqldump} | {$xz} | {$openssl} > " . escapeshellarg($outPath); - $cmd = '/bin/bash -c ' . escapeshellarg('set -o pipefail; ' . $pipeline); + $pipeline = "{$mysqldump} | {$xz} | {$openssl} > ".escapeshellarg($outPath); + $cmd = '/bin/bash -c '.escapeshellarg('set -o pipefail; '.$pipeline); try { self::run($cmd, [ 'MYSQL_PWD' => (string) ($cfg['password'] ?? ''), - 'WK_KEY' => self::passphrase(), + 'WK_KEY' => self::passphrase(), ]); } catch (RuntimeException $e) { // Don't leave a half-written encrypted file lying around — @@ -86,12 +105,30 @@ class BackupService if (! file_exists($outPath) || filesize($outPath) === 0) { throw new RuntimeException("Backup pipeline produced no output at {$outPath}"); } + + // Best-effort metadata sidecar — never fail a good backup over it. + self::writeMeta($outPath, $cfg, $level); } /** * openssl dec → xz -d → mysql. Streams through the same pipeline * in reverse. Does no PHP-side decoding so the file size is bounded * only by disk I/O. + * + * A `SET FOREIGN_KEY_CHECKS=0;` line is prepended to the SQL stream so + * that even a hand-rolled dump lacking mysqldump's own header imports + * regardless of table order (forward FK references resolve once every + * table exists). This is harmless for mysqldump output, which sets and + * restores its own session vars, and the whole import is a single + * short-lived mysql session so the relaxed checks never leak out. + * + * NOTE: this does NOT rescue a restore onto a *dirty* schema whose + * leftover tables have incompatible column types — MySQL still raises + * errno 150 / error 3780 at CREATE TABLE because the referenced table + * already exists with the wrong type. For that case the target must be + * empty first (see RestoreCommand --fresh). The openssl|xz stages stay + * a real pipeline segment so `set -o pipefail` still surfaces a bad key + * or corrupt file as a non-zero exit. */ public static function decryptDecompressImport(string $inPath, array $cfg): void { @@ -100,38 +137,235 @@ class BackupService self::requireBinary('openssl'); self::requireBinary('bash'); - // Sanity check on the file's magic bytes. `openssl enc -salt` - // output always starts with the literal "Salted__" header. - $head = (string) @file_get_contents($inPath, false, null, 0, 8); - if ($head !== 'Salted__') { - throw new RuntimeException( - "File at {$inPath} doesn't look like a streaming openssl backup. " - . 'Legacy backups (made with the pre-streaming Crypt envelope) need ' - . 'a separate restore path; see workkit:db:restore-legacy or restore ' - . 'manually with `Crypt::decryptString` after raising memory_limit.' - ); - } + self::assertLooksEncrypted($inPath); - $openssl = 'openssl enc -d -' . self::CIPHER - . ' -pbkdf2 -iter ' . self::PBKDF2_ITER - . ' -pass env:WK_KEY ' - . '-in ' . escapeshellarg($inPath); + $openssl = self::opensslDecrypt($inPath); + $mysql = self::mysqlClient($cfg); - $mysql = 'mysql ' - . '--user=' . escapeshellarg((string) ($cfg['username'] ?? 'root')) . ' ' - . '--host=' . escapeshellarg((string) ($cfg['host'] ?? 'localhost')) . ' ' - . '--port=' . escapeshellarg((string) ($cfg['port'] ?? 3306)) . ' ' - . escapeshellarg((string) $cfg['database']); - - $pipeline = "{$openssl} | xz -d -T0 | {$mysql}"; - $cmd = '/bin/bash -c ' . escapeshellarg('set -o pipefail; ' . $pipeline); + // The brace group's exit status is that of its last command (the + // openssl|xz pipeline), so under `set -o pipefail` a decrypt/decompress + // failure still propagates out as the head of the outer pipeline. + $payload = "{ printf 'SET FOREIGN_KEY_CHECKS=0;\\n'; {$openssl} | xz -d -T0; }"; + $pipeline = "{$payload} | {$mysql}"; + $cmd = '/bin/bash -c '.escapeshellarg('set -o pipefail; '.$pipeline); self::run($cmd, [ 'MYSQL_PWD' => (string) ($cfg['password'] ?? ''), - 'WK_KEY' => self::passphrase(), + 'WK_KEY' => self::passphrase(), ]); } + /** + * Drop every base table and view in the target database, then leave + * the (empty) database in place. Built entirely from $cfg via the same + * mysql client the import uses, and scoped to `DATABASE()` (the schema + * the CLI connects to) so the wipe provably hits the same server + + * schema the subsequent import writes to. + * + * Implemented as in-SQL prepared statements rather than DROP DATABASE + * so it never needs CREATE privilege and preserves the database's + * charset/collation and any grants attached to it. + */ + public static function dropAllTables(array $cfg): void + { + self::requireBinary('mysql'); + self::requireBinary('bash'); + + // Tables and views dropped in two single statements (GROUP_CONCAT + // builds `a`,`b`,`c`); FK checks off so table order is irrelevant. + $sql = <<<'SQL' +SET FOREIGN_KEY_CHECKS = 0; +SET SESSION group_concat_max_len = 1000000000; +SET @t := (SELECT GROUP_CONCAT(CONCAT('`', table_name, '`')) FROM information_schema.tables WHERE table_schema = DATABASE() AND table_type = 'BASE TABLE'); +SET @v := (SELECT GROUP_CONCAT(CONCAT('`', table_name, '`')) FROM information_schema.tables WHERE table_schema = DATABASE() AND table_type = 'VIEW'); +SET @dt := IF(@t IS NULL, 'SELECT 1', CONCAT('DROP TABLE IF EXISTS ', @t)); +PREPARE s1 FROM @dt; EXECUTE s1; DEALLOCATE PREPARE s1; +SET @dv := IF(@v IS NULL, 'SELECT 1', CONCAT('DROP VIEW IF EXISTS ', @v)); +PREPARE s2 FROM @dv; EXECUTE s2; DEALLOCATE PREPARE s2; +SET FOREIGN_KEY_CHECKS = 1; +SQL; + + $pipeline = 'printf %s '.escapeshellarg($sql).' | '.self::mysqlClient($cfg); + $cmd = '/bin/bash -c '.escapeshellarg('set -o pipefail; '.$pipeline); + + self::run($cmd, ['MYSQL_PWD' => (string) ($cfg['password'] ?? '')]); + } + + /** + * Verify a backup without touching MySQL: decrypt + fully decompress and + * inspect the tail. Returns a structured result rather than throwing so + * callers (workkit:db:verify, the --fresh pre-drop gate) can branch on it. + * + * What it proves: + * - integrity_ok : decrypts with the current APP_KEY AND is a valid, + * non-truncated xz stream (xz -d fails otherwise). + * - looks_complete : the plaintext ends with mysqldump's + * "-- Dump completed" marker — catches a dump that + * was truncated AT SOURCE (mysqldump crashed mid-run) + * yet compressed cleanly, which xz integrity can't. + * - checksum_matches: on-disk sha256 equals the value recorded in the + * .meta.json sidecar (proves the file wasn't altered + * on disk since it was written), or null if no sidecar. + * + * It does NOT prove the dump is restorable against a given schema — a + * valid backup of the wrong/empty database still verifies "OK". + * + * @return array{file:string,bytes:int,integrity_ok:bool,looks_complete:bool,sha256:?string,recorded_sha256:?string,checksum_matches:?bool,message:string} + */ + public static function verify(string $path): array + { + self::requireBinary('openssl'); + self::requireBinary('xz'); + self::requireBinary('bash'); + + $result = [ + 'file' => $path, + 'bytes' => is_file($path) ? (int) filesize($path) : 0, + 'integrity_ok' => false, + 'looks_complete' => false, + 'sha256' => null, + 'recorded_sha256' => null, + 'checksum_matches' => null, + 'message' => '', + ]; + + if (! is_file($path)) { + $result['message'] = "Backup file not found: {$path}"; + + return $result; + } + + $head = (string) @file_get_contents($path, false, null, 0, 8); + if ($head !== 'Salted__') { + $result['message'] = "Not a streaming openssl backup (missing 'Salted__' header). " + .'Legacy Crypt-envelope backups need a different restore path.'; + + return $result; + } + + // Decrypt + decompress in one streaming pass, keeping only the tail + // for the completion-marker check. xz -d fails on truncation/bit-rot. + $pipeline = self::opensslDecrypt($path).' | xz -d -T0 | tail -c 512'; + $cmd = '/bin/bash -c '.escapeshellarg('set -o pipefail; '.$pipeline); + [$exit, $stdout, $stderr] = self::runResult($cmd, ['WK_KEY' => self::passphrase()]); + + if ($exit !== 0) { + if (str_contains($stderr, 'bad decrypt') || str_contains($stderr, 'bad magic')) { + $result['message'] = 'Decryption failed — APP_KEY likely does not match the host that produced this backup.'; + } else { + // openssl can (rarely, ~1/256) yield valid PKCS#7 padding on a + // wrong key and exit 0, leaving xz to choke on garbage — so a + // post-decrypt xz failure is ambiguous between the two causes. + $result['message'] = 'Decrypted but the stream is not valid xz — the backup is corrupt OR ' + .'(rarely) the APP_KEY is wrong and the padding happened to validate. Verify APP_KEY first. ' + .'stderr: '.trim($stderr); + } + + return $result; + } + + $result['integrity_ok'] = true; + $result['looks_complete'] = str_contains($stdout, '-- Dump completed'); + + // sha256 + sidecar comparison (best-effort; streams the file). + $sha = @hash_file('sha256', $path); + $result['sha256'] = $sha === false ? null : $sha; + $meta = self::readMeta($path); + if ($meta !== null && isset($meta['sha256']) && is_string($meta['sha256'])) { + $result['recorded_sha256'] = $meta['sha256']; + if ($result['sha256'] !== null) { + $result['checksum_matches'] = hash_equals($meta['sha256'], $result['sha256']); + } + } + + if ($result['checksum_matches'] === false) { + $result['message'] = 'CHECKSUM MISMATCH — the file changed on disk since it was written (integrity of the xz stream is still OK).'; + } elseif (! $result['looks_complete']) { + $result['message'] = 'Integrity OK (decrypts + valid xz) but no mysqldump completion marker — ' + .'may be truncated at source or a non-mysqldump dump.'; + } else { + $result['message'] = 'OK — decrypts, valid xz, mysqldump completion marker present' + .($result['checksum_matches'] === true ? ', checksum matches sidecar.' : '.'); + } + + return $result; + } + + /** + * Count base tables + views in the target database, via the same mysql + * client the import/drop use. Returns null if the query can't run (no + * connection, database missing) so callers can treat it as "unknown". + */ + public static function countTables(array $cfg): ?int + { + $val = self::scalarQuery( + $cfg, + 'SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = DATABASE()' + ); + + return $val === null ? null : (int) $val; + } + + /** + * Approximate on-disk size (bytes) of the target database from + * information_schema. Null if unavailable. Used only for a best-effort + * pre-snapshot disk-space sanity check. + */ + public static function estimateDatabaseBytes(array $cfg): ?int + { + $val = self::scalarQuery( + $cfg, + 'SELECT IFNULL(SUM(data_length + index_length), 0) FROM information_schema.tables WHERE table_schema = DATABASE()' + ); + + return $val === null ? null : (int) $val; + } + + /** + * Real backup files in the backup directory, newest first. Excludes + * `.meta.json` sidecars but KEEPS custom `--out`-named backups (we only + * deny the sidecar suffix rather than allow-listing one extension, so a + * consumer's nightly.enc / db.bak is still found and pruned). + * + * @return string[] absolute paths, newest (by mtime) first + */ + public static function backupFiles(): array + { + $base = self::backupDirectory(); + $files = glob($base.'/*') ?: []; + $files = array_values(array_filter($files, function ($f) { + return is_file($f) && ! str_ends_with($f, self::META_EXT); + })); + usort($files, fn ($a, $b) => filemtime($b) <=> filemtime($a)); + + return $files; + } + + /** + * Like {@see backupFiles()} but also drops pre-restore safety snapshots, + * so the "newest backup" auto-resolver never silently picks a snapshot of + * a known-bad database taken moments before a wipe. + * + * @return string[] + */ + public static function restoreCandidates(): array + { + return array_values(array_filter( + self::backupFiles(), + fn ($f) => ! str_contains(basename($f), self::PRE_RESTORE_MARK.'.') + )); + } + + /** + * Derive the metadata sidecar path for a backup file. Always + * backup → sidecar (never the reverse), so a backup with no sidecar is + * still handled and a lone sidecar is recognisably an orphan. + */ + public static function metaPathFor(string $backupPath): string + { + return $backupPath.self::META_EXT; + } + /** * Derive the openssl passphrase from APP_KEY. Laravel stores APP_KEY * as "base64:"; we strip the prefix and feed the rest @@ -147,6 +381,7 @@ class BackupService if (str_starts_with($key, 'base64:')) { $key = substr($key, 7); } + return $key; } @@ -190,8 +425,8 @@ class BackupService } throw new RuntimeException(sprintf( "Backup directory not writable: %s\n" - . " owner=%s, current user=%s\n" - . " Fix as root: chown -R %s %s && chmod 0775 %s", + ." owner=%s, current user=%s\n" + .' Fix as root: chown -R %s %s && chmod 0775 %s', $path, $owner, $current, @@ -211,7 +446,7 @@ class BackupService */ public static function requireBinary(string $bin): void { - $found = trim((string) @shell_exec('command -v ' . escapeshellarg($bin))); + $found = trim((string) @shell_exec('command -v '.escapeshellarg($bin))); if ($found === '') { throw new RuntimeException("Required binary `{$bin}` not found on PATH."); } @@ -219,11 +454,34 @@ class BackupService /** * Run a shell command and throw on non-zero exit, capturing stderr - * for the error message. Env vars are passed via proc_open's env - * arg — they're scoped to the child process and not visible in the - * host's `ps` listing. + * for the error message. */ public static function run(string $command, array $env = []): void + { + [$exit, , $stderr] = self::runResult($command, $env); + + if ($exit !== 0) { + throw new RuntimeException(sprintf( + "Command failed (exit %d):\n %s\nstderr:\n%s", + $exit, + self::redactCommand($command), + trim((string) $stderr) ?: '(empty)' + )); + } + } + + /** + * Run a shell command and return [exitCode, stdout, stderr] without + * throwing. Env vars are passed via proc_open's env arg — scoped to the + * child process, never visible in the host's `ps` listing. + * + * stdout is read fully; in this package the only stdout producers are + * tiny (a `tail -c 512`, a scalar query, a row of SHOW output), so this + * stays well within the zero-DB-bytes-in-PHP invariant. stdout and stderr + * are drained concurrently (non-blocking + stream_select) so a child that + * fills both pipe buffers at once can never deadlock against a serial read. + */ + public static function runResult(string $command, array $env = []): array { $descriptors = [ 0 => ['pipe', 'r'], @@ -238,19 +496,161 @@ class BackupService } fclose($pipes[0]); - fclose($pipes[1]); - $stderr = stream_get_contents($pipes[2]); - fclose($pipes[2]); + stream_set_blocking($pipes[1], false); + stream_set_blocking($pipes[2], false); + + $stdout = ''; + $stderr = ''; + $open = [1 => $pipes[1], 2 => $pipes[2]]; + + while ($open !== []) { + $read = array_values($open); + $write = null; + $except = null; + if (@stream_select($read, $write, $except, null) === false) { + break; + } + foreach ($read as $stream) { + $chunk = fread($stream, 8192); + if (($chunk === '' || $chunk === false) && feof($stream)) { + $key = array_search($stream, $open, true); + if ($key !== false) { + fclose($stream); + unset($open[$key]); + } + + continue; + } + if ($chunk === false) { + continue; + } + if ($stream === $pipes[1]) { + $stdout .= $chunk; + } else { + $stderr .= $chunk; + } + } + } + + // Close anything stream_select bailed on, then reap the process. + foreach ($open as $stream) { + fclose($stream); + } $exit = proc_close($proc); - if ($exit !== 0) { - throw new RuntimeException(sprintf( - "Command failed (exit %d):\n %s\nstderr:\n%s", - $exit, - self::redactCommand($command), - trim((string) $stderr) ?: '(empty)' - )); + return [$exit, $stdout, $stderr]; + } + + /** Human-readable byte size, e.g. 13631488 → "13.00 MB". */ + public static function humanBytes(int $bytes): string + { + $units = ['B', 'KB', 'MB', 'GB', 'TB']; + $i = 0; + $value = (float) $bytes; + while ($value >= 1024 && $i < count($units) - 1) { + $value /= 1024; + $i++; } + + return sprintf('%.2f %s', $value, $units[$i]); + } + + // ---- internals --------------------------------------------------------- + + /** + * Build the mysql client invocation from $cfg. Single source of truth + * for the target server across import, drop and queries. + */ + private static function mysqlClient(array $cfg): string + { + return 'mysql ' + .'--user='.escapeshellarg((string) ($cfg['username'] ?? 'root')).' ' + .'--host='.escapeshellarg((string) ($cfg['host'] ?? 'localhost')).' ' + .'--port='.escapeshellarg((string) ($cfg['port'] ?? 3306)).' ' + .escapeshellarg((string) $cfg['database']); + } + + /** The openssl decrypt stage, reading from $inPath. */ + private static function opensslDecrypt(string $inPath): string + { + return 'openssl enc -d -'.self::CIPHER + .' -pbkdf2 -iter '.self::PBKDF2_ITER + .' -pass env:WK_KEY ' + .'-in '.escapeshellarg($inPath); + } + + /** Sanity-check the openssl `-salt` magic header. */ + private static function assertLooksEncrypted(string $inPath): void + { + $head = (string) @file_get_contents($inPath, false, null, 0, 8); + if ($head !== 'Salted__') { + throw new RuntimeException( + "File at {$inPath} doesn't look like a streaming openssl backup. " + .'Legacy backups (made with the pre-streaming Crypt envelope) need ' + .'a separate restore path; see workkit:db:restore-legacy or restore ' + .'manually with `Crypt::decryptString` after raising memory_limit.' + ); + } + } + + /** + * Run a single-row, single-column query against $cfg and return the + * trimmed scalar, or null if the query fails (no connection / missing db). + */ + private static function scalarQuery(array $cfg, string $sql): ?string + { + self::requireBinary('mysql'); + self::requireBinary('bash'); + + $pipeline = self::mysqlClient($cfg).' -N -e '.escapeshellarg($sql); + $cmd = '/bin/bash -c '.escapeshellarg('set -o pipefail; '.$pipeline); + [$exit, $stdout] = self::runResult($cmd, ['MYSQL_PWD' => (string) ($cfg['password'] ?? '')]); + + if ($exit !== 0) { + return null; + } + + return trim($stdout); + } + + /** + * Write a best-effort metadata sidecar next to a freshly-created backup. + * Never throws: a backup is valid with or without its sidecar. + */ + private static function writeMeta(string $backupPath, array $cfg, int $xzLevel): void + { + try { + $sha = @hash_file('sha256', $backupPath); + $meta = [ + 'file' => basename($backupPath), + 'database' => $cfg['database'] ?? null, + 'host' => $cfg['host'] ?? null, + 'created_at' => date('c'), + 'bytes' => (int) @filesize($backupPath), + 'sha256' => $sha === false ? null : $sha, + 'cipher' => self::CIPHER, + 'pbkdf2_iter' => self::PBKDF2_ITER, + 'xz_level' => $xzLevel, + ]; + @file_put_contents( + self::metaPathFor($backupPath), + json_encode($meta, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES) + ); + } catch (\Throwable $e) { + // Sidecar is a nicety, not a requirement. + } + } + + /** Read a backup's metadata sidecar, or null if absent/unreadable. */ + private static function readMeta(string $backupPath): ?array + { + $metaPath = self::metaPathFor($backupPath); + if (! is_file($metaPath)) { + return null; + } + $decoded = json_decode((string) @file_get_contents($metaPath), true); + + return is_array($decoded) ? $decoded : null; } /** diff --git a/src/Services/QueueHealthService.php b/src/Services/QueueHealthService.php new file mode 100644 index 0000000..cd77d72 --- /dev/null +++ b/src/Services/QueueHealthService.php @@ -0,0 +1,211 @@ +, + * failed_total:?int, failed_recent:?int, failed_window_hours:int, now:int + * } + */ + public static function inspect(?string $connection = null, int $failedWindowHours = 24): array + { + $connection = $connection ?: config('queue.default'); + $cfg = config("queue.connections.{$connection}", []); + $driver = $cfg['driver'] ?? null; + $now = time(); + + $snapshot = [ + 'connection' => (string) $connection, + 'driver' => $driver, + 'supported' => false, + 'queues' => [], + 'failed_total' => null, + 'failed_recent' => null, + 'failed_window_hours' => $failedWindowHours, + 'errors' => [], + 'now' => $now, + ]; + + if ($driver === 'database') { + $dbConn = $cfg['connection'] ?? config('database.default'); + $table = $cfg['table'] ?? 'jobs'; + try { + // hasTable() === false is fine (a fresh app with no jobs table); + // only a thrown error means the database is actually unreachable, + // which must surface as a breach so cron doesn't read OK. + if (Schema::connection($dbConn)->hasTable($table)) { + $snapshot['supported'] = true; + $snapshot['queues'] = self::queueStats($dbConn, $table, $now); + } + } catch (Throwable $e) { + $snapshot['errors'][] = 'queue introspection failed: '.$e->getMessage(); + } + } + + // Failed jobs live in the DB regardless of the queue driver. + $failed = self::failedJobStats($failedWindowHours); + $snapshot['failed_total'] = $failed['total']; + $snapshot['failed_recent'] = $failed['recent']; + if ($failed['error'] !== null) { + $snapshot['errors'][] = $failed['error']; + } + + return $snapshot; + } + + /** + * Evaluate a snapshot against thresholds. Pure — no I/O. + * + * @param array $snapshot output of {@see inspect()} + * @param array{max_depth:int, max_depth_per_queue:array, max_age_minutes:int, max_failed:int} $thresholds + * @return array{ok:bool, breaches:array} + */ + public static function evaluate(array $snapshot, array $thresholds): array + { + $maxDepth = (int) ($thresholds['max_depth'] ?? 250); + $perQueue = (array) ($thresholds['max_depth_per_queue'] ?? []); + $maxAgeSec = (int) ($thresholds['max_age_minutes'] ?? 15) * 60; + $maxFailed = (int) ($thresholds['max_failed'] ?? 25); + + $breaches = []; + + foreach (($snapshot['queues'] ?? []) as $name => $q) { + $limit = array_key_exists($name, $perQueue) ? (int) $perQueue[$name] : $maxDepth; + if (($q['pending'] ?? 0) > $limit) { + $breaches[] = [ + 'type' => 'depth', + 'queue' => $name, + 'message' => sprintf("queue '%s' depth %d exceeds %d", $name, $q['pending'], $limit), + ]; + } + + // Stalled: due work older than the age limit, nothing in flight. + $oldest = $q['oldest_due_age'] ?? null; + if (($q['due'] ?? 0) > 0 && (int) ($q['reserved'] ?? 0) === 0 && $oldest !== null && $oldest > $maxAgeSec) { + $breaches[] = [ + 'type' => 'stalled', + 'queue' => $name, + 'message' => sprintf( + "queue '%s' looks STALLED: oldest due job is %dm old with 0 reserved (worker down?)", + $name, + intdiv($oldest, 60), + ), + ]; + } + } + + $recent = $snapshot['failed_recent'] ?? null; + if ($recent !== null && $recent > $maxFailed) { + $breaches[] = [ + 'type' => 'failed', + 'queue' => null, + 'message' => sprintf('%d jobs failed in the last %dh (max %d)', $recent, (int) ($snapshot['failed_window_hours'] ?? 24), $maxFailed), + ]; + } + + // A DB/connection error (not a merely-absent table) must surface as a + // breach so a monitoring probe exits non-zero instead of reading OK. + foreach (($snapshot['errors'] ?? []) as $err) { + $breaches[] = ['type' => 'error', 'queue' => null, 'message' => $err]; + } + + return ['ok' => $breaches === [], 'breaches' => $breaches]; + } + + /** Thresholds from config, each read with an explicit code-side default. */ + public static function thresholds(): array + { + return [ + 'max_depth' => (int) config('workkit.queue.max_depth', 250), + 'max_depth_per_queue' => (array) config('workkit.queue.max_depth_per_queue', []), + 'max_age_minutes' => (int) config('workkit.queue.max_age_minutes', 15), + 'max_failed' => (int) config('workkit.queue.max_failed', 25), + ]; + } + + // ---- internals --------------------------------------------------------- + + /** + * Per-queue conditional aggregates in a single grouped query (works on + * MySQL + SQLite + Postgres; jobs timestamps are unix ints). + * + * @return array + */ + private static function queueStats(string $dbConn, string $table, int $now): array + { + $rows = DB::connection($dbConn)->table($table) + ->selectRaw('queue') + ->selectRaw('COUNT(*) as pending') + ->selectRaw('SUM(CASE WHEN reserved_at IS NOT NULL THEN 1 ELSE 0 END) as reserved') + ->selectRaw('SUM(CASE WHEN reserved_at IS NULL AND available_at <= ? THEN 1 ELSE 0 END) as due', [$now]) + ->selectRaw('SUM(CASE WHEN reserved_at IS NULL AND available_at > ? THEN 1 ELSE 0 END) as delayed', [$now]) + ->selectRaw('MIN(CASE WHEN reserved_at IS NULL AND available_at <= ? THEN available_at END) as oldest_due', [$now]) + ->groupBy('queue') + ->get(); + + $out = []; + foreach ($rows as $r) { + $oldest = $r->oldest_due; + $out[(string) $r->queue] = [ + 'pending' => (int) $r->pending, + 'due' => (int) $r->due, + 'delayed' => (int) $r->delayed, + 'reserved' => (int) $r->reserved, + 'oldest_due_age' => $oldest === null ? null : max(0, $now - (int) $oldest), + ]; + } + + return $out; + } + + /** + * @return array{total:?int, recent:?int, error:?string} + */ + private static function failedJobStats(int $windowHours): array + { + $dbConn = config('queue.failed.database') ?: config('database.default'); + $table = config('queue.failed.table', 'failed_jobs'); + + try { + if (! Schema::connection($dbConn)->hasTable($table)) { + return ['total' => null, 'recent' => null, 'error' => null]; + } + $total = DB::connection($dbConn)->table($table)->count(); + $since = date('Y-m-d H:i:s', time() - $windowHours * 3600); + $recent = DB::connection($dbConn)->table($table)->where('failed_at', '>=', $since)->count(); + + return ['total' => $total, 'recent' => $recent, 'error' => null]; + } catch (Throwable $e) { + return ['total' => null, 'recent' => null, 'error' => 'failed-jobs query failed: '.$e->getMessage()]; + } + } +} diff --git a/src/Services/StatsService.php b/src/Services/StatsService.php new file mode 100644 index 0000000..fc6e190 --- /dev/null +++ b/src/Services/StatsService.php @@ -0,0 +1,292 @@ + + */ + public static function tableStats(?string $connection = null): array + { + $connection = $connection ?: config('database.default'); + if (array_key_exists($connection, self::$tableStatsCache)) { + return self::$tableStatsCache[$connection]; + } + + if (! self::usesInformationSchema($connection)) { + // No information_schema equivalent we rely on — fall back to exact + // counts per known table, no size data. + return self::$tableStatsCache[$connection] = self::tableStatsFallback($connection); + } + + $rows = DB::connection($connection)->select( + 'SELECT table_name AS t, table_rows AS r, ' + .'(data_length + index_length) AS b, engine AS e ' + .'FROM information_schema.tables ' + .'WHERE table_schema = DATABASE() AND table_type = ? ' + .'ORDER BY b DESC', + ['BASE TABLE'] + ); + + return self::$tableStatsCache[$connection] = array_map(fn ($row) => [ + 'table' => (string) $row->t, + 'rows' => $row->r === null ? null : (int) $row->r, + 'bytes' => $row->b === null ? null : (int) $row->b, + 'engine' => $row->e === null ? null : (string) $row->e, + ], $rows); + } + + /** + * Total database size + table count for a connection. On MySQL/MariaDB + * `tables` is the true BASE TABLE count and `bytes` the real total; on + * other drivers `bytes` is null and `tables` reflects only the discovered + * model tables (the fallback can't enumerate the full schema). + * + * @return array{bytes:?int, tables:int} + */ + public static function databaseSize(?string $connection = null): array + { + $connection = $connection ?: config('database.default'); + $stats = self::tableStats($connection); + + $bytes = 0; + $haveBytes = false; + foreach ($stats as $s) { + if ($s['bytes'] !== null) { + $bytes += $s['bytes']; + $haveBytes = true; + } + } + + return [ + 'bytes' => $haveBytes ? $bytes : null, + 'tables' => count($stats), + ]; + } + + /** + * Row count per Eloquent model. + * + * Models are taken from config('workkit.stats.models') when set (an + * explicit FQCN list), otherwise auto-discovered by scanning + * config('workkit.stats.models_path') (default: app_path('Models')). + * + * @param string[]|null $models explicit class list (overrides discovery) + * @return array + */ + public static function modelCounts(?array $models = null, bool $exact = false, ?string $connection = null): array + { + $models = $models ?? self::discoverModels(); + $mapConn = $connection ?: config('database.default'); + $estimates = (! $exact && self::usesInformationSchema($mapConn)) + ? self::estimateMap($mapConn) + : []; + + $out = []; + foreach ($models as $class) { + $row = ['model' => self::shortName($class), 'table' => '', 'count' => null, 'estimated' => false, 'error' => null]; + try { + /** @var Model $instance */ + $instance = new $class; + $conn = $connection ?: $instance->getConnectionName() ?: config('database.default'); + $table = $instance->getTable(); + $row['table'] = $table; + + // Only trust the estimate map for the SAME connection it was + // built from, key it with that connection's table prefix, and + // skip a null TABLE_ROWS (empty / never-analysed table) so it + // falls through to an exact count instead of reporting a bogus + // (or cross-database) value. + $prefixed = DB::connection($conn)->getTablePrefix().$table; + if (! $exact && $conn === $mapConn + && array_key_exists($prefixed, $estimates) + && $estimates[$prefixed] !== null) { + $row['count'] = $estimates[$prefixed]; + $row['estimated'] = true; + } else { + $row['count'] = DB::connection($conn)->table($table)->count(); + } + } catch (Throwable $e) { + $row['error'] = $e->getMessage(); + } + $out[] = $row; + } + + // Largest first; un-countable (error) rows sink to the bottom. + usort($out, fn ($a, $b) => ($b['count'] ?? -1) <=> ($a['count'] ?? -1)); + + return $out; + } + + /** + * Discover instantiable Eloquent model classes under the models path. + * + * @return string[] fully-qualified class names + */ + public static function discoverModels(): array + { + $explicit = config('workkit.stats.models'); + if (is_array($explicit) && $explicit !== []) { + return array_values(array_filter($explicit, 'is_string')); + } + + $path = config('workkit.stats.models_path'); + if (! $path && function_exists('app_path')) { + $path = app_path('Models'); + } + if (! $path || ! is_dir($path)) { + return []; + } + + $found = []; + $iterator = new \RecursiveIteratorIterator( + new \RecursiveDirectoryIterator($path, \FilesystemIterator::SKIP_DOTS) + ); + foreach ($iterator as $file) { + if (! $file->isFile() || $file->getExtension() !== 'php') { + continue; + } + $class = self::classFromFile((string) $file->getPathname()); + if ($class === null || ! class_exists($class)) { + continue; + } + try { + $ref = new ReflectionClass($class); + } catch (Throwable $e) { + continue; + } + if ($ref->isAbstract() || ! $ref->isInstantiable() || ! $ref->isSubclassOf(Model::class)) { + continue; + } + $found[$class] = true; + } + + $classes = array_keys($found); + sort($classes); + + return $classes; + } + + /** Human byte size — delegates to BackupService for one implementation. */ + public static function humanBytes(int $bytes): string + { + return BackupService::humanBytes($bytes); + } + + // ---- internals --------------------------------------------------------- + + /** + * MySQL and MariaDB expose the same information_schema.tables columns we + * query (table_rows, data_length, index_length, engine), so both take the + * fast/size-aware path. Laravel 11+ ships 'mariadb' as a distinct driver. + */ + private static function usesInformationSchema(string $connection): bool + { + return in_array( + config("database.connections.{$connection}.driver"), + ['mysql', 'mariadb'], + true + ); + } + + /** table => estimated row count, from information_schema (MySQL). */ + private static function estimateMap(string $connection): array + { + $map = []; + foreach (self::tableStats($connection) as $s) { + $map[$s['table']] = $s['rows']; + } + + return $map; + } + + /** + * Non-MySQL fallback: exact COUNT per discovered model's table (no sizes). + * + * @return array + */ + private static function tableStatsFallback(string $connection): array + { + $seen = []; + $out = []; + foreach (self::discoverModels() as $class) { + try { + $instance = new $class; + $table = $instance->getTable(); + if (isset($seen[$table])) { + continue; + } + $seen[$table] = true; + $out[] = [ + 'table' => $table, + 'rows' => DB::connection($connection)->table($table)->count(), + 'bytes' => null, + 'engine' => null, + ]; + } catch (Throwable $e) { + // skip un-countable model + } + } + usort($out, fn ($a, $b) => ($b['rows'] ?? -1) <=> ($a['rows'] ?? -1)); + + return $out; + } + + /** Parse the FQCN from a PHP file via its namespace + class declaration. */ + public static function classFromFile(string $path): ?string + { + $src = @file_get_contents($path); + if ($src === false) { + return null; + } + if (! preg_match('/^\s*namespace\s+([^;]+);/m', $src, $ns)) { + return null; + } + if (! preg_match_all('/^\s*(?:final\s+|abstract\s+)*class\s+(\w+)/m', $src, $all)) { + return null; + } + + // Prefer the class whose name matches the file (PSR-4 expectation), so + // a file with a base/abstract class declared before the real model + // doesn't hide it; otherwise fall back to the first declared class. + $expected = pathinfo($path, PATHINFO_FILENAME); + $chosen = in_array($expected, $all[1], true) ? $expected : $all[1][0]; + + return trim($ns[1]).'\\'.$chosen; + } + + private static function shortName(string $class): string + { + $pos = strrpos($class, '\\'); + + return $pos === false ? $class : substr($class, $pos + 1); + } +} diff --git a/src/WorkkitServiceProvider.php b/src/WorkkitServiceProvider.php index 7e2aaf1..4639a31 100644 --- a/src/WorkkitServiceProvider.php +++ b/src/WorkkitServiceProvider.php @@ -6,7 +6,10 @@ use Blax\Workkit\Attributes\VariablePaginatable; use Blax\Workkit\Commands\Database\BackupCommand; use Blax\Workkit\Commands\Database\PruneBackupsCommand; use Blax\Workkit\Commands\Database\RestoreCommand; +use Blax\Workkit\Commands\Database\StatsCommand; +use Blax\Workkit\Commands\Database\VerifyCommand; use Blax\Workkit\Commands\PlugNPrayCommand; +use Blax\Workkit\Commands\Queue\QueueHealthCommand; use Illuminate\Http\Request; use ReflectionException; use ReflectionMethod; @@ -20,7 +23,7 @@ class WorkkitServiceProvider extends \Illuminate\Support\ServiceProvider */ public function register() { - $this->mergeConfigFrom(__DIR__ . '/../config/workkit.php', 'workkit'); + $this->mergeConfigFrom(__DIR__.'/../config/workkit.php', 'workkit'); } /** @@ -37,13 +40,16 @@ class WorkkitServiceProvider extends \Illuminate\Support\ServiceProvider PlugNPrayCommand::class, BackupCommand::class, RestoreCommand::class, + VerifyCommand::class, PruneBackupsCommand::class, + StatsCommand::class, + QueueHealthCommand::class, ]); // Hosts that want to override path / retention publish the // config; otherwise mergeConfigFrom() above provides defaults. $this->publishes([ - __DIR__ . '/../config/workkit.php' => $this->app->configPath('workkit.php'), + __DIR__.'/../config/workkit.php' => $this->app->configPath('workkit.php'), ], 'workkit-config'); } }