Script integration conventions and scheduling

Moved out of README.md (see development.md for running from source and an overview, and the README for installing).

Script Integration Conventions

Every my_scripts/*.py file starts with a small comment header parsed by utils/shared/script_metadata.py - see templates/script_template.py for a copyable starting point covering all of this: - # Name: / # Name (EN): and # Description: / # Description (EN): - display name/description shown in the GUI sidebar (Hebrew/English). - # Concurrency: light (optional, default heavy) - opts a fast, non-GPU script (e.g. disk_scanner.py, file_renamer.py) into a small concurrent pool in utils/shared/worker.py, so it can run immediately instead of queueing behind a long ffmpeg render (shown with a ⚡ prefix in the sidebar). Leave unset for anything GPU/CPU-heavy (video encodes) so it stays serialized. - # Safety: (required - read-only or modifies) - whether the script ever renames/moves/deletes a pre-existing file (creating new output files doesn't count). Shown as a ⚠️ prefix in the sidebar for modifies; nothing shown for read-only. Missing/invalid values are treated as modifies (the conservative default) by get_script_metadata(), but the header-convention test requires every script to state it explicitly. read-only is enforced, not just documented - tests/bot/test_read_only_scripts_do_not_mutate_files.py statically scans (via ast, not regex) every read-only-tagged script for calls to os.remove/os.rename/os.replace/os.rmdir/os.unlink/shutil.move/shutil.rmtree/send2trash. A call to one of those needs a # safe: ... comment justifying it on that exact line (e.g. cleaning up a temp file the script created itself, not the user's real input) or the test fails - forcing either that justification or an honest modifies tag, instead of a script silently drifting out of sync with its own safety label.

Scripts can optionally report structured progress/results back to the GUI (utils/shared/worker.py parses these from stdout; neither is required - a script that prints neither behaves exactly as before): - print(f"PROGRESS:{pct}") - explicit 0-100 progress, for scripts that don't use tqdm (whose bar format is also still recognized). - print("RESULT: " + json.dumps({...})) as the last line - a short JSON summary (e.g. files scanned, errors) shown in the GUI's toast/webhook notification and saved to the job history (Stats tab), instead of a bare success/failure icon.

A test in tests/bot/test_script_metadata_conventions.py validates every script's header against these rules (catches typos like # Concurrency: ligth that would otherwise silently fall back to heavy).

Scheduling (⏰ Scheduler tab)

Tasks can also be created from the web super-editor (⏰ button) - they keep the input folder and the editor's exact flags (folder/extra_args in the task), and the web app itself now checks due tasks every 30 seconds, so scheduled runs fire even when only the browser interface is running.

Schedule any my_scripts/*.py script to run automatically while the GUI is open - daily at a set time, weekly on a chosen day, or every N hours. Stored in data/scheduled_tasks.json (utils/bot/scheduler_store.py) and checked once a minute (utils/bot/scheduler.py), reusing the exact same utils/shared/worker.py path a manual "Run Script" click uses - light/heavy concurrency and job history recording both apply the same way. A run missed while the computer was asleep or off runs once when it wakes, then the task is back on its schedule: a daily or weekly task never catches up more than once (not once per missed day or week), and an every-N-hours task counts the next N hours from that run. This is separate from cogs/scheduler_cog.py, which schedules fixed jobs (cleanup, weekly recap) on the Discord bot side and isn't configurable from the GUI.

Waking the PC for the next task (19.10.2026, off by default - "💤 הערת המחשב למשימה הבאה" in the scheduler screen, state in data/wake_timer.json): while it's on, utils/bot/wake_timer.py keeps ONE Windows Task Scheduler task, ClipForge\WakeForNextTask, with WakeToRun, about two minutes before the nearest enabled task. It only wakes the PC (its action is an empty cmd /c exit 0): the app is already running, resumes, and its scheduler loop runs the task as usual. The scheduler loop re-syncs it every 30 seconds but calls schtasks only when the time changes (a task ran, the schedule changed); turning it off or having no upcoming task deletes it. Requirements: Windows' power-plan setting "Allow wake timers" must be Enable (Power Options -> Sleep -> Allow wake timers; shown in Diagnostics, read via powercfg /query, read-only). The BIOS "RTC/Resume by Alarm" option is NOT needed for Task Scheduler wake timers. powercfg /waketimers needs an elevated prompt, so Diagnostics shows it only when available. Tests never create a real task: tests/conftest.py sets CLIPFORGE_NO_SCHTASKS, and the tests use a fake schtasks.

If a Discord webhook is configured (Global Settings tab), a weekly digest (utils/bot/weekly_digest.py) is sent automatically - a summary of the last 7 days' job history (success/failure counts, per-run RESULT: summaries) - checked on the same once-a-minute loop, tracked via a last-sent timestamp so it only actually sends once every 7 days.

The job history search box (Stats tab / web app, utils/shared/job_history.py's search_jobs()) matches script name, status, and RESULT: content (e.g. a filename that appeared in a run's output) - and now also started_at directly, so typing a date (2026-09-18, or a fragment like 09-18) finds every run from that day without needing to scroll/guess where it is in the list. Plain substring LIKE against the ISO timestamp, not a real date-range picker - deliberately simple, matching the same "good enough for this table's size" reasoning as the rest of the search.

cogs/scheduler_cog.py's daily maintenance also backs up the whole data/ folder (settings, counters, scheduler state - not just clipforge_data.db, which was the only thing backed up before) to a dated zip in data/backups/, and DMs the bot owner if any of the daily maintenance steps (DB backup, data/ backup, cleanup, weekly recap) fails - previously a failure there was only ever visible in the log file, easy to miss silently for weeks.

Backup-restore verification: the existing backup tests only checked that a backup file gets created with roughly the right shape (a .db file exists, a .zip contains certain names) - not that the backup is actually restorable. tests/bot/test_scheduler_db_backup_restore.py opens the backed-up .db file itself (not the source) and reads real rows back out of it, and tests/bot/test_scheduler_data_backup.py extracts files from the backup .zip and compares their bytes against the originals - a backup that "looks right" in file listing but is corrupt or empty inside would now fail these.

Encrypted local backup bundle (utils/shared/encrypted_backup.py): the daily automatic backup above writes a plain, readable zip/DB straight to this machine's own disk - fine as a local safety net, but not something you'd want to copy to an external drive or cloud storage as-is. This is a separate, on-demand, password-protected alternative: a single .ibkp file containing the same content (config, rocket_settings.json, scheduled tasks, and a consistent clipforge_data.db snapshot via SQLite's own backup API, so a file mid-write is never captured half-written) encrypted with a passphrase you choose (PBKDF2-HMAC-SHA256, 480,000 iterations per OWASP's current guidance, plus Fernet/AES) - safe to move off this machine precisely because it's encrypted at rest, unlike the daily zip. The passphrase is never stored anywhere; losing it means losing access to that backup, with no recovery path. Available from the desktop GUI's "🌐 כללי" settings tab ("🔐 צור גיבוי מוצפן" / "♻️ שחזר מגיבוי מוצפן", via a small masked-password dialog) and from the web app's "⚙️ הגדרות" modal (create, list, download, and restore, all under /api/backup/* - create/restore are same-origin-protected, audit-logged, and rate-limited to 5/minute). Restoring backs up any file about to be overwritten to <name>.bak first (same safety net as settings import/reset), and validates every archive entry stays inside the target data/ folder before writing it (rejects a crafted/corrupt archive trying to write outside that folder). Deliberately excludes secrets_vault.py's credentials (Discord token, API keys) - those already live encrypted in Windows Credential Manager, not as a file on disk, so there's nothing to bundle without building a separate, more involved secrets-export flow.

Log rotation: every persistent log (data/logs/*.log, from utils/shared/logging_setup.py) already rotates by size on its own - 5MB per file, 3 backups kept (gui.log.1, gui.log.2, ...). The daily maintenance loop adds a second, time-based layer on top (utils/shared/log_cleanup.py) - a rotated backup file older than 30 days gets deleted even if it never grew large enough to rotate again, so a log stream from something that runs rarely (web_dashboard.py, say) doesn't accumulate stale backups forever. It never touches the live .log file itself (no numeric suffix) - that one may be open by a running process, and deleting an open file on Windows is asking for trouble.

Fixed a real crash: log rotation failure could recurse infinitely and kill an in-progress render - _TeeStream (the stdout/stderr duplication above) forwards every write into the same rotating logger it's attached to. If that logger's own handler ever fails mid-write (e.g. RotatingFileHandler's rollover hits a file Windows still has locked - not rare when several of this app's processes have log handles open at once) the standard library's default failure handling writes an error message straight back to sys.stderr - which is this same _TeeStream, so that write re-entered the logger, hit the same failing handler again, and recursed until Python's stack limit killed whatever was logging at the time with a RecursionError (seen in practice as a whole video_editor.py batch dying mid-run with "Process Exec Error: maximum recursion depth exceeded" in utils/shared/worker.py, unrelated to anything about the render itself). _TeeStream.write() now tracks per-thread whether it's already mid-write and skips the recursive re-entry into the logger if so (still writes to the real underlying stream either way) - a genuine handler failure now surfaces once, the normal way, instead of crashing the process.

Because the GUI (utils/shared/worker.py) and the bot (cogs/scheduler_cog.py/cogs/remote_execution.py) are separate processes that can each launch a heavy GPU render independently, utils/shared/render_lock.py provides a cross-process file lock (data/render.lock) so a scheduled/remote render never runs at the same time as one started from the GUI (or vice versa) and fights over the same GPU encoder.

Proactive notification if the render lock stays stuck - SchedulerCog.check_stuck_render_lock polls render_lock.get_status() every 15 minutes (far more often than the once-a-day maintenance loop, since a stuck lock blocks every render on the machine right now) and DMs the bot owner if a lock has been held longer than RENDER_LOCK_STUCK_WARNING_SECONDS (45 minutes) - well before MAX_LOCK_AGE_SECONDS (4 hours), the point at which render_lock.py itself auto-recovers a lock from a dead process. This is a genuine early warning, not just an echo of the auto-recovery threshold: a still-alive-but-stuck process wouldn't trigger auto-recovery at all, so without this you'd only find out something was wrong by noticing renders weren't happening. Deduplicated by the lock's own acquired_at timestamp (not by date) - the same stuck lock won't re-notify every 15 minutes, but a new lock acquisition that also gets stuck notifies again.

Scheduled canary render - as part of the daily maintenance loop, SchedulerCog._maybe_run_canary_render generates a tiny synthetic clip (ffmpeg's lavfi testsrc, 2 seconds) and runs it through the real video_editor.py --codec h264 pipeline (CPU codec, so it doesn't depend on GPU availability) every night. The point is to catch a broken ffmpeg/dependency/environment issue (an OS update, a missing DLL, a bad PATH change) from an automated check overnight, rather than discovering it only when someone tries a real render. A failure DMs the bot owner with the actual error tail. Deliberately excluded from job_history (record_history=False on _run_script_blocking, a new parameter alongside a shorter timeout for this specific call) - a synthetic 2-second clip has no business skewing real render-time statistics or the hwaccel-savings comparison.

Retry/backoff adopted for Steam and RSS - cogs/steam_monitor.py's check_steam_deals and cogs/rss_monitor.py's fetch_rss_feeds (both unattended background loops - 12 hours and 30 minutes respectively) now retry a transient network failure instead of silently skipping that feed/game until the next cycle. utils/shared/retry.py gained call_with_retry_async alongside the existing call_with_retry - same backoff math (compute_backoff_delay), but awaits an async callable and sleeps via asyncio.sleep instead of blocking time.sleep, so a retry's wait doesn't freeze the bot's entire event loop (every other command) while it waits.