Skip to content

Data videos (dataviz)

Moved out of README.md (see development.md for running from source and an overview, and the README for installing).

Data Videos (my_scripts/video/dataviz_maker.py, engine in src/dataviz/)

"Data is Beautiful" style videos (bar chart race and more) as a series. Each episode is a folder under data/dataviz_episodes/<name>/ with a config.yaml (title, chart type, theme, sources, events, notes, intro/outro, music) and its data files; defaults match the first episode, so a new one only states what differs. src/dataviz is an isolated package (no imports from the rest of ClipForge): sources/ reads CSV/Excel/JSON into one time/entity/value table and spreads yearly or monthly totals over days, plan.py computes every frame without drawing (adaptive pacing so no bar moves more than 0.5% of the axis per frame, slowdown around events, smoothed ranking and axis, event cards, "new record"), charts/ draws one frame from the plan, and render.py renders in parallel processes piped into ffmpeg with the music trimmed and faded. The input folder is an episode folder or a folder of episodes; output goes to workspace/<orientation>/dataviz/: 

uv run python my_scripts/video/dataviz_maker.py --episode my_episode --mode stills
--mode full (default), test (a short video) or stills (single PNG frames for checking the design). Chart types (chart: in the config, or --chart to try another on the same episode): bar_race, line_race, stacked_area_race and scatter_race (cumulative vs. the last 30 days), all drawn from the same plan; themes light and dark. - Quick preview - --mode preview renders timing.preview_seconds (default 5) plus half a second of intro/outro at half resolution (dataviz.config.preview_config: width, height and dpi shrink together, so the layout is identical) into <name>_preview.mp4. - Publishing steps (off by default, publish: in config.yaml or --publish / --no-publish) - after a full render, utils/dataviz_publish.py writes next to the video a thumbnail (<name>_thumb.jpg, with the title drawn on it), an SEO sheet (<name>_metadata_seo.txt: TikTok caption, YouTube title/description/hashtags) and <name>.publish.json with status: pending_approval. The title comes from publish.title or the episode title; publish.seo_ai: true asks Gemini instead (off by default so it never spends quota unasked, any failure falls back to the config). Nothing is uploaded: --decide approved|rejected --package <video or .publish.json> records the decision (a decision is never silently overwritten). Each step fails on its own without failing the render. In the browser, the same packages are listed in the "✅ אישור קליפים" tab under "📊 סרטוני נתונים לפרסום" (/api/dataviz/packages*, web/routes/dataviz.py): thumbnail, SEO sheet, status, and approve/reject buttons that only change the status in the .publish.json. tests/bot/test_gemini_fake_server.py drives story_cutter.py and publish.seo_ai through real HTTP against a local fake Gemini (tests/fake_gemini.py, via GOOGLE_GEMINI_BASE_URL), including a real 429. The web server loads the project .env at startup (secrets_vault.load_project_env) before the startup health check. - Data videos screen (web UI, nav button "📊 סרטוני נתונים" + a home-screen shortcut) - the whole dataviz flow from the browser, in Hebrew: lists the episodes under data/dataviz_episodes (read-only; title, chart, theme, size), choose what to produce (preview / single frames / short test / full), an optional chart-type override (bar_race, line_race, stacked_area_race, scatter_race) and the publishing package (per episode / on / off), then Run. The run goes through the same queue as /api/run (POST /api/dataviz/run -> dataviz_maker.py, closed whitelists for the args), so progress, the log and the run history work as usual. Outputs from workspace/dataviz/outputs/<orientation>/ (before 19.10.2026: workspace/<orientation>/dataviz/, still shown until the migration script moves them) appear below with a video player / stills thumbnails, download and "open folder", and a link to the publishing-package approval section. "➕ New episode from a CSV" creates data/dataviz_episodes/<name>/ (CSV + generated config.yaml) only under a new name - it validates everything in a temp folder first and refuses (409) if the name exists, never overwriting or deleting. CLIPFORGE_WORKSPACE_DIR points the outputs elsewhere (tests / dry runs). - Hide finished episodes - every episode card has a "hide from the list" button and the screen has a "show hidden episodes" checkbox. Hiding only records the name in data/dataviz_hidden.json (written atomically, git-ignored); the episode folder and its files are never touched, and un-hiding removes the name again. A missing/corrupt marker file just means nothing is hidden. - Studio navigation - the Studio (video editor) runs in a focus mode without the nav rail. A ☰ button at the start of the studio toolbar shows/hides the rail without leaving the studio, and switching screen by any other route (Ctrl+K palette, shortcuts) always restores the rail and top bar. - Gemini quota circuit breaker (utils/media/gemini_quota.py) - story_cutter.py stops calling Gemini after 3 consecutive quota failures (no more 2s/4s backoff per clip), falls straight to the local judge, and retries once after a 5-minute cool-down. - Ambient env - tests ignore a leftover CLIPFORGE_WORKSPACE_DIR from a manual run (it used to cause an intermittent failure of the manifest e2e test).

Dataviz additions

Chart types bump_race and heatmap_race; episode settings editable in the browser (stored in data/dataviz_overrides/<episode>.json, config.yaml is never written); approved publish packages can be sent via dry-run / copy to the Rocket Shorts input folder / private YouTube upload (explicit confirmation, off by default); a missing ETA for PROGRESS:-only scripts is estimated from progress + job_history.

Engine reference (waves 39-42)

  • Settings - every editable setting is declared once in src/dataviz/schema.py (path, type, range, group, Hebrew label and help). The browser form, the validation of every value from the browser, the JSON Schema of config.yaml (GET /api/dataviz/config_schema) and the engine guide are all derived from it. Defaults stay in DEFAULTS (src/dataviz/config.py).
  • Chart types - bar_race, line_race, stacked_area_race, scatter_race, bump_race, heatmap_race and map_race (needs map.geojson). Names and one-line descriptions: CHART_INFO in src/dataviz/charts/__init__.py.
  • Modes - full, test, preview, stills and share (a picture and a GIF of the strongest moment).
  • Targets - --target youtube|shorts|tiktok; Shorts and TikTok are vertical 1080x1920 with a portrait layout and their own file name (src/dataviz/targets.py).
  • Extras, all off by default - captions (captions.*, <name>.srt), narration (narration.*, Windows voices), a corner logo (logo.*), series (series.*), fixed colors per entity (colors) and entity aliases / exclusions (entities.*).
  • Command line - --episode, --episodes a,b, --mode, --chart, --target, --captions, --publish / --no-publish, --decide ... --package ..., --workers, --output_dir.

Graph editor upgrades (waves 49+)

Every addition is off by default - an episode that does not turn it on renders exactly as before (a test compares the pixels).

  • Wave 49 - numbers and values. values.mode: rolling shows the sum of the last values.window_days days instead of the running total (values can go down; the pacing follows the change that is shown and the axis follows the current leader). values.per_capita divides every entity by its population (file: a CSV in the episode folder with entity,population or its first two columns, and/or inline populations) times per; entities without a population are left out. numbers.* formats every number on screen - style: plain | compact (1.2M) | words (1.2 מיליון), decimals, prefix / suffix (a word gets a space, a symbol such as % or ₪ sticks to the number) and thousands. ranking.log_scale draws a logarithmic axis in bars (drawn in log10 space so the bar shading is unchanged), lines, scatter (X), heatmap and map colors. labels.rank_arrows adds ▲N / ▼N next to an entity that moved in the ranking compared with labels.rank_window_days days back (bars and lines), and labels.gains adds "+N" next to a bar or line that jumped on the day being animated (at least labels.gains_min of the axis). The settings are in the schema, so they appear in the full editor (group "🔢 Numbers and values"), the guide and the JSON Schema, in Hebrew and English.
  • Wave 50 - the story. Edited in the full editor's "📖 Story" tab (lists) and the "📖 Story" settings group (durations). story.chapters (date, title, subtitle, optional seconds, default story.chapter_seconds) freeze the frame of that day and cover it with a title card; story.pauses (date, text, seconds) freeze it with an explanation box at the top - both are copies of that frame with a hold (fade in / out), so captions and narration shift with them and the SRT gets a line per chapter. story.callouts (date, entity, text, days) point at one entity's bar or line end (other charts: a box in the corner). story.periods (start, end, label, color) are shaded on the bottom timeline of every chart, on the time axis itself in line / area / bump / heatmap charts, and shown as a tag in the others. story.speed (start, end, speed: 2 = twice as fast) divides the frames per day in that range, on top of the step rule. teaser (enabled, seconds, label) opens with a few seconds around the strongest moment, before the intro. end_screen (enabled, seconds 5-20, text, guides) makes the outro at least that long and, in its last seconds, dims the chart, moves the top places to the top-left corner and keeps the right half and the bottom corner free for YouTube's end-screen elements (guides draws them dashed, for checking only). outro.credits (lines) replace the notes at the bottom during the outro and are added to the YouTube description as their own "data sources" block (it survives editing the description in the wizard). The new frontend code lives in a second lazy bundle of the screen, dataviz_pro (js/82-dataviz-story.js, styles/27-dataviz-pro.css), loaded right after dataviz.
  • Wave 51 - the look. A new chart type, pie_race - a donut of each entity's share of the total at this moment (the top_n of the smoothed ranking as slices, everyone else in one labels.others slice, the total in the middle, the current entity pulled out). An image per entity (images.folder / shape: circle | square | original / size): a file named after the entity (characters that are not allowed in file names become -), or <entity>@<anything> - the newest wins; drawn after the bar end (the value label moves after it), at the line end and the bump head, and on pie slices. Images uploaded in the entities tab are new files under data/dataviz_overrides/<episode>/images/ (the episode points there by itself; removal = recycle bin), and the live preview's cache key includes that folder. style.* (bar_height, grid, title_size, year_mark) and three ready looks in the colors tab (src/dataviz/looks.py - minimal, news, bold) that are staged as a draft, with every setting another look changes going back to the file. map.legend draws a color scale in the map's corner and map.labels: none | top | all writes region names at the center of each region's largest part. The colors tab also simulates protanopia / deuteranopia / tritanopia on the live preview (SVG color matrices, Machado 2009) and lists palette pairs that become hard to tell apart (ΔE < 12).
  • Wave 52 - the data. src/dataviz/cleaning.py: similar_names groups entity names that are probably the same entity spelled differently (case, Hebrew final letters, doubled letters, hyphens and quotes, then a 0.86 similarity ratio; numbers are skipped, names already merged by an alias are not suggested again; the one with the bigger total is the suggested target), find_outliers flags a day where an entity got at least 6 times its usual (the median of the 15 days around it) and at least 5% of its whole total, and apply_fixes applies the saved data.fixes (spread: what is above the usual is spread over the days before - the total is kept; smooth: the usual value; set: a given value) to the daily matrix before anything else. data.missing (and a per-source missing) fills days with no rows in a source that reports a value per day: previous or interpolate (linear, only between known days). A source can be a url (a file on the web, or a shared Google Sheets link - exported as CSV of that tab): the engine only reads the downloaded copy (data.cache_dir, src/dataviz/sources.url_cache_path); downloading is ClipForge's (src/utils/dataviz_remote.py) and happens only when the user clicks "Refresh data" / adds the source, or in a scheduled task they created - into data/dataviz_cache/<episode>/ with a state file (hash, downloaded at, last changed at). The data window got two tabs: "🧹 Cleaning" (merge suggestions, suspicious jumps with spread / smooth, saved fixes, missing days) and "🌐 From an address and scheduling" (sources from an address, refresh, a new source, and POST /api/dataviz/schedule, which adds a Scheduler task running dataviz_maker.py --episode <name> --mode <mode> with optional --refresh-data --only-if-changed, which skips an episode whose remote data did not change). Entities of different types (years from one source, names from another) now sort together.
  • Wave 53 - the editor. Undo / redo of the full editor's draft (↶ ↷ next to "Save", Ctrl+Z / Ctrl+Y outside text fields; typing in one field within 1.2 seconds is one step; saving starts a new history; applying a ready look can be undone). Keyboard shortcuts on the screen: Space - animation, arrows - move the moment (Shift - a big step, mirrored in Hebrew), Home / End, Ctrl+S - save, "?" - the list. Markers under the preview slider (GET /api/dataviz/preview_markers, from the same cached plan as the preview, including an unsaved draft) for event cards, chapters, pauses, the start of the race and the outro; clicking one jumps there. A search box at the top of the settings tab filters fields by label, help or path. "▶ Animation" swaps the still frame for a 3-second GIF around the moment (GET /api/dataviz/preview_clip, a fifth of the size, 10 frames per second, cached like the frames). Tags and a folder per episode (POST /api/dataviz/episodes/tags, stored in data/dataviz_tags.json): tags show on the card and #tag in the search filters by them, and episodes with a folder are grouped under its heading.
  • Wave 54 - rendering. video.encoder: cpu | nvenc | auto - NVENC (h264_nvenc, preset mapped from video.preset, -cq = crf + 1) or auto (NVENC when a tiny test encode succeeds, checked once per ffmpeg); a render whose NVENC encode fails is redone with libx264, so a video is always made. The full render goes through a segment cache (workspace/.dataviz_segments, newest 16 files): the plan is split into runs of the same frame kind (teaser, intro, main, outro), and each run is keyed by its frames, the chart context without the texts of the other parts (the main part ignores intro / outro / end_screen / credits), the logo and entity-image files, the size, fps, encoder and a hash of the drawing code - so changing only the intro or outro re-renders only that part, and the rest is joined with -c:v copy (concat_command) with the music faded over the whole length. Every real render is recorded in data/dataviz_render_stats.json (last 50: frames actually drawn, pixels, processes, seconds, encoder), and the estimate uses the median of those runs (process-seconds per megapixel-frame) instead of the fixed guess, showing "based on N runs". Visual regression: tests/dataviz_visual.py renders a small frame of every chart type (plus numbers / log axis / arrows, a dark style and an outro with credits) in DejaVu Sans and tests/dataviz/test_dataviz_visual.py compares them with tests/data/dataviz_golden/*.png (mean difference under 2/255 and under 0.5% strongly changed pixels; a failure saves <name>.actual.png); after a deliberate change: uv run python -m tests.dataviz_visual --write. Found on the way: on a log axis the label room is now measured in log space (it was cut off).
  • Wave 55 - publishing. YouTube chapters: after a full render, <name>.chapters.txt (dataviz/captions.youtube_chapters - 0:00 + every story chapter and event card, merged when under 10 seconds apart, only when there are at least 3; shifted by an intro bumper) goes into the description before the data sources (publish.chapters). Language versions: translations.en (titles, intro/outro, labels, notes, story, publish texts, events by date and entities display names) is merged by dataviz/languages.apply_language and validated; --lang en|both renders <name>_en or both, "words" numbers are written in English; the editor's "🌍 Languages" tab edits it and a language select appears next to Run for episodes that have it. Automatic Shorts: --mode shorts (dataviz/shorts.py) picks the shorts.count most dramatic days (biggest jump relative to the axis, plus a change of leader, kept apart) and renders each as a vertical <name>_shorts_N.mp4 - a 1.5 s intro, shorts.seconds around the moment and a frozen shorts.cta. The chosen thumbnail is uploaded with the video (youtube_uploader.set_thumbnail, failures recorded in the package). Every views refresh is kept in stats_history, and GET /api/dataviz/series_board + the 📈 button show views per series with a line over time and "refresh all".
  • Wave 56 - the graph editor and the studio. They stay two tools with one frame: on a desktop-width window the graph editor opens full screen like the studio (body.dvz-focus, set in 65-ui-shell.js whenever the dataviz tab is shown) with the same kind of top bar - ☰ brings the navigation rail back, the selected episode, "🎬 To the studio" and "✕ Close" - and the studio's Pro Dark colours; the studio's ⋯ menu has "📊 To the graph editor". Every finished video has "🎬 Open in studio": POST /api/dataviz/outputs/studio (utils/dataviz_handoff.py) gives it its own folder, workspace/dataviz/studio_handoff/<orientation>/<name>/<name>.mp4 (a hard link, or a copy when a link is not possible), and the studio opens on it - the studio processes every video in a folder and its subfolders, and the dataviz output folder also holds previews, tests and Shorts. The same folder is reused, so the studio project (logo, targets, edits) survives a re-render; an outdated copy moves to dataviz_studio/_previous/ (outside the video's folder) and is never deleted, and the dataviz output itself is never moved or changed.
  • Wave 59 - one settings form. The basic form (title, subtitle, chart, theme, places shown, publishing title, description and hashtags) became the "⭐ Basics" group of the full editor, with "↩️ Reset all"; the run-only choices fold into "Changes for this run"; and the preview line shows the date and time in the video. Details in web_ui.md ("Studio and graph editor layout").
  • Stabilization round (16.10.2026). Numbers on screen are now rounded half up (dataviz/charts/base.round_half_up: half always goes away from zero, using the number's short decimal form). Until now format_int truncated - after the daily spread 2014 in the first episode was 4,035.99 and showed as 4,035 instead of 4,036 - and the other styles (decimals, compact, words, the pie percentages, the rank ticks) used Python's round-half-to-even. So the first episode no longer reproduces the published video byte for byte: some values are 1 higher, as in the data. A compact number that rounds to 1000 of a unit moves up a unit (999,960 -> 1M, not 1000K), and a value axis with a plain whole-number format only gets whole ticks (BaseChart.whole_ticks; a 0-1 axis showed 0, 0, 1, 1). Also in this round: the thumbnail title is drawn by utils/dataviz_thumbnail.py (matplotlib, so Hebrew reads right to left, and a long title breaks into two lines), ⏸ / ⏩ / ▶ fall back to Segoe UI Symbol, the invite at the end of a Short is a centered end card (hold.kind: invite), the publish description keeps its line breaks, the English version translates the source legends (translations.en.legends) and hashtags and puts the title and legend on the left (ctx.ltr), and js/47-dataviz-engine.js moved to the second bundle. The render estimate (utils/dataviz_preview.learned_rate) shows a useful time again (23:47 was shown for a 6:16 render): a run's fixed overhead (RUN_OVERHEAD_SECONDS, starting the processes and encoding) is taken off before its rate is learned, runs shorter than three times that are not learned from, only the last 5 runs count, and a run slower than twice the fastest of them (the machine was busy) is left out; the estimate adds the overhead and the start-up back. In the English version the outro box's lines are left aligned. Tests: tests/dataviz/test_dataviz_dogfood.py, tests/dataviz/test_web_app_dataviz_dogfood_browser.py.
  • Level sources (18.10.2026). A source can set kind: level (default amount - every row adds, as before): each value is a level (a temperature, a lake level), taken as it is - not added up and not spread. Each line starts at its first value on the first day of its first period and holds it through that period; after that a yearly / monthly value is reached on the last day of its period with a straight line in between (the same shape an evenly spread change gives), a daily source fills missing days with a straight line (missing: previous - the previous value), and the last value holds. Several rows for one period and entity are averaged. A level line enters at its value without the STEP slowdown. kind: level needs a value column.
  • Usability round (19.10.2026). A new-user walkthrough led to: three levels in the full editor like the studio (simple - basic, texts, video, music; regular - without series, branding, captions, map, share and cleanup; expert - all; a group with a change always shows, search finds everything, the choice is kept per browser); one chip on each field naming where its value comes from (draft / changed in the browser / config.yaml / default) and "📜 Final settings" (GET /api/dataviz/episodes/final_config: the merged config the render gets, with the source of each setting, read only); defaults for level sources (when every entity is a level the line_race y-axis runs between labeled ticks around the whole episode's values instead of from 0, one decimal when the level data has fractions unless numbers.decimals is set, the outro box goes where the lines are not, a level whose first value is 0 enters on its first day); unique names for files saved from the browser (O_EXCL, _2, _3...); room for more than three note lines. The screen's JS is one file per topic (editor 44 in the first bundle; look 83, data 84, publish 86 in dataviz_pro), and src/utils/dataviz_*.py moved to src/dataviz/web/ - the old names are shims that alias the same module.
  • The 25 ideas from the technical audit (27.10.2026). A fourth lazy bundle, dataviz_plus (js/85-dataviz-plus.js, styles/28-dataviz-plus.css), wraps the existing screen. Editing on the preview: every frame reports its clickable regions (GET /api/dataviz/preview_frame?meta=1 - hits from BaseChart.hit_regions, plus the ISO date, the data area and the size in points); clicking a title / subtitle / intro / end text edits it in place, clicking an entity opens its fixed color, dragging the outro box sets outro.position and dragging a callout sets its dx / dy (story.callouts). Preview: "📱 Also vertical" (preview_frame?target=shorts), "🛡 Safe zones" for Shorts / TikTok, smooth slider scrubbing with small cached frames, the animation as a short MP4 encoded with NVENC when possible (preview_clip?format=mp4, GIF fallback), and "⇆ Before / after" with a draggable divider. Story: director mode (M or right-click - chapter / pause / callout / render range at the current date, via GET /api/dataviz/timeline), draggable story blocks under the slider, "🎙 Continuous recording" (narration.track / track_start, mixed by dataviz/narration.narrate_track) and range renders (--range START,END -> <name>_range.mp4, dataviz/plan.clip_range). Also: sparklines with suspicious jumps in the entities tab, flags for countries (images.flags, dataviz/countries.py, flags downloaded only on click from flagcdn.com into data/dataviz/dataviz_flags), look buttons with real thumbnails, a font sample, before / after on hover over a field, an episode score in "✅ Before rendering", thumbnails in the version history (preview_frame?version=), "📑 Save as template" (data/dataviz/dataviz_templates, tpl:<name> in the new-episode form), chapter transitions (story.chapter_transition: fade / slide / zoom / wipe), video.encoder: auto and video.priority: low as the defaults (dataviz/render.lower_priority), a GPU chip while rendering (GET /api/dataviz/gpu, nvidia-smi) and YouTube audience retention over the slider (POST /api/dataviz/retention, YouTube Analytics API - a new yt-analytics.readonly scope, only on click).