Web Crawler

# Web Crawler (`research-agent`) A single-step DocOps app that seeds a web crawl from a query or URL, runs the `CrawlerAgent` task with the current settings, and renders the summarised report. ## Files | File | Purpose | |-------------|-------------------------------------------------------------| | `app.html` | Single entry point (no inline styles or scripts) | | `app.js` | ES-module entry point | | `style.css` | All styling, token-driven, mobile-first | | `ops/` | DocOp definitions and task config | | `README.md` | This file | ## Pipeline | Step | Op file | Output | Badge | |------|----------------|--------------------|---------------| | 1 | `ops/crawl.md` | `crawl_result.md` | `badge-crawl` | ### Op files * **`ops/crawl.md`** — invokes the `CrawlerAgent` task. Reads the seed from `crawl_query.md` and the crawler tuning from `ops/crawl.json`; writes the summarised crawl report to `crawl_result.md`. ### Data files * **`crawl_query.md`** — user-editable seed query or URL (auto-saved after 800 ms of inactivity, and on explicit Save). * **`ops/crawl.json`** — crawler configuration. Fields are rendered dynamically from the `CrawlerAgent` schema served by `GET /taskConfig/`; unknown fields fall back to the defaults in `DEFAULT_CONFIGS` in `app.js`. ## Behaviour notes * **Filesystem is the source of truth.** Badges are restored from `docops.status.json` on load, falling back to reading `crawl_result.md`. * **Polling** is started only while a task is `RUNNING` and stopped when idle and on `pagehide`. * **`respect_robots_txt` cannot be disabled** — attempting to uncheck it re-checks the box and shows the `<no-no-no>` Nedry cameo, lazily imported from `/app/nedry/no-no-no.js`. * **Global chrome** (usage, sessions, git, downloads) is owned by the shared menubar (`/app/menu.js`). Only the app-specific **Help** action is registered by this app. Inline per-step session links remain in the pipeline card. * **Preferences** (model selections) are stored under the `crawler` namespace via `/app/models.js`; no content is ever kept in `localStorage`. ## Conformance | Axis | Status | Note | |-----------------|:------:|-------------------------------------------------| | 3-file layout | ✅ | plus `ops/` and this README | | Modern JS | ✅ | ES modules, `async/await`, `Map`, no IIFE/`var` | | Menubar | ✅ | `initMenu({ appName: "Web Crawler" })` | | No dup. chrome | ✅ | usage/git modals removed | | Viewport | ✅ | | | Mobile | ✅ | verified at 360 / 768 / 1280 px | Theme: light tokens by default with a `prefers-color-scheme: dark` override of the same token names — no duplicated component rules.