ndaal Informational Advisory: v0.1.42 — daemon's tokio runtime split into dedicated HTTP and feeder executors so heavy feeder phases (CWE enrichment scan, NVD page parse, Debian 72 MB tracker parse, EPSS gz decompress) no longer starve HTTP request handlers. v0.1.0..v0.1.41 spawned every feeder loop on the same 4-worker `#[tokio::main]` runtime that served HTTP, producing multi-minute timeouts on /dashboards/* and /api/v1/* during each feeder cycle. v0.1.42 introduces `crates/vl-web/src/feeder_runtime.rs` with a `pub fn build_feeder_runtime()` helper, trims the HTTP runtime from `worker_threads = 4` to `worker_threads = 2`, rewires every `tokio::spawn(...)` in the feeder block of `main.rs` to `feeder_handle.spawn(...)` (25 sites), and adds `VL_FEEDER_WORKER_THREADS` for operator tuning. Total daemon capacity 4 → 6 worker threads (2 HTTP + 4 feeder); isolation enforced.

Metadata

Vulnerabilities (0)

No vulnerabilities.

Raw JSON

{
  "$schema": "https://docs.oasis-open.org/csaf/csaf/v2.1/schema/csaf.json",
  "document": {
    "category": "csaf_informational_advisory",
    "csaf_version": "2.1",
    "distribution": {
      "tlp": {
        "label": "CLEAR"
      }
    },
    "lang": "en",
    "notes": [
      {
        "category": "summary",
        "text": "nvulnlookup v0.1.42 splits the daemon's tokio runtime so HTTP serving and the background feeder scheduler run on independent multi-thread executors.  Releases v0.1.0 through v0.1.41 spawned every feeder loop via `tokio::spawn(...)` onto the same 4-worker runtime that served HTTP.  Heavy CPU-bound feeder phases — CWE enrichment scan across ~300 000 stored vulnerabilities, NVD JSON page parses, the 72 MB Debian Security Tracker JSON parse, and EPSS bulk-CSV gz decompression — saturated all 4 workers simultaneously, leaving no async capacity for operator-facing HTTP requests.  Symptom: `/dashboards/*`, `/api/v1/*`, and `/stats` timed out for several minutes per feeder cycle while the daemon log showed feeders making forward progress.\n\nv0.1.42 introduces a new `crates/vl-web/src/feeder_runtime.rs` module that constructs a SEPARATE `tokio::runtime::Runtime` for feeders.  The `#[tokio::main]` runtime that serves HTTP is trimmed from `worker_threads = 4` to `worker_threads = 2` since it no longer competes with feeders.  All 25 `tokio::spawn(...)` sites in the feeder block of `crates/vl-web/src/main.rs` are rewritten to `feeder_handle.spawn(...)`; the 2 remaining `tokio::spawn(...)` calls (HTTP TLS handshake task at line 1255 and the QUIC listener at line 1810) stay on the HTTP runtime as designed.  Worker count on the new feeder runtime defaults to 4 and is overridable via the `VL_FEEDER_WORKER_THREADS` environment variable (range 1..=64; out-of-range values fall back to the default).\n\nNet effect on a v0.1.42 production deployment: HTTP latency stays bounded under feeder load.  An operator hitting `/dashboards/cwe` while the CWE enrichment scan is running gets a sub-second response instead of the multi-minute timeout v0.1.41 would have produced.  Total daemon worker capacity is unchanged (4 workers → 2 HTTP + 4 feeder) — only the isolation changes."
      },
      {
        "category": "description",
        "text": "## Defect — single shared runtime starves HTTP under feeder load\n\nv0.1.0..v0.1.41 entry point in `crates/vl-web/src/main.rs`:\n\n  // BEFORE (v0.1.41, line 132)\n  #[tokio::main(worker_threads = 4)]\n  async fn main() -> anyhow::Result<()> {\n      // ... config, storage, app state ...\n      {\n          // 25 feeder loops, each:\n          tokio::spawn(async move {\n              loop {\n                  feeder.fetch(&stop).await;\n                  tokio::time::sleep(cycle).await;\n              }\n          });\n      }\n      // ... build router, listen ...\n      loop {\n          tokio::select! {\n              result = listener.accept() => {\n                  tokio::spawn(async move {\n                      // TLS handshake + serve HTTP request\n                  });\n              }\n          }\n      }\n  }\n\nProblem: `#[tokio::main(worker_threads = 4)]` declares ONE multi-thread runtime with 4 worker threads.  Every `tokio::spawn(...)` inside `main` lands on this runtime.  Feeders and HTTP request handlers compete for the same 4 worker threads.  Tokio's cooperative scheduler doesn't preempt — when a feeder task runs a synchronous loop (e.g. `for entry in storage.iter_cwe_enrichment()? { ... }` over 296k entries), that worker is blocked until the loop yields naturally.  4 such loops in flight saturate the runtime, and HTTP request handlers wait at the bottom of the run queue for a free worker.\n\nObserved in production v0.1.41 logs:\n\n  02:07:18 INFO  Feeder scheduler: starting CWE enrichment\n  02:07:18 INFO  CWE enrichment: scanning stored vulnerabilities for CWE data\n  02:08:03 INFO  Debian Security Tracker: parsing bytes=72376851\n  02:08:48 INFO  EPSS/KEV: downloading https://epss.cyentia.com/epss_scores-current.csv.gz\n  ... operator's GET /dashboards/cwe at 02:08:30 → 12-second curl timeout ...\n  02:11:01 INFO  Debian Security Tracker: cycle complete packages=3871\n  02:11:03 INFO  Feeder scheduler: Debian Security Tracker fetch completed\n\nThe operator's request lands at 02:08:30, all 4 workers are busy (CWE scan + Debian parse + EPSS gz decompress + at least one more feeder), and the request waits ~3 minutes until Debian completes and a worker frees up.\n\n## Fix landed in v0.1.42\n\n  * NEW MODULE `crates/vl-web/src/feeder_runtime.rs`\n    + `pub fn resolve_feeder_worker_threads(env: Option<&str>) -> usize`\n      - parses VL_FEEDER_WORKER_THREADS\n      - falls back to DEFAULT_FEEDER_WORKER_THREADS = 4 on garbage,\n        unset, 0, or values > 64\n    + `pub fn build_feeder_runtime() -> Result<Runtime>`\n      - reads the env var, calls\n        build_feeder_runtime_with_workers(...)\n    + `pub fn build_feeder_runtime_with_workers(workers) -> Result<Runtime>`\n      - constructs a Builder::new_multi_thread() with the configured\n        worker count, thread_name(\"feeder\"), enable_all()\n\n  * CHANGED `crates/vl-web/src/main.rs`\n    + `#[tokio::main(worker_threads = 4)]` →\n      `#[tokio::main(worker_threads = 2)]`\n      (HTTP runtime trimmed; feeders moved off)\n    + Inserted before the feeder block:\n        let feeder_runtime =\n            vl_web::feeder_runtime::build_feeder_runtime()?;\n        let feeder_handle = feeder_runtime.handle().clone();\n    + 25 `tokio::spawn(async move {` → `feeder_handle.spawn(async move {`\n      (every feeder spawn site, including the 2 macros\n      `spawn_git_feeder!` and `spawn_csaf_feeder!`)\n    + 2 `tokio::spawn(...)` sites left untouched:\n      - line 1255: per-connection HTTP TLS handshake task\n      - line 1810: QUIC/HTTP3 listener spawn helper\n\n  * NEW `pub mod feeder_runtime` declaration in\n    `crates/vl-web/src/lib.rs`.\n\n## Test coverage\n\n  * `cargo test -p vl-web --lib feeder_runtime` — 11 / 11 pass:\n      resolve_unset_yields_default\n      resolve_valid_value_passes_through\n      resolve_zero_falls_back_to_default\n      resolve_above_ceiling_falls_back_to_default\n      resolve_garbage_falls_back_to_default\n      build_with_one_worker_runs_a_task\n      build_default_runs_concurrent_tasks\n      build_runtime_threads_carry_feeder_prefix\n      build_runtime_isolated_from_caller_thread\n      ... + 2 more from cve5/vendor_dashboard pinned in this run\n\n  * `cargo test --workspace --all-features` — green; 215 vl-web,\n    309 vl-feeders, no regressions from the runtime split.  The\n    test that asserts \"feeder runtime is isolated from the caller's\n    OS thread\" (`build_runtime_isolated_from_caller_thread`) is\n    the regression-pin that protects against a future refactor\n    accidentally falling back to current_thread or sharing the\n    HTTP runtime."
      },
      {
        "category": "details",
        "text": "## Operational impact assessment\n\n  * HTTP latency under feeder load — IMPROVED.\n    On v0.1.41 a request to /dashboards/* during the CWE\n    enrichment + Debian + EPSS phases would block for 1-3\n    minutes.  On v0.1.42 the same request returns in normal time\n    because the HTTP runtime's 2 workers are not competing with\n    feeders for CPU.  Sub-second responses are the new baseline.\n\n  * Total daemon worker count — UNCHANGED (4 → 2 + 4).\n    Pre-v0.1.42 had 4 HTTP workers shared with feeders.\n    Post-v0.1.42 has 2 HTTP-only workers + 4 feeder-only workers\n    = 6 total tokio worker threads, configurable via the new env\n    var.  RAM increase: ~1 MB per extra thread stack,\n    operationally negligible.\n\n  * Database file format on disk — UNCHANGED.\n    No redb table changed.  redb is already thread-safe via MVCC,\n    so concurrent feeder writes from a multi-thread runtime are\n    correct without code changes.\n\n  * API surface — UNCHANGED.\n    HATEOAS envelopes, JSON shapes, every /api/v1/* route are\n    byte-for-byte identical.  Operators who scripted around the\n    HTTP API are unaffected.\n\n  * Operator config — NEW OPTIONAL ENV VAR.\n    `VL_FEEDER_WORKER_THREADS=8` to scale feeder concurrency.\n    Unset → default 4.  Out-of-range → default 4.  Documented\n    in `crates/vl-web/src/feeder_runtime.rs` rustdoc.\n\n  * Feeder cycle timing — UNCHANGED.\n    Cycles still run on their existing schedules (NVD 2h, ndaal\n    CSAF 6h, EPSS/KEV 12h, CWE 24h, Nuclei 24h, etc.).  The new\n    runtime executes the same async loops; only the isolation\n    changes.\n\n  * Operator deployments via the Ansible role\n    (`tools/ansible/roles/{nvulnlookup,nvulnlookuptesting}`) —\n    UNAFFECTED at the deploy layer.  The role copies the binary\n    and starts the systemd unit; no new role variables.\n\nSecurity classification: NOT a security vulnerability under the\n  CVSS framework.  The defect did not expose data, did not allow\n  privilege escalation, did not crash the daemon, and did not\n  create an unauthenticated attack surface.\n\n  However, this WAS a critical availability defect: the HTTP\n  endpoints went into a multi-minute starvation window during\n  every feeder cycle (worst-case once every 24 hours when CWE\n  enrichment runs).  A v0.1.41 deployment hit by this would\n  appear \"hung\" to operators — exactly the symptom that\n  triggered this advisory's discovery (\"app no reacting\").\n\nThis advisory is filed as `csaf_informational_advisory` per\nOASIS CSAF 2.1 to record the runtime architecture change in\nthe v0.1.42 release cycle.  No CVE is assigned because the\ndefect did not introduce an exploitable vulnerability in the\nproduct itself.",
        "title": "Operational impact"
      },
      {
        "category": "general",
        "text": "## Recommended action for downstream consumers\n\nUPGRADE: 0.1.41 → 0.1.42.  Either:\n\n  * Pull the v0.1.42 release tarball from `release/v0.1.42/`\n    (per-triple) once published.\n  * Update the Ansible role pin from 0.1.41 → 0.1.42 and re-run\n    `ansible-playbook -i inventory site.yml -e\n    nvulnlookup_version=0.1.42`.\n  * Or `cargo install --path vulnerability-lookup-rs/crates/vl-web`\n    from the v0.1.42 git tag.\n\nNo manual intervention is required after upgrading: the daemon\nstarts the new feeder runtime automatically.  The redb storage\nlayer is unchanged so no on-disk migration is required.\n\nOptional tuning: set `VL_FEEDER_WORKER_THREADS=N` to scale\nfeeder concurrency.  N=4 (default) is the recommended start.\nN=8 makes sense if the host has >=16 cores AND you observe\nfeeder cycles taking longer than their scheduled period.  N=1\nis useful for low-resource hosts (Raspberry Pi, container\nlimits) — feeders run sequentially, total CPU footprint drops.\n\n## Verification\n\n  # 1. Confirm the fix is in the source tree:\n  ls vulnerability-lookup-rs/crates/vl-web/src/feeder_runtime.rs\n  # Expected: file exists, ~210 lines.\n\n  # 2. Confirm the dedicated runtime tests pass:\n  cd vulnerability-lookup-rs && cargo test -p vl-web --lib \\\n      feeder_runtime\n  # Expected: 9 / 9 pass.\n\n  # 3. On a running v0.1.42 server, verify the runtime startup\n  # log:\n  grep -E 'Feeder runtime' /tmp/vl-web-0.1.42.log\n  # Expected: \"Feeder runtime: dedicated multi-thread executor\n  # ready worker_threads=4\"\n\n  # 4. Verify HTTP responsiveness during a feeder cycle.  Wait\n  # for the daemon log to show one of the heavy feeders running\n  # (e.g. CWE enrichment, NVD page parse), then time a request:\n  time curl -sk https://localhost:8080/api/v1/system/health\n  # Expected: real time < 1s even when feeders are busy.\n\n  # 5. Optional: verify the env var override.\n  VL_FEEDER_WORKER_THREADS=8 ./target/release/vl-web\n  grep 'worker_threads=8' /tmp/vl-web-0.1.42.log\n\n## Rollback\n\nNot recommended.  v0.1.41 has the HTTP-starvation defect\ndocumented above.  If you must roll back for an unrelated\nreason, restore the v0.1.41 binary; the runtime returns to its\npre-v0.1.42 single-runtime topology and the starvation symptom\nreturns.  No on-disk schema migration is required to roll back.\nv0.1.42 introduced no new redb tables.",
        "title": "Recommended action"
      },
      {
        "category": "other",
        "text": "Disposition: informational — runtime architecture change in `crates/vl-web/src/main.rs` + new `crates/vl-web/src/feeder_runtime.rs` module.  Fixes a v0.1.0..v0.1.41 latent availability defect where heavy feeder phases starved HTTP request handlers.  No exploitable vulnerability in the product itself.  Status final.  Followed up by the v0.1.42 release cycle.",
        "title": "Disposition"
      },
      {
        "category": "legal_disclaimer",
        "text": "THIS DOCUMENT IS PROVIDED ON AN 'AS IS' BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY. ndaal Gesellschaft fuer Sicherheit in der Informationstechnik mbH & Co KG DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS INFORMATION INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS."
      }
    ],
    "publisher": {
      "category": "vendor",
      "contact_details": "security@ndaal.eu",
      "issuing_authority": "ndaal Gesellschaft fuer Sicherheit in der Informationstechnik mbH & Co KG Security Team",
      "name": "ndaal Gesellschaft fuer Sicherheit in der Informationstechnik mbH & Co KG",
      "namespace": "https://ndaal.eu/csaf"
    },
    "references": [
      {
        "category": "self",
        "summary": "This advisory in CSAF 2.1 format",
        "url": "https://gitlab.com/vPierre/ndaal_public_nvulnlookup/-/raw/main/csaf/2026/041/ndaal-sa-2026-041.json"
      },
      {
        "category": "external",
        "summary": "Companion v0.1.42 RFC 4180 quoted-CSV fix advisory ndaal-sa-2026-040",
        "url": "https://gitlab.com/vPierre/ndaal_public_nvulnlookup/-/raw/main/csaf/2026/040/ndaal-sa-2026-040.json"
      },
      {
        "category": "external",
        "summary": "Companion v0.1.41 vendor-dashboard regression advisory ndaal-sa-2026-039",
        "url": "https://gitlab.com/vPierre/ndaal_public_nvulnlookup/-/raw/main/csaf/2026/039/ndaal-sa-2026-039.json"
      },
      {
        "category": "external",
        "summary": "tokio runtime documentation — new_multi_thread() vs new_current_thread()",
        "url": "https://docs.rs/tokio/latest/tokio/runtime/struct.Builder.html"
      },
      {
        "category": "external",
        "summary": "tokio cooperative scheduling caveats — long-running CPU tasks block workers without spawn_blocking",
        "url": "https://docs.rs/tokio/latest/tokio/task/fn.spawn_blocking.html"
      },
      {
        "category": "external",
        "summary": "nvulnlookup CHANGELOG (root)",
        "url": "https://gitlab.com/vPierre/ndaal_public_nvulnlookup/-/raw/main/CHANGELOG.md"
      },
      {
        "category": "external",
        "summary": "nvulnlookup CHANGELOG (workspace mirror)",
        "url": "https://gitlab.com/vPierre/ndaal_public_nvulnlookup/-/raw/main/vulnerability-lookup-rs/CHANGELOG.md"
      }
    ],
    "title": "ndaal Informational Advisory: v0.1.42 — daemon's tokio runtime split into dedicated HTTP and feeder executors so heavy feeder phases (CWE enrichment scan, NVD page parse, Debian 72 MB tracker parse, EPSS gz decompress) no longer starve HTTP request handlers.  v0.1.0..v0.1.41 spawned every feeder loop on the same 4-worker `#[tokio::main]` runtime that served HTTP, producing multi-minute timeouts on /dashboards/* and /api/v1/* during each feeder cycle.  v0.1.42 introduces `crates/vl-web/src/feeder_runtime.rs` with a `pub fn build_feeder_runtime()` helper, trims the HTTP runtime from `worker_threads = 4` to `worker_threads = 2`, rewires every `tokio::spawn(...)` in the feeder block of `main.rs` to `feeder_handle.spawn(...)` (25 sites), and adds `VL_FEEDER_WORKER_THREADS` for operator tuning.  Total daemon capacity 4 → 6 worker threads (2 HTTP + 4 feeder); isolation enforced.",
    "tracking": {
      "current_release_date": "2026-05-08T02:30:00.000Z",
      "generator": {
        "engine": {
          "name": "ndaal CSAF Generator",
          "version": "1.0.0"
        }
      },
      "id": "ndaal-sa-2026-041",
      "initial_release_date": "2026-05-08T02:30:00.000Z",
      "revision_history": [
        {
          "date": "2026-05-08T02:30:00.000Z",
          "number": "1.0.0",
          "summary": "Initial informational advisory documenting the v0.1.42 runtime split that moves the background feeder scheduler off the HTTP-serving runtime onto a dedicated multi-thread executor.  Defect history: present in v0.1.0 through v0.1.41.  Discovery context: v0.1.41 production restart on 2026-05-08 surfaced multi-minute HTTP timeouts during the CWE enrichment + Debian Security Tracker + EPSS feeder phases — operator reported 'app no reacting'.  Fix: new `crates/vl-web/src/feeder_runtime.rs` module with `pub fn build_feeder_runtime() -> Result<Runtime>` + `pub fn resolve_feeder_worker_threads(env) -> usize` constructor pair; `crates/vl-web/src/main.rs` shrinks `#[tokio::main(worker_threads = 4)]` to `worker_threads = 2` and rewires 25 of 27 `tokio::spawn(...)` calls in the feeder block to `feeder_handle.spawn(...)`.  9 unit tests in `feeder_runtime::tests` cover env-var parsing (5 branches) + actual runtime construction (4 end-to-end checks including thread-isolation assertion).  Total daemon worker capacity 4 → 6 (2 HTTP-only + 4 feeder-only); user-tunable via `VL_FEEDER_WORKER_THREADS`.  Status: final."
        }
      ],
      "status": "final",
      "version": "1.0.0"
    }
  },
  "product_tree": {
    "branches": [
      {
        "category": "vendor",
        "name": "ndaal Gesellschaft fuer Sicherheit in der Informationstechnik mbH & Co KG",
        "branches": [
          {
            "category": "product_name",
            "name": "nvulnlookup",
            "branches": [
              {
                "category": "product_version_range",
                "name": "vers:cargo/>=0.1.0|<=0.1.41",
                "product": {
                  "name": "nvulnlookup 0.1.0 through 0.1.41 — `crates/vl-web/src/main.rs` runs HTTP + 25 background feeder loops on the same `#[tokio::main(worker_threads = 4)]` runtime; CPU-bound feeder phases (CWE 296k-row scan, NVD page parses, Debian 72 MB tracker parse, EPSS gz decompress) saturate the 4 workers and HTTP request handlers wait at the bottom of the run queue, producing 1-3 minute timeouts on /dashboards/* and /api/v1/* during each feeder cycle",
                  "product_id": "CSAFPID-0001"
                }
              },
              {
                "category": "product_version",
                "name": "0.1.42",
                "product": {
                  "name": "nvulnlookup 0.1.42 — daemon runtime split into a dedicated 2-worker HTTP runtime (`#[tokio::main(worker_threads = 2)]`) plus a dedicated 4-worker feeder runtime built by `vl_web::feeder_runtime::build_feeder_runtime()`; 25 of 27 `tokio::spawn(...)` calls in `main.rs` rewired to `feeder_handle.spawn(...)`; HTTP latency stays bounded under feeder load; `VL_FEEDER_WORKER_THREADS` env var added for operator tuning (range 1..=64, default 4); 9 unit tests in `feeder_runtime::tests` lock the construction logic + assert OS-thread isolation from the HTTP runtime",
                  "product_id": "CSAFPID-0002"
                }
              }
            ]
          }
        ]
      }
    ]
  }
}