Przejdź do treści

Fetching pages

When you send a url, gspamjudge does not download the page from its own servers. In production every page and sitemap is fetched through an external fetching provider, configured by the operator. Providers are tried in order: if one is blocked (403/429/503, a challenge page or a captcha) or times out, the next one is used.

Plain fetch and JavaScript rendering

Each project has a Render JavaScript setting (panel: project → Settings; API: PATCH /v1/projects/{id} with render_js_default). You can override it per check, audit, API call or CSV row.

Mode What happens Price
off (default) Raw HTML as the server sends it, never rendered. The cheapest mode and the best evidence for hidden text and cloaking. no surcharge
on Every page is rendered in a browser. +$0.0060 per page (shown in the estimate)
auto Raw HTML first; rendered only if the plain fetch is blocked or returns an empty JavaScript shell. +$0.0060 only for pages actually rendered

Overrides:

  • POST /v1/judge, POST /v1/judge/batch items and POST /v1/jobs items: "render_js": true | false | "auto" (missing = project setting);
  • site audits: render_js in the body (the older js: true means on), the Render JavaScript switch in the wizard, or a CSV column render_js (true, false, auto; empty = the audit setting);
  • panel: the Render JavaScript switch on the new check form.

With off, a page that looks like an empty JavaScript shell is still checked, and the report warns “This page may need JavaScript” with a + Retry with rendering · $0.0060 button. Pages with your own HTML are never fetched or rendered. The report metadata (fetch: provider, rendered, js_shell) shows how the page was fetched; it is not sent to the model.

Stealth mode (residential proxies) is available to admins only. The surcharge exists only when a provider with rendering is configured.

Errors

A failed fetch returns 502 fetch_failed. Typical causes: the site blocks all providers, the page does not exist (404 - the fetch chain stops there) or no provider is configured. Error messages contain only the host.

Alternative: upload your own HTML

If a site cannot be fetched (login wall, strong bot protection) or you do not want its URLs sent to a third party, send the HTML yourself - nothing is fetched:

  • POST /v1/judge with html instead of url;
  • site audits: items with url and html, or in the panel a CSV with columns url,html or a ZIP with index.csv (url,filename) and the .html files (see Site audits).

Privacy

Page URLs are sent to the fetching provider. Tracking and secret query parameters are removed before the fetch.