Extract website URLs from sitemaps

Provide a website domain to collect the URLs listed in its sitemaps with Context.dev. Limit the number of links, filter addresses, or choose a specific sitemap alongside the required domain.

Your AI

Use with your AI

For tasks with several steps, continue in the AI you already use.

CodexClaude CodeCursorWindsurfCline

View instructions

Use Toolmu Tool crawl-sitemap-context-dev to find public webpage links for the website I share. Accept a full website link or a link to a page. Derive the required website name from a link by removing the scheme, port, path, query, fragment and one leading www.; keep other subdomains. Never ask the user to type a bare domain or edit their link. A link that already lists the website pages is optional; use it as the page-list source if supplied. Ask how many page links the user wants if no number is given. If they want only certain kinds of pages, accept a plain description and translate it only when clear; otherwise ask a simple follow-up. Do not ask the user to provide code or a technical filter. Reuse the existing Toolmu connection and authorization, follow the on-page setup guide if needed, and stay within the agreed budget. Show the returned page links and any reported counts or processing errors. Reading page contents is a separate task; do not claim the list covers the whole website or proves search-engine indexing.

Task:
Find up to [number of pages, e.g. 20] from [website link]. If I also share a link that lists its pages, use it too so I can choose what to read.

Paste into your connected AI's conversation.

Connection guide

Up to 0.00108 credits / call

The final charge is based on confirmed usage. Unused reserved credits return to your available balance.

Context.dev

Extract website URLs from sitemaps

Provide a website domain to collect the URLs listed in its sitemaps with Context.dev. Limit the number of links, filter addresses, or choose a specific sitemap alongside the required domain.

You receive

You receive a list of webpage URLs to review before choosing which pages your Agent should read. The response can also identify the domain and report how many sitemaps were discovered, fetched or skipped, along with processing errors. Your Agent can organize the returned links for selection. Reading the selected pages is a separate task.

Start with your task

Find up to [number of pages, e.g. 20] from [website link]. If I also share a link that lists its pages, use it too so I can choose what to read.

What to know

The list is limited by the requested link count, any URL filter and the sitemap sources processed. It does not establish complete website coverage or search-engine indexing, and it does not include the contents of the listed pages.

Inputs & outputs

Inspect the current inputs, availability, and maximum charge before starting.

Input schema

Input schema
Field / Path	Type	Requirement	Meaning
Sitemap request settings`queryParams`	`object`	Required	Required settings object. Supply domain here; optional settings choose a sitemap, limit the URL list, filter addresses or set request options. Required fields `domain`
Zero data retention mode`queryParams.zdr`	`string`	Optional	Optional enabled or disabled value. enabled bypasses Context.dev’s shared caches and omits request and response content from Context.dev’s retained usage logs. The Context.dev organization must already have zero data retention enabled; otherwise the request fails with ZDR_NOT_ENABLED. Options `enabled, disabled`
Website domain`queryParams.domain`	`string`	Required	Required domain, such as stripe.com, with at least 3 characters. It remains required when sitemapUrl selects a particular sitemap. Minimum length `3`
Maximum returned URLs`queryParams.maxLinks`	`integer`	Optional	Optional upper limit for returned URLs: 1 to 100,000, with a documented default of 10,000. This is a limit, not a promised result count. Minimum `1` Maximum `100000`
URL filter`queryParams.urlRegex`	`string`	Optional	Optional RE2-compatible expression of at most 256 characters. Only matching URLs are returned and counted toward maxLinks. Maximum length `256`
Supplier response timeout`queryParams.timeoutMS`	`integer`	Optional	Optional supplier wait limit in milliseconds, from 1,000 to 300,000. A supplier timeout returns HTTP 408; keep this value below the Tool’s own request timeout. Minimum `1000` Maximum `300000`
Selected sitemap URL`queryParams.sitemapUrl`	`string`	Optional	Optional full sitemap URL. When supplied, the service uses this sitemap instead of discovering the domain’s sitemaps; domain must still be provided. Format `uri`

View raw JSON

{
  "type": "object",
  "required": [
    "queryParams"
  ],
  "properties": {
    "queryParams": {
      "type": "object",
      "$schema": "https://json-schema.org/draft/2020-12/schema",
      "required": [
        "domain"
      ],
      "~standard": {
        "vendor": "zod",
        "version": 1,
        "jsonSchema": {}
      },
      "properties": {
        "zdr": {
          "enum": [
            "enabled",
            "disabled"
          ],
          "type": "string",
          "description": "Set 'enabled' to bypass Context.dev's shared caches and omit request and response content from its retained usage logs. Requires zero data retention on the Context.dev organization, otherwise the call fails with ZDR_NOT_ENABLED."
        },
        "domain": {
          "type": "string",
          "minLength": 3,
          "description": "Domain to build a sitemap for, e.g. 'example.com'."
        },
        "maxLinks": {
          "type": "integer",
          "maximum": 100000,
          "minimum": 1,
          "description": "Maximum number of links to return. Default 10000, max 100000."
        },
        "urlRegex": {
          "type": "string",
          "maxLength": 256,
          "description": "RE2-compatible pattern. Only matching URLs are returned and counted against maxLinks."
        },
        "timeoutMS": {
          "type": "integer",
          "maximum": 300000,
          "minimum": 1000,
          "description": "Upstream timeout in milliseconds (max 300000). Context.dev aborts the call with a 408 when exceeded. Keep it below the endpoint's requestTimeoutMs so the provider answers before the platform's own budget expires."
        },
        "sitemapUrl": {
          "type": "string",
          "format": "uri",
          "description": "Explicit sitemap URL. When provided, exactly this sitemap is crawled instead of discovering the domain's sitemaps."
        }
      },
      "additionalProperties": false
    }
  },
  "additionalProperties": false
}

Result schema

No fixed output contract is available. Field types reflect saved result evidence; individual responses can differ.

Result schema
Field / Path	Type	Meaning
Returned webpage URLs`result.urls`	`array`	URLs listed by the processed sitemap source, subject to the requested limit and URL filter. These are addresses for selecting pages, rather than the contents of those pages.
Result domain`result.domain`	`string`	The domain identified in the sitemap result, which helps associate the returned addresses with the requested website.
Sitemap processing counts`result.meta`	`object`	Processing details accompanying the URL list. Where returned, distinguish discovered, fetched and skipped sitemap counts from the processing error count.
Sitemaps discovered`result.meta.sitemapsDiscovered`	`number`	Number of sitemaps reported as discovered during this request. It is a sitemap count, not a count of website pages.
Sitemaps fetched`result.meta.sitemapsFetched`	`number`	Number of sitemaps reported as fetched during processing. It does not mean that the pages listed in those sitemaps were read.
Sitemaps skipped`result.meta.sitemapsSkipped`	`number`	Number of sitemaps reported as skipped. Compare it with the other processing counts when interpreting the returned URL list.
Sitemap processing errors`result.meta.errors`	`number`	Error count reported for sitemap processing. This does not establish how many of the returned page addresses are unavailable.

Questions about this tool.

Inputs, results and details to know before running it.

Get connected

Set maxLinks between 1 and 100,000; the documented default is 10,000. The actual list can be shorter because this is a maximum, and URL filters also affect which addresses are returned.

Extract website URLs from sitemaps

Extract website URLs from sitemaps

You receive

Start with your task

What to know

Inputs & outputs

Input schema

Result schema

Use this tool with your AI assistant

Share a website link

Choose pages to read next

Questions about this tool.

You receive

Start with your task

What to know

Inputs & outputs

Input schema

Result schema

Use this tool with your AI assistant

Share a website link

Choose pages to read next

Questions about this tool.

How many URLs can I request?

Can I supply a specific sitemap URL?

How do I limit results to a section of the website?

Does an empty list mean the website has no pages?

What is required to use zero data retention?