Discover Website URLs

Start with a website URL and discover page addresses for a URL inventory or later content collection. You can bound the URL count, crawl depth and pages visited. Your Agent can organize the links actually returned and help choose pages to inspect next.

Your AI

Use with your AI

For tasks with several steps, continue in the AI you already use.

CodexClaude CodeCursorWindsurfCline

View instructions

Reuse my existing Toolmu connection; follow https://toolmu.com/SKILL.md if setup is needed. Get map-site-urls-mrscraper (Tool ID ebf98510-92bb-488a-86d0-8edcd170d52d) and check current availability, inputs and price against my authorized budget. Ask only for the missing starting URL. Use it as body.url and add my chosen limits or access settings only when useful. Treat limit, maxDepth and maxPages as upper bounds, not promised result counts. Use includePatterns or excludePatterns only with confirmed supplier-supported syntax; do not guess matching rules or precedence. Organize the URLs actually returned, retain their original form, and distinguish your grouping from source information. Do not claim complete coverage, reachable pages, page-content extraction or search indexing. Report the actual outcome and confirmed charge when available.

Task:
Find pages starting from [website page link]. If useful, collect up to [50] page links and keep the links actually found so I can choose what to review next.

Paste into your connected AI's conversation.

Connection guide

Up to 120 credits / call

The final charge is based on confirmed usage. Unused reserved credits return to your available balance.

MrScraper

Discover Website URLs

You receive

When discovery returns URLs, use the collected web addresses as a working list of pages to review. Your Agent can arrange the returned links for planning while preserving their original form. The list identifies discovery candidates; reading or extracting each page’s content is a separate step.

Start with your task

Find pages starting from [website page link]. If useful, collect up to [50] page links and keep the links actually found so I can choose what to review next.

What to know

Discovery may be partial because of access, limits or filtering. It does not establish every page on a site or confirm that each returned link is currently reachable. Inclusion in this list is not proof of search indexing or content quality.

Inputs & outputs

Inspect the current inputs, availability, and maximum charge before starting.

Input schema

Input schema
Field / Path	Type	Requirement	Meaning
URL discovery request`body`	`object`	Required	Required object with the starting url and optional discovery/access controls. Only the listed fields are supported. Required fields `url`
Starting website URL`body.url`	`string`	Required	Required full URI identifying where discovery starts. Provide the complete URL, including its http:// or https:// scheme. Format `uri`
Maximum URLs to collect`body.limit`	`integer`	Optional	Optional positive integer cap on collected URLs. It is not a promise that this many URLs will be found; it differs from the number of pages visited. Minimum `1` Maximum `9007199254740991`
Higher-resource access mode`body.super`	`boolean`	Optional	Optional true/false supplier access flag. The description states false as the default and warns that this mode uses more resources. Supply your choice explicitly when needed; success is not guaranteed.
Page-access country`body.geoCode`	`string`	Optional	Optional string of exactly two characters for the country used to load the page. The description states us as the default; it does not define a supported-code list or result language. Minimum length `2` Maximum length `2`
Page-loading wait in seconds`body.timeout`	`integer`	Optional	Optional integer from 1 to 300 seconds for the maximum page-loading wait. The description states 300 as the default. This is not a guaranteed whole-crawl completion time. Minimum `1` Maximum `300`
Maximum crawl depth`body.maxDepth`	`integer`	Optional	Optional positive integer limiting crawl depth from the starting URL. The exact counting convention for the starting level is not specified here. Minimum `1` Maximum `9007199254740991`
Maximum pages to crawl`body.maxPages`	`integer`	Optional	Optional integer from 1 to 20 limiting pages crawled. It is a visit cap, not a requested count of returned URLs or a complete-site guarantee. Minimum `1` Maximum `20`
Proxy-exit country`body.proxyCountry`	`string`	Optional	Optional string of exactly two characters for the proxy-exit country. The description states us as the default; supported codes and regional access success are not guaranteed here. Minimum length `2` Maximum length `2`
URL exclusion patterns`body.excludePatterns`	`array`	Optional	Optional array whose entries are nonempty strings. URLs matching these supplier patterns are skipped. Pattern syntax and precedence with includePatterns are not specified; do not assume a regex or glob dialect.
One URL exclusion pattern`body.excludePatterns[]`	`string`	—	Each entry in excludePatterns must be a nonempty string describing a supplier-supported URL exclusion pattern. This item constraint does not define regex, wildcard syntax or precedence with inclusion rules. Minimum length `1`
URL inclusion patterns`body.includePatterns`	`array`	Optional	Optional array whose entries are nonempty strings. Only matching URLs are requested for discovery. Syntax and precedence with excludePatterns are not specified; empty-array behavior is not established.
One URL inclusion pattern`body.includePatterns[]`	`string`	—	Each entry in includePatterns must be a nonempty string describing a supplier-supported URL inclusion pattern. This item constraint does not establish matching syntax, overlap precedence or how an empty array behaves. Minimum length `1`

View raw JSON

{
  "type": "object",
  "required": [
    "body"
  ],
  "properties": {
    "body": {
      "type": "object",
      "$schema": "https://json-schema.org/draft/2020-12/schema",
      "required": [
        "url"
      ],
      "~standard": {
        "vendor": "zod",
        "version": 1,
        "jsonSchema": {}
      },
      "properties": {
        "url": {
          "type": "string",
          "format": "uri",
          "description": "Full URL to scrape, including the http:// or https:// scheme."
        },
        "limit": {
          "type": "integer",
          "maximum": 9007199254740991,
          "minimum": 1,
          "description": "Maximum number of URLs to collect."
        },
        "super": {
          "type": "boolean",
          "description": "Route through real devices to bypass hard anti-bot protection. Uses more upstream resources, so runs consume more tokens. Default false."
        },
        "geoCode": {
          "type": "string",
          "maxLength": 2,
          "minLength": 2,
          "description": "Two-letter country code to load the page from (default 'us')."
        },
        "timeout": {
          "type": "integer",
          "maximum": 300,
          "minimum": 1,
          "description": "Maximum seconds to wait for the page to fully load (default 300)."
        },
        "maxDepth": {
          "type": "integer",
          "maximum": 9007199254740991,
          "minimum": 1,
          "description": "Maximum crawl depth from the starting URL."
        },
        "maxPages": {
          "type": "integer",
          "maximum": 20,
          "minimum": 1,
          "description": "Maximum pages to crawl (1-20)."
        },
        "proxyCountry": {
          "type": "string",
          "maxLength": 2,
          "minLength": 2,
          "description": "Two-letter country code of the proxy exit (default 'us')."
        },
        "excludePatterns": {
          "type": "array",
          "items": {
            "type": "string",
            "minLength": 1
          },
          "description": "URLs matching these patterns are skipped."
        },
        "includePatterns": {
          "type": "array",
          "items": {
            "type": "string",
            "minLength": 1
          },
          "description": "Only URLs matching these patterns are crawled."
        }
      },
      "additionalProperties": false
    }
  },
  "additionalProperties": false
}

Result schema

Output fields have not been documented yet.

Additional notes

This task follows a website’s link structure to discover URL candidates and build a working page inventory. It provides a starting list for later content collection, rather than the full content of every discovered page or a visual site-design diagram.

The three scope controls have different jobs: limit caps collected URLs, maxPages caps pages crawled, and maxDepth bounds exploration from the starting URL. Fewer links may be found than the chosen limits. These settings do not prove complete site coverage.

includePatterns requests discovery only for matching URLs; excludePatterns asks to skip matching URLs. Each is an optional array of nonempty strings. Matching syntax and the precedence of overlapping include/exclude rules are not established here, so do not invent regex or wildcard examples.

Access options are separate from discovery scope. Country settings describe where page access or proxy exit occurs; they do not translate the result. Super mode uses more supplier resources and does not guarantee access. Timeout limits the page-loading wait, not a promised completion time for the whole crawl.

Keep the returned URL forms and explain any grouping or cleanup the Agent performs. A discovered address is not a verified page-content record, working-link test or indexing result. Missing URLs may reflect scope or access limits rather than removed pages.

MrScraper: general URL-discovery guidance

Questions about this tool.

Inputs, results and details to know before running it.

Get connected

The intended result is discovered page addresses for further work. Reading page content, checking links or drawing a diagram requires a separate step; a URL in the list is not proof that its page has been extracted.

Discover Website URLs

Discover Website URLs

You receive

Start with your task

What to know

Inputs & outputs

Input schema

Result schema

Copy a link from the site you want to explore

Choose how many page links to look for

Review the links you actually found

Questions about this tool.

You receive

Start with your task

What to know

Inputs & outputs

Input schema

Result schema

Copy a link from the site you want to explore

Choose how many page links to look for

Review the links you actually found

Questions about this tool.

Does discovery return each page’s text or a visual sitemap?

How do URL count, page count and depth differ?

Can I use any regex or wildcard in include/exclude rules?

Does a missing URL prove the page is gone?