| The Scrapebox Sitemap Scraper Addon allows you to scrape all the urls from a sitemap.
So if a domain has a sitemap on it, it will generally list the vast majority of urls on that domain, if not all of them. With this addon you can extract all of those urls.
You can also do a deep crawl, where the addon will visit each one of those pages on the site map and extract internal links from them as well.
You can of course adjust your settings for connections as well as export the results found.
It uses valid sitemap.xml or other valid sitemap formats. |
No comments:
Post a Comment