The full response headers are also included in the Internal tab to allow them to be queried alongside crawl data. HTTP Headers – This will store full HTTP request and response headers which can be seen in the lower ‘HTTP Headers’ tab.If there server does not provide this the value will be empty. Last-Modified – Read from the Last-Modified header in the servers HTTP response.More detailed information can be found in our FAQ. Response Time – Time in seconds to download the URL.The following URL Details are configurable to be stored in the SEO Spider. This is similar to behaviour of a site: query in Google search.Ĭonfiguration > Spider > Extraction > URL Details Please note – If a crawl is started from the root, and a subdomain is not specified at the outset (for example, starting the crawl from ), then all subdomains will be crawled by default. ![]() ![]() The CDNs configuration option can be used to treat external URLs as internal. To crawl all subdomains of a root domain (such as or ), then this configuration should be enabled. These will only be crawled to a single level and shown under the External tab.įor example, if is entered as the start URL, then other subdomains discovered in the crawl such as or will be treated as ‘external’, as well as other domains such as etc. Configuration > Spider > Crawl > Crawl All Subdomainsīy default the SEO Spider will only crawl the subdomain you crawl from and treat all other subdomains encountered as external sites.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |