A website's structure refers to how the website is set up, i.e. how the individual subpages are linked to one another.
It is particularly important that can find all subpages quickly and easily when websites have a large number of subpages. For this reason, a website’s needs to have links to the most important subpages. Files such as and also help the crawler do its job in this regard.