Improve crawler impact

[kyzh]

What

• We could improve and limit crawler impact by changing the robots.txt and creating a sitemap.
• The first one we could use to tell robot not to index some part like created graphs etc..
• The second could help us put emphasis on places that are the most relevant like the country landing page.

Part of

• 🚅 Performance (Tracker) #8764
• 🕷️ Indexation (tracker) #7583

[stephanegigandet]

Search queries, graphs etc. are disallowed in the current robots.txt:

User-agent: *
Disallow: /cgi

Our URLs are user friendly (as in readable, understandable and typable) : /[tag type]/[tag value]/[other tag type]/[other tag value]

But they are not well suited for robots.txt rules: too many combinations at the first level already. no follow rules at the link level are probably more suited. e.g. to no follow the [other tag] links and make engine index only one level.

[raphael0202]

I'm closing this issue, as we already have more detailed issues that describe indexing and crawling.