Opened 4 years ago

Closed 2 years ago

#16596 closed bug (fixed)

extend review.haiku-os.org/robots.txt

Reported by: korli Owned by: kallisti5
Priority: normal Milestone: R1/beta4
Component: Website/Gerrit Version:
Keywords: Cc:
Blocked By: Blocking:
Platform: All

Description

Google returns search results from Gerrit main page, inclusive email addresses.

Change History (4)

comment:1 by waddlesplash, 4 years ago

Owner: changed from haiku-web to kallisti5
Status: new → assigned

comment:2 by pulkomandy, 3 years ago

Component: Sys-Admin → Website/Gerrit

comment:3 by kallisti5, 2 years ago

Gerrit already includes one by default:

​https://review.haiku-os.org/robots.txt

comment:4 by kallisti5, 2 years ago

Milestone: Unscheduled → R1/beta4
Resolution: → fixed
Status: assigned → closed

Old default: 

# Directions for web crawlers.
# See http://www.robotstxt.org/wc/norobots.html.

User-agent: HTTrack
User-agent: puf
User-agent: MSIECrawler
User-agent: Nutch
Disallow: /

New: 

# We really don't like crawlers on review.haiku-os.org
Disallow: /
Note: See TracTickets for help on using tickets.