In many cases, bad guys compromise websites or exploit a website's scripts to use that site for their own malicious purposes. One of the common purposes is to post or advertise their own business and increase their ranking in search engine queries.

Unfortunately, the material or businesses posted on these websites (or seemingly posted on these websites) does not meet our acceptable use criteria, or the expectations of our stakeholders. In many cases the material may be offensive and detrimental to the image of Berkeley Lab.

Even after cleaning the LBL system of this content, major search engines may continue to show it in links and or cached web pages for very long periods of time. Because of LBL's relatively strong position in search engine rankings, a compromised site could easily show up in the top ten (or top one!) results for a, umm, decidedly non-science oriented search.

For these reasons, the Computer Protection Program requires that web caches, such as those of Yahoo and Google, be cleaned of inappropriate content that points to Berkeley Lab.

The major search engines provide procedures and tools to remove material from their caches. Below we post the links to the procedures for the most frequently used search engines. We will update this list as necessary.

Google:
http://googlewebmastercentral.blogspot.com/2007/04/requesting-removal-of-content-from-our.html

Yahoo and Alta vista:
http://help.yahoo.com/l/us/yahoo/search/siteexplorer/delete/siteexplorer-46.html

Typical Patterns for Managing Search Engine Content:

Case 1: LBL System is Actually Compromised

1. System is taken offline for forensics

2. If system can be quickly rebuilt, return it to service clean. If system cannot be quickly rebuilt, point the domain name(s) at an alternate site which will return 404 errors for the content.

3. Execute the procedures to clean the search engines

Case 2: LBL System is Spammed (no actual compromise)

1. Clean the system of the content - wherever possible, ensure that 404 or equivalent errors are returned for pages which should not exist - this ensures quick cleaning.

2. Execute the procedures to clean the search engines

For both cases, it is critical that system owners take the time to understand the nature of the compromise and take steps to prevent these issues in the future.

See also:
Minimum Expectations for Web Servers

Dangers of Open Posting