The VLsearch Robot

The WWW Virtual Library indexes its members' pages. To do so it uses a 'robot' or search engine, the part of this that visits members' sites is called "VLsearch".

The robot is tightly controlled, it visits:

• The central Catalog at http://vlib.org/
• Member sites listed on the Catalog, that are listed as 'active'.
• Occasionally, it will go out from member sites and visit a very shallow layer of the sites that are themselves listed in member libraries.
  • This is relatively rare.
  • The depth is unlikely to be beyond one page.
  • The robot is programmed to respect the Robots Exclusion Protocol.

If you are reading this page it is probably because you have found references to the robot's visit in your server logs. If you are a VL member this is normal, the central VL is indexing your site for the common VL search engine.

If the visits are unwanted, you can simply exclude them with the Robots Exclusion Protocol.

Unwanted visits are most likely due to:

• A URL incorrectly placed in the VL Catalog;
• A URL in the Catalog which whilst it should be there, has other material within the same directory tree. For example, if your virtual library is at http://www.somewhere.edu/myname/vl/index.html the robot will attempt to index everything in the directory http://www.somewhere.edu/myname/vl/. (So, if you have, say, http://www.somewhere.edu/myname/vl.html but with other material than your virtual library in http://www.somewhere.edu/myname/, the robot will 'escape'. This wastes our resources and your bandwidth (and 'pollutes' the central index). Please try and keep you VL in its own directory.)
• Poor/non- use of the Robots Exclusion Protocol.

If the above does not solve any problem, do please contact us and we will try and resolve matters from this end.