When you install Hyrax for the first time it is pre-configured to serve test data sets that come with each of the installed data handlers. This will allow you to test the server and make sure it is functioning correctly. After that you can customize it for your data.
Deploying a robots.txt file for Hyrax is synonymous with deploying it for Tomcat. This means that your robots.txt file must be accessible here:
http://you.host:port/robots.txt
For example:
http://www.opendap.org/robots.txt
Note: Placing robots.txt lower in the URL path does not seem to work
In order to get Tomcat to serve the file from that location you must place it in $CATALINA_HOME/webapps/ROOT.
If you find that you system is still burdened with robot traffic then you might want to try the BotBlocker handler for the OLFS.