Added agent function, modified robots.txt handler to use it.
Modified HttpHandler.connect() to have a head argument with a default of false.