Rapid Web Assessments with RAWR

A few weeks ago I had an opportunity to chat with Adam “@al14s” Byers and Tom “@c0ncealed” Moore at AIDE about an interesting new assessment tool they created called RAWR or Rapid Assessment of Web Resources. Adam was kind enough to write up a quick post for us so we could pass the word on to others in the security community.

In summary … feed it an Nmap XML results file and RAWR parses through it looking for any web resources to interrogate. After RAWR’s analysis the output consists of any useful information it extracts from a service as well as a picture of the web interface.

Anyway … on to Adam’s post with all the details…

Web Assessments

One of the highest threats to organizations today is, in most cases, also one of their most prevalent – web services. Through the years, the landscape has changed from simple static websites to fully functional web-based applications that provide access to internal information gold mines. Most organizations have little to no knowledge as to how many internal web resources they have within their environments, many of which leave clients open to network compromise. Getting a clear picture of an organization’s internal or external websites can be a time consuming process. If you are tasked to ensure the security of your client’s web interfaces, you’ll find that there is a lot involved – and usually not a lot of time to get the report out. Over the past few months, I’ve written a cross-platform, open-source tool that will take your next web assessment from discovery to analysis in one fell swoop.

The Innards

RAWR is a python application (tested w/ 2.7) that takes your scan data, which must be nmap .xml or .nessus at the moment, and uses it to gather as much data as possible on any web services that turn up. It utilizes multi-threading and queues to quickly and efficiently pull data from each host. In the background, it uses phantomJS for taking screenshots of the interface.

We gave an outline of how it works at CarolinaCon 9:

There are a few features that aren’t necessarily documented in the ‘help’ text. In the script, scroll to the ‘Settings’ header (~line 240). You’ve got options for the user agent, number of threads to use when making the web calls, amongst others. The value for ‘csv_sort_col’ enables you to choose a column by which to sort the .csv output. The order of the .csv’s columns can be changed by moving the values around in ‘flist’. While they didn’t really fit as command-line switches, I tried to make these ‘under the hood’ options as usable as possible.

Be sure to check out the ENUM options. You’ll probably get a little more info using HTTP 1.0 via the ‘–downgrade’ switch, or find a web server that still has TRACE/TRACK enabled using the ‘-o’ switch to get available methods. There’s room for more, and I’m always interested in the tidbits pentesters find valuable when they go to do one of these assessments. Feature requests are definitely welcome!


The goal with RAWR’s output is to give the pentester as much as we can for their report. I believe that the value in a good assessment isn’t all about the tools, so much as it is about the ability to use the tools to gather the right info and then the understanding to interpret it.

The HTML report was designed to make it easy to find interesting interfaces by having a jQuery-driven, fully searchable interface. All of the gathered info on a specific host can be pulled up by clicking the ‘i’ when you hover over a thumbnail. You can form lists by selecting hosts, then pull up ‘iplist’ and copy/paste the info into, say, an input list for Nikto.

I recommend pulling the .CSV up along with the HTML report. The two used in conjunction make LHF identification a ‘snap’. Within the .CSV is every bit of information gathered, along with a ‘notes’ column for marking down any items of interest. Something I didn’t consider until recently was that RAWR can be used for SSL certificate assessments as well.

Images, Cookies, SSL_Certs, and Robots are folders within the log folder that will contain files that were obtained. Other than that, you have the nmap output and the .log file, which is where I’m directing error messages. If there are any problems during execution, they’ll show up in the .log.

Thanks for checking out RAWR. If there’s something you see that would make it more useful – be sure to submit a feature request to our BitBucket Issues page!


And don’t forget … if you are interesting in posting an article on NovaInfosec.com, please head on over to our Submit Article page for all the details as well as our submission form. Today’s post image is from the good folks over at Twitter.com.

1 comment for “Rapid Web Assessments with RAWR

  1. May 15, 2013 at 9:07 am

    Rapid Web Assessments with RAWR https://t.co/5KyEDXesKQ

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.