Implement bfac tricks into url_fuzzer #17434

alexandreborgo · 2018-11-23T00:49:41Z

Hey, here is a first try.

What I added:

new appendables, backup_exts and file_types
new set of prependables (like Copy_of_)
a function _mutate_by_prepending to generate new URLs from this set
a function _mutate_file_name to generate change on the name of the file like : index.html -> index-backup.html, index (copy).html, .index.html.swp, etc
and ofc the calls to the new functions

I checked that new patterns don't create duplicates of existing URLs.
I don't know if the function _mutate_file_name is the best way to do it. I used the same technique as bfac, what do you think about this function ?

I performed some tests, everything works properly but I found something strange. The plugin finds all 26 URLs it could find on the server but the list of fuzzable requests contains several times http://localhost/.
See the attached file output.txt for the list of file on the test server and w3af output.

This bug as something to do with the new function _mutate_file_name. It found 6 new URLs and there's 6 more http://localhost/ in the fuzzable requests. If the plugin don't call the function the result is good. I'm sure that the function doesn't create this URL directly. I'll work on it.

andresriancho · 2018-11-23T11:30:51Z

w3af/plugins/crawl/url_fuzzer.py

-                    '.gzip', '.bzip2', '.inc', '.zip', '.rar', '.jar', '.java',
-                    '.class', '.properties', '.bak', '.bak1', '.bkp', '.back',
-                    '.backup', '.backup1', '.old', '.old1', '.$$$'
+    _appendables = ('~', '~~', '_', '.', '.tar.gz', '.gz', '.7z', '.cab',


Now that these lists are getting larger, we have the risk of duplicating one item in them.

Let's change the type from tuple (current) to set (desired). By doing so we make sure that even if we add duplicates in the variable definition, the set will eliminate them.

Just change: _appendables = (...) with _appendables = {...}.

The same change should be applied to all other class attributes that define lists like this in the url_fuzzer class.

andresriancho · 2018-11-23T11:34:57Z

w3af/plugins/crawl/url_fuzzer.py

+        if filename:
+            domain = url.get_domain_path()
+            url_string = domain.url_string
+            name = filename[:filename.rfind(u'.')]


What happens here when the input URL is http://w3af.org/foobar (there is no point in the filename)

See below's answer.

andresriancho · 2018-11-23T11:36:39Z

w3af/plugins/crawl/url_fuzzer.py

+
+            mutate_name_testing = (
+                url_string + '#' + filename + '#',
+                url_string + name + ' (copy).' + extension,


And also, following the same example above where the filename has no "name", what happens in these lines when extension is empty? Does it make sense to send these tests?

These cases should be investigated and handled:

http://w3af.org/foobar

http://w3af.org/foobar.

http://w3af.org/.hidden

http://w3af.org/.

Yes it wasn't handling these files correctly, I did some modifications and I believe it does it well now.

Here are the tests I did and the results : https://pastebin.com/WG5szsnB (GitHub importing files system doesn't work right now so I put the output on Pastebin)

andresriancho · 2018-11-23T11:40:47Z

w3af/plugins/crawl/url_fuzzer.py

+            )
+
+            for change in mutate_name_testing:
+                newurl = URL(change)


Just in case, do a try/except around the URL instance creation, we never know what could go wrong when crafting these URLs.

I'm not sure if you want the try/except in w3af or if it is just to find out what's generating all http://localhost/ in the list fuzzable requests.

I did this:

try : newurl = URL(change) yield newurl except ValueError as exception: om.out.information('Error while generating a new URL: '\ + exception)

And nothing wrong came from it during all my tests.

Let me know if I have to add the exception to the pull request.

Yes, the code you show in the comment is what I wanted, just in case... you never know...

The only change I would propose to that code is to move the yield newurl to the else: section.

andresriancho · 2018-11-23T11:50:37Z

I took a quick look at the output.txt file and noticed these lines:

The list of fuzzable requests is:
- Method: GET | http://localhost/
- Method: GET | http://localhost/
- Method: GET | http://localhost/
- Method: GET | http://localhost/
- Method: GET | http://localhost/
- Method: GET | http://localhost/
- Method: GET | http://localhost/
- Method: GET | http://localhost/
- Method: GET | http://localhost/
- Method: GET | http://localhost/
- Method: GET | http://localhost/

This is what you said was strange, right?

I also believe it is strange... but can't understand why it is like that...

Could you share the w3af script you are using to run the scan, and a little bit more about the target web application? Thanks!

alexandreborgo · 2018-11-24T12:27:09Z

I took a quick look at the output.txt file and noticed these lines:
The list of fuzzable requests is:
- Method: GET | http://localhost/
- Method: GET | http://localhost/
- Method: GET | http://localhost/
- Method: GET | http://localhost/
- Method: GET | http://localhost/
- Method: GET | http://localhost/
- Method: GET | http://localhost/
- Method: GET | http://localhost/
- Method: GET | http://localhost/
- Method: GET | http://localhost/
- Method: GET | http://localhost/
This is what you said was strange, right?

I also believe it is strange... but can't understand why it is like that...

Could you share the w3af script you are using to run the scan, and a little bit more about the target web application? Thanks!

Yes this is it. I haven't looked for what causes it yet.

I'm using w3af with these modifications (my fork's branch url_fuzzer-bfac-tricks). It only appears when the function _mutate_file_name is called (and not with _mutate_by_prepending).

I tested it with two servers and find the same result in both:

first with Python's SimpleHTTServer with the filles in the working directory
then with Apache server with default configuration and all the files in /var/www/html

alexandreborgo added 9 commits November 22, 2018 01:19

new appendable extensions

5174f49

new prepandable patterns

e58fe70

new backup extensions

5fa9032

new file types

9e7235a

mutate by prepending patterns to file names

d827ec7

remove .inc (duplicate) and add .$

73bc932

mutate by prepending and/or appending to file names

ab85c79

call to new mutate functions in _do_request

6f75847

put the extension inside few transformations like in bfac

3a5fd89

andresriancho reviewed Nov 23, 2018

View reviewed changes

andresriancho requested changes Nov 23, 2018

View reviewed changes

andresriancho reviewed Nov 23, 2018

View reviewed changes

alexandreborgo added 2 commits November 23, 2018 22:54

tuple to set to avoid duplicates

ba8a1d2

proper handling of files like foo, foo. and .foo

4668c0d

add try/except around new url's generation

5fa2747

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement bfac tricks into url_fuzzer #17434

Implement bfac tricks into url_fuzzer #17434

alexandreborgo commented Nov 23, 2018

andresriancho Nov 23, 2018

alexandreborgo Nov 24, 2018

andresriancho Nov 23, 2018

alexandreborgo Nov 24, 2018

andresriancho Nov 23, 2018

alexandreborgo Nov 24, 2018

andresriancho Nov 23, 2018

alexandreborgo Nov 24, 2018

andresriancho Nov 29, 2018

andresriancho commented Nov 23, 2018

alexandreborgo commented Nov 24, 2018

Implement bfac tricks into url_fuzzer #17434

Are you sure you want to change the base?

Implement bfac tricks into url_fuzzer #17434

Conversation

alexandreborgo commented Nov 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andresriancho commented Nov 23, 2018

alexandreborgo commented Nov 24, 2018