3.0.5

New

• Add a none writer. Useful for testing scenarios when you want to silence warnings etc. (7ccc4c9)
• (build) Add makefile target to ensure API documentation is up to date (c0a577c)

Fixed

• Show an error when Chrome is not available and you're using the default chromedp driver (216c03d)
• Don't create empty SQLite3 databases when not required (83b61c3)
• Show screenshots in the report server when they are stored in the database (using the --write-screenshot flag) (a40f82a)
• Fix a UI error in the job submission view (ff73ef8)
• Ensure startup flags are used when using the gorod driver (4ff862a)
• Make cleanup routines using the gorod driver more reliable (2321e6c)
• Update nmap.go to use HTTPS as well instead of HTTP only by @Mickdep in #252

New Contributors

• @Mickdep made their first contribution in #252

Full Changelog: 3.0.4...3.0.5

d92a75becd30ddf89c399742a085af92e1b4c790  gowitness-3.0.5-darwin-amd64
b29de43fb107ed5b8d9da145e32bed06615e3421  gowitness-3.0.5-darwin-arm64
2ace88689e0e38e37b8b48bc017284140e9918b8  gowitness-3.0.5-linux-amd64
c379fda1567cbb07b430e993741ac5b2f82b6536  gowitness-3.0.5-linux-arm
437b32e196e840262189ab8974cbcc54830c2431  gowitness-3.0.5-linux-arm64
bc4d4480cc5cd06bb2f1d862965d3ad50e3715ce  gowitness-3.0.5-windows-amd64.exe
b0a4cf9663353534ab2f6b54c2f1c374a7a8c8ef  gowitness-3.0.5-windows-arm64.exe

3.0.4

This release adds HTML searching and keyboard navigation to the detail view, while also making slight UI tweaks and fixing some CLI bugs.

new

• Add search by HTML body, fixing #236 (97f1a78)
• (ui) Add keyboard navigation to the detail view so that you can use the left and right arrows to quickly flick between results (c5c57f7) 🔥

changed

• (ui) Tweak HTML copy modal in report server (1e57d37)
• (ui) Remove duplicate close icon (74e7b60)
• (ui) Screenshot modals in the detail view now show the URL and timestamp taken (992742f)

fixed

• Use correct command options structure for report generate, fixing #240 (c7eace5)
• Write - when an illegal character is found in a potential file name, makes parsing as described #242 possible again too (4918ba4)

other

• Bump dependencies (737f590) (db94b2f) (f5713f3)
• (ci) add workflow to test go install (f6bd224)

80ddae1f00dae61d9f8f5f4011153a5edf66b531  gowitness-3.0.4-darwin-amd64
439c30f3c2bbf9b3cf68626c407957c2f5a173e5  gowitness-3.0.4-darwin-arm64
5b185c6ac65896efee4acb031aa022e477cafe2a  gowitness-3.0.4-linux-amd64
c26f9a8f2e1c80a9e5b025b2579355e518c49c7f  gowitness-3.0.4-linux-arm
02793e436226038c08ed5f7ecf3389662457aca7  gowitness-3.0.4-linux-arm64
d197d907d273f78a70b93de8a21279396fc13d51  gowitness-3.0.4-windows-amd64.exe
4a375c8a878bd60c47abfbd6e0c470aabe7d999e  gowitness-3.0.4-windows-arm64.exe

Full Changelog: 3.0.3...3.0.4

3.0.3 - API Delay && Immediate Mode UI

new

• Add the delay option to the /submit and /submit/single API endpoints. This adds a delay between when the page is considered loading and when a screenshot should be taken. The command line already adds a delay, but this adds it to the API as well (see Swagger docs) (1db81b3)
• Add new web UI for probing using immediate mode. Immediate mode does not record data to the database but instead shows a modal with the results (see screenshots below) (bb5cf4f)

fixed

• Fix API documentation for /submit/single to show the full return object (a9e254d)
• Trim spaces for file targets to help improve reliability of the reader (fd9f55f)

Full Changelog: 3.0.2...3.0.3

ba104c2f84fcebd3f9a6d42417d44afbdfa3baf5  gowitness-3.0.3-darwin-amd64
05a1abce98df0fe5747aceeed91d6253d8e2db93  gowitness-3.0.3-darwin-arm64
c32914a24b777178ed5068d6d10b15dc4e410a5c  gowitness-3.0.3-linux-amd64
d1317177f94c9823b7c948c3a358e7ddbe7ea497  gowitness-3.0.3-linux-arm
c5813ef4fbe4417563c28e24ccf0e4c1b222b8c8  gowitness-3.0.3-linux-arm64
9c1b4d193d3d217446458fb2abc667f812be37a8  gowitness-3.0.3-windows-amd64.exe
ff6078602f02f89b48b3a875404d575f7874e4d3  gowitness-3.0.3-windows-arm64.exe

3.0.2 - File scanner fixes

fixed

• Allow file parser to parse <host>:<port> targets (66eec62)

Full Changelog: 3.0.1...3.0.2

9e13af78bda5322f9f55270fd47b827bdb5f7c2c  gowitness-3.0.2-darwin-amd64
3aeb745f9d8844c221bcba18a10fc6711390b596  gowitness-3.0.2-darwin-arm64
694a150209f71aaa37827be735152689f8d61e8d  gowitness-3.0.2-linux-amd64
155e13bb93aa22f9de8808046b7e2f3c3e8cf9c6  gowitness-3.0.2-linux-arm
5239ec4810b0eeff5ee2b49356e06aabd1eed2ba  gowitness-3.0.2-linux-arm64
c7a93e49fa442301b80ae6cf0ef0f181d4c764da  gowitness-3.0.2-windows-amd64.exe
18de5172ed5ac779e22b613afdb4f89205c72249  gowitness-3.0.2-windows-arm64.exe

3.0.1 - Bug fixes

new

• add /submit/single api endpoint for immediate screenshot results without invoking writers (c4478e2)
• add a link to the swagger api documentation in the ui (f5fd917)

fixes

• dont send completely failed results to writers (5780809)
• handle results without tls information better in the ui (fc06051)
• fix handling of ports specifications in the file reader (and add ipv6 support while im here) (0d3cd8d)
• dont check for file existence when using stdin (d3fbac4)

Full Changelog: 3.0.0...3.0.1

v3.0.0 - The *major* refactor release!

gowitness v3

Gosh, so much work went into this release, I really don't know where to start. v3 has been a while coming, and after I needed some changes for an internal tool, I figured now was as good a time as any to finally fix the stuff that had been bugging me. So, I branched to v3, deleted everything but the README and .gitignore files, and scaffolded a new cobra project.

I had lots of ideas, and a fresh start helped me finally get those in. A lot has changed, been reworked, refactored, upgraded, and more. I had fun building this, and hope you have fun using it!

If this were your typical mobile application, the release notes would have just read "Bug fixes and improvements". While thats true, it's not all there is to say.

overview

If I had to give a TL;DR of what changed, I'd summarise it as:

• Reworked the CLI. Commands are now properly categorised into subcommands. Flags also properly inherit from their parents now.
• Refactored the scanning and screenshotting logic. Most notably, the old preflight logic is removed. It was nice and fast, but when it mattered, it was a huge pain to deal with and came at the cost of result accuracy. Instead, in v3, results are now grabbed from network events like Network.responseReceived.
• Introduced the concept of "drivers," where chromedp was the original (and still default) driver. However, rod is also a driver option now that you can choose using a command-line flag.
• Significantly improved and fixed code quality, concurrency-related issues, and general screenshot reliability.
• Rewrote the report web server frontend in React. It just looks so much better, has significantly more features, and is easier to change now.
• Added an official API, complete with code-generated Swagger documentation!
• Introduced the concept of "writers," which can be used simultaneously. For example, you could write results to an SQLite database, JSON Lines, and stdout all at the same time.
• Fixed perception hashing to use Hamming Distance for grouping.

There's a lot more that's changed, so if you're curious about that and want a bit of story time, feel free to continue reading.

cgo and SQLite

SQLite has been a (required) storage mechanism since version 2. I experimented with using buntdb in version 1, but that didn't last long. It's much nicer using a format other programs could easily build on top of.

Anyway, the problem with SQLite and Golang is that the drivers often compile against the SQLite C headers. This makes cross-compilation in Go harder than it needs to be, where you need a build environment with CGO_ENABLED=1. To deal with this in version 2, I used the Elastic golang-crossbuild Docker images to target different operating systems and architectures for releases.

Thankfully, I have since discovered a pure Go SQLite implementation that comes at an immaterial performance cost! That means no more CGO_ENABLED=1, and easier/faster builds.

architecture

Every day is a school day, and I've learned a lot about Go in the years since gowitness was first released. Like any library in the Go ecosystem, if it's well-structured, anyone can technically include and use it in their own project. Unfortunately, gowitness v1 and v2 were not well-suited for this use case. I mean, you could have imported some gowitness code, but you really shouldn't have! :D

For version 3, I chose to adopt some of the project structure as described in this project-layout project. What's neat about this (apart from learning) is that now, with version 3, you should be able to import and use gowitness as a library in your own project.

Apart from the overall project structure, gowitness also underwent a significant restructuring of the codebase. There used to be one chrome package that mostly had all the important bits. Now, there are a few new concepts which include:

• Drivers: Effectively the libraries that drive Google Chrome using CDP.
• Readers: Functions that read from various sources (files, nmap, nessus, etc.)
• Writers: Functions that write driver results somewhere (SQLite, JSON Lines, stdout, etc.)
• Runner: A core component that "runs" drivers, reading from readers and writing to writers.

These all make up the internals of how an end-to-end probe of a remote website will happen. All of the scan commands use this runner pattern, including the web interface's "New Probe" feature.

scanner Drivers

Version 1 was basically the graduation of a bash script. It literally spawned a shell to run Chrome with the --headless flag. What's even funnier is I started a small https-to-http proxy in a temporary goroutine to get around TLS-related errors too. Good times. It was scruffy, but in the places where I needed it (including that one project where this started), I got screenshots faster and more reliably than I'd ever been able to before!

In version 2 though, I learned about the Chrome DevTools Protocol (CDP) and discovered chromedp as a wrapper for Golang, making it possible to drive Google Chrome without handling all of the shell execution. Sure, you still need to launch Google Chrome, but the library took care of that. I also then learned about --ignore-certificate-errors, which meant the death of my crappy proxy. Overall, using chromedp to do the heavy lifting worked out great. There were (still are?) definitely some bugs in v2, but the most painful issue was the "preflighter" concept that I used. Basically, instead of having Chrome browse to a URL that would inevitably fail (and thereby waste time), I had a simple (and cheap) Golang http.Client perform a so-called preflight to determine if a URL was up. If it was, we'd continue to let Chrome browse there and take a screenshot. The problem though was that the preflighter would often fail (turns out, behaving like a browser can be hard). This meant that results would be incorrectly missed, and that wasn't great.

So, for version 3, I removed the preflighter. Now, Chrome is used for all requests, and results are recorded based on events emitted by CDP. This meant a significant increase in probing accuracy at an acceptable performance cost. Better, but not yet amazing. While spending time on this I discovered rod. Excitedly, I read the documentation and began experimenting. It was... very fast. Like, almost unbelievably fast. At this point, I decided it was time to ditch chromedp and move to rod. It's clearly the winner.

Unfortunately, that win did not last long. I used a cleaned-up version of the Tranco list (removing obvious porn, other NSFW content, etc.) to test scanning with, and found that most websites probed fine from a network perspective, but the screenshots would often fail for no obvious reason. More often than not, actually, and that began my deep dive into why. I'll give you the short version of a few late nights. It seems like if you use a single browser process with tabs, screenshots fail often. Use a fresh browser for each screenshot, and your accuracy goes up significantly. I honestly don't know why. Talking to @singe about this, he asked about per-process ulimits and whatnot, and maybe that could be it. Regardless, because of all the testing I was doing, switching between chromedp and go-rod, I ended up implementing what I'm now calling "drivers". With gowitness v3, you can use gorod as the scanning driver by setting --driver gorod, with chromedp being the default still.

The question though is, what's the difference then? In both driver cases, when the tab strategy is used for screenshots, I'd reliably get poor screenshot accuracy. So, I changed chromedp to spawn a new browser window for each target and kept go-rod using a tabbed strategy. This means, for accuracy use the default driver, chromedp, but for speed (or if resource usage is an issue), use gorod.

readers

Instead of defining how source data should be read in the commands themselves, for version 3, I built the concept of readers which makes them easier to maintain and ultimately reusable and extendable! Readers all implement the same interface, which looks like this at the time of writing:

type Reader interface {
 Read(chan<- string) error
}

The Read method will receive a channel that accepts strings. This means that any reader, regardless of how it sources candidate URLs (i.e., a database, a file, a CIDR parser, Nessus, etc.) should ultimately write a string candidate (as a full, well-formed URL) to the channel. It's a simple concept but powerful when combined with a runner. When a runner is created, it prepares a Targets channel that you can write to. Combining them means we'll have something like this (using the file reader as an exam...

2.5.1

What's Changed

• html a-tag unclosed bug fixed by @gorgiaxx in #187
• Fix database file path location parser by @mr-pmillz in #188
• fix: typo by @testwill in #201
• (fix) MarshalJSON spells with one "l" by @alexandear in #198
• Fixing the missing port issue with Nessus by @quentinpraz in #191

New Contributors

• @gorgiaxx made their first contribution in #187
• @mr-pmillz made their first contribution in #188
• @testwill made their first contribution in #201
• @alexandear made their first contribution in #198
• @quentinpraz made their first contribution in #191

Full Changelog: 2.5.0...2.5.1

1edfe2209731e68621006b3cc0376d6a4b97e85e  gowitness-2.5.1-darwin-amd64
3c817d57d704a3de1dcb084a59b77e684dddb154  gowitness-2.5.1-darwin-arm64
19c75086ac90ae2891f24fa89d76c84748eb3d06  gowitness-2.5.1-linux-amd64
105ecb560afccd5f29fe1d748c32862848f556df  gowitness-2.5.1-linux-arm64
e893957658f560911864eeecec606c0cd9ea8e05  gowitness-2.5.1-linux-armv7
32315a0b50ccd4e28b941a4b0121200b6625eb9e  gowitness-2.5.1-windows-amd64.exe

2.5.0

new

• Make URLs clickable in static report exports (thanks @initstring) (via #172)
• Add ability to execute externally introduced JavaScript via the new --js flag on screenshotted pages. (thanks @djallalzoldik) (via #180)
• Add PostgreSQL support (thanks @habitualdev) (via #166). You should now specify the database to use using a full URI. Eg: sqlite://gowitness.sqlite3 (the default), or postgres://user:pass@host/database.
• Add search API endpoint (docs: search (thanks @habitualdev) (via #166)
• Add ability to store screenshots in the database using a new flag --screenshot-db-store (thanks @habitualdev) (originally via #166 but refactored in 62d6de3). The report viewer will automatically fallback to the filesystem if database screenshots are not available.
• Add HTTP response code filtering, controlling which HTTP codes get screenshotted. (thanks @nickspring) (originally via #137 but implemented in 1503c5b)

fixes

• Fix report server range selector. (thanks @randomactsofsecurity ) (via #163)
• Fix example nessus file syntax (thanks @brettgus) (via #170)
• Prevent duplicate file extensions added to filenames (thanks @maik-s) (via #183)

other

• Bump dependencies and Go version (in 4907cea, 921811d, a7941d8)
• Remove deprecated use of io/ioutil (in f939dec)

new contributors

• @brettgus made their first contribution in #170
• @initstring made their first contribution in #172
• @maik-s made their first contribution in #183
• @djallalzoldik made their first contribution in #180
• @habitualdev made their first contribution in #166

Full Changelog: 2.4.2...2.5.0

72f9578e558527bd5e8c6212d5e63b18867cd0b7  gowitness-2.5.0-darwin-amd64
d3fa213e6e0c8543256e26c1c3b3b71f23175485  gowitness-2.5.0-darwin-arm64
5f76bc689612b0b1ca5266834c76cb0c55a120b8  gowitness-2.5.0-linux-amd64
9c567241d9202689d395e704735fd8f3c1a47cfb  gowitness-2.5.0-linux-arm64
c8005cc40c8a7a9c1c690737e14853c829263acd  gowitness-2.5.0-linux-armv7
d906179afa9b59950a3ded496f9c0ede260c2cb7  gowitness-2.5.0-windows-amd64.exe

2.4.2

fixes

• Improve web UI reverse proxy support when served under a sub directory. A new wiki article was also added (in e904933) (thanks @random-robbie for PR's prompting this).

other

• Bump dependencies and build with Go 1.19 (in 3b2aceb)

8577bca1f581d7f163144b5c9068861fcf401524  gowitness-2.4.2-darwin-amd64
64b7469d97a511650f1efb74499878ad3ef8e76d  gowitness-2.4.2-darwin-arm64
e98b223ae71ef7a8df75ba0a6461b6b772a2176e  gowitness-2.4.2-linux-amd64
18556f2b0b856d90865ddae653c2e05182541e37  gowitness-2.4.2-linux-arm64
adb157143d84ff697c0cf965df3ce04ae34a5a48  gowitness-2.4.2-linux-armv7
6de652cf5ddf8f4f541f6172f127d23747353097  gowitness-2.4.2-windows-amd64.exe

2.4.1

new

• Add ability to specify additional headers when screenshotting via the HTTP api. See the API docs for a usage example (in f5e2aea).
• Add linux/arm and linux/armv7 targets to release binaries (in 12f01a9)

fixes

• Fix nmap command example documentation. (thanks @crypt0rr ) (via #138)

other

• Build release binaries with Go 1.18.4 (in 31c6e64)
• Bump dependencies (in 2041445)
• Fix internal flag name typo (in 86134ed)

8c5dfc2b7f5a66aec4d861522c2d78452e1950ac  gowitness-2.4.1-darwin-amd64
bd4026cad944b6143fbad90b0a9f2e41671509f6  gowitness-2.4.1-darwin-arm64
82c56c41caf8e1474adc851a90df48cb5b7c9ee1  gowitness-2.4.1-linux-amd64
b6391b19c6b5316e74c18f8c05dcec1987186170  gowitness-2.4.1-linux-arm64
9f79dfa11aaf6788a1150da41c12da03185e26af  gowitness-2.4.1-linux-armv7
5ec37979e14290200f0d0919006102cefa89edd2  gowitness-2.4.1-windows-amd64.exe