-
Notifications
You must be signed in to change notification settings - Fork 9
Denial of service caused by invalid windowBits parameter passed to zlib.createDeflateRaw()
Affected components:
- permessage-deflate-node <= 0.1.5
- websocket-extensions-node <= 0.1.1
Potentially affected:
- permessage-deflate-ruby <= 0.1.3
The permessage-deflate-node package contains a vulnerability where user input can be used to crash the server. If an incoming WebSocket request contains the header
Sec-WebSocket-Extensions: permessage-deflate; server_max_window_bits=8
Then the package will try to initialize a zlib raw deflate stream with
windowBits set to 8, which under recent versions of zlib packaged with
Node.js will throw an exception. This exception is not caught either in the
permessage-deflate package or in the websocket-extensions-node
container, and so if the user does not have an uncaughtException handler
installed, the process will immediately crash. If such a handler is installed,
the process won't crash but messages sent from the application to the client may
remain in a buffer inside websocket-extensions-node rather than being sent to
the client. This will cause the application to be unresponsive and eventually
run out of memory.
In addition, users of permessage-deflate-ruby may be affected if their Ruby is dynamically linked to a copy of zlib that contains the relevant changes.
The recommended mitigation is to install the following packages or to remove permessage-deflate from your WebSocket deployment.
- For Node.js projects:
- permessage-deflate 0.1.6
- websocket-extensions 0.1.2
- For Ruby projects:
- permessage_deflate 0.1.4
16 August 2017: I (jcoglan) discovered that
zlib.createDeflateRaw({windowBits: 8}) caused an error while I was testing the
Faye collection of WebSocket libraries using Autobahn. I
reported nodejs/node#14847.
17 August 2017: The following morning, I realised this error meant that user
input could be used to crash a Node WebSocket server, and made an
announcement on the Faye mailing list alerting users to install
an uncaughtException handler or remove the permessage-deflate-node from their
system.
18 August 2017: As a result of nodejs/node#14847, MylesBorins opened madler/zlib#291.
Over the following few weeks Myles and I discussed how to solve the issue and I worked on fixing the issue in permessage-deflate. The following are my notes from that process.
10 September 2017: I released the aforementioned packages to mitigate the issue.
On 4 April 2017, Node.js versions v4.8.2 and v6.10.2 were
released. These versions bumped the vendored zlib library from v1.2.8 to v1.2.11
in response to what it describes as low-severity CVEs. In zlib v1.2.9
(released 31 December 2016), a change was made that causes an error to
be raised when a raw deflate stream is initialised with windowBits set to 8.
The windowBits parameter controls how much of a message zlib keeps in memory
while compressing it. A larger window, as it's called, means more
opportunities to spot and compress repeated bits of text, but results in higher
memory usage. windowBits is the base-2 logarithm of the size of this window in
bytes, and previously could take any integer value from 8 to 15.
In zlib v1.2.9, you can no longer pass 8 for this parameter when creating what's called a raw deflate stream, and Node's zlib module will throw an error if you call this:
zlib.createDeflateRaw({windowBits: 8})On some versions of Node, this crashes the process and you cannot recover from it, while on some versions it throws an exception. The permessage-deflate-node library up to version v0.1.5 does make such a call with no try/catch when, for example, it is running on the server and the client sends this request:
GET / HTTP/1.1
Host: www.example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dmZRc9eli98SUFIl8QKn4w==
Sec-WebSocket-Extensions: permessage-deflate; server_max_window_bits=8
Sec-WebSocket-Version: 13
On receiving this request, permessage-deflate-node running on Node v4.8.2+,
v6.10.2+, or any v8.0+ release will accept the request and respond with a header
including server_max_window_bits=8 to indicate it will comply with the
requested settings, but when it first tries to compress a WebSocket message it
will create a raw deflate stream using zlib.createDeflateRaw(), fail to handle
the exception and crash unless the user has an uncaughtException handler
installed. This will result either in the user's server being vulnerable to a
denial-of-service attack, or to that single WebSocket connection failing if an
uncaughtException handler exists. That failed connection will likely enter a
state where messages sent by the application are never written out to the
socket, and they will build up in a queue in
websocket-extensions-node.
According to RFC 7692, section 7.1.2.1:
By including this parameter in an extension negotiation offer, a client limits the LZ77 sliding window size that the server will use to compress messages. If the peer server uses a small LZ77 sliding window to compress messages, the client can reduce the memory needed for the LZ77 sliding window.
A server accepts an extension negotiation offer with this parameter by including the
server_max_window_bitsextension parameter in the extension negotiation response to send back to the client with the same or smaller value as the offer. Theserver_max_window_bitsextension parameter in an extension negotiation response has a decimal integer value without leading zeroes between 8 to 15, inclusive
So, when accepting a request with server_max_window_bits set, the server
should respond with an equal or smaller value for this parameter. The idea is
that the server is promising to limit its deflate window size to that requested
by the client, allowing the client to save memory when decompressing. Similarly,
if the server's response includes client_max_window_bits=N, the client must
use at most an N-bit window size to compress its messages, even if the client
sent a larger version for client_max_window_bits in its request.
permessage-deflate-node does validate the parameters in these requests and limits the acceptable values to the integers from 8 to 15. However, if the value 8 is no longer acceptable when creating a raw deflate stream to compress a message, the server might initially agree to use an 8-bit window size during the negotiation, but then fail when creating a deflate context when it comes time to actually start compressing messages.
Some insight into what's going on with zlib here is available in the aforementioned commit and in madler/zlib#171. There we discover that 256-byte (8-bit) windows have not worked when compressing for some time (I do not know since when), and that zlib has actually been replacing 8 with 9. We can see this if we try compressing the same message with different window sizes:
var zlib = require('zlib');
function compressMessage(stream, message) {
var chunks = [];
stream.on('data', d => chunks.push(d));
stream.write(message);
stream.flush(() => console.log(Buffer.concat(chunks)));
}
var message = 'hello world';
[8, 9, 10, 11].forEach(bits => {
var stream = zlib.createDeflate({windowBits: bits});
compressMessage(stream, message);
});
/*
-> <Buffer 18 95 ca 48 cd c9 c9 57 28 cf 2f ca 49 01 00 00 00 ff ff>
<Buffer 18 95 ca 48 cd c9 c9 57 28 cf 2f ca 49 01 00 00 00 ff ff>
<Buffer 28 91 ca 48 cd c9 c9 57 28 cf 2f ca 49 01 00 00 00 ff ff>
<Buffer 38 8d ca 48 cd c9 c9 57 28 cf 2f ca 49 01 00 00 00 ff ff>
*/This message is very short so it compresses to the same thing given any window size, but the important thing here is the start of the messages. From the issue thread:
Nothing should be broken, since an 8-bit window zlib stream is never generated by zlib. If 8 is requested, a 9-bit window is used instead.
In addition to compressing with a 512-byte window, 9 bits is put in the zlib header when 8 bits is requested. So zlib, or any compliant decompressor reading that header, will use at least a 512-byte window and there will be no error on decompression.
So zlib sticks a header at the front of the result that indicates the
compression parameters. This header is the first two bytes of those messages;
it's 18 95 when windowBits is 8 or 9, 28 91 when it's 10, and 38 8d when
it's 11. It's the same for 8 and 9, because when windowBits is 8 zlib just
pretends you meant 9. Even if one party intends to use a 8-bit window size, zlib
uses a 9-bit window and includes information in the stream to tell the other party
that's what's happening.
However, WebSocket permessage-deflate uses what's called raw deflate streams,
which omit these two header bytes. The messages would all begin ca 48 cd ...
if sent using WebSocket. It's for this reason that zlib has removed the ability
to initialize raw deflate streams with windowBits set to 8; from the
commit:
There is a bug in deflate for windowBits == 8 (256-byte window). As a result, zlib silently changes a request for 8 to a request for 9 (512-byte window), and sets the zlib header accordingly so that the decompressor knows to use a 512-byte window. However if deflateInit2() is used for raw deflate or gzip streams, then there is no indication that the request was not honored, and the application might assume that it can use a 256-byte window when decompressing. This commit returns an error if the user requests a 256-byte window when using raw deflate or gzip encoding.
The situation this tries to prevent is: Alice wants to send Bob a message, and they agreed beforehand to use an 8-bit window size -- the message itself does not contain that information. Alice asks zlib for a raw deflate stream with an 8-bit window size, but zlib secretly uses a 9-bit one. She sends the message to Bob, who only allocates an 8-bit window size for his inflate stream, and fails to read Alice's message. Since this is a parsing/interpretation problem it could cause security issues; the problem is that this fix also causes potential denial-of-service in Node programs.
However in the case of WebSocket there is another mechanism for sending the
metadata about the window size: the Sec-WebSocket-Extensions header
parameters. So the argument that Bob will not realise what window size Alice is
using does not hold, so long as Alice does not lie. It is important to note that
zlib only replaces 8 with 9 in when compressing a message in deflate.c, not
when decompressing it in inflate.c, so Bob needs to know Alice is actually
using a 9-bit window size.
In deciding how to mitigate this issue, we should take into account what the
spec says about the use of windowBits in permessage-deflate and what happens
when it is wrong. This information is based on RFC 7692 § 7.1.2.
-
A server receiving a request including
server_max_window_bits=Nis being told by the client to use at most N bits to compress messages, so the client can save memory. It should reply withserver_max_window_bits=Mwhere M <= N, and use a window size <= M to compress messages. If it cannot do so, it should reject the extension offer, meaning the WebSocket connection proceeds, but without compression. -
A client receiving a response including
server_max_window_bits=Nis being advised by the server that the server will use at most an N-bit window size to compress messages, and so the client can save memory by allocating an N-bit window for decompression. If the client does not support changing the window size it can simply ignore this parameter, as long as it allocates at least an N-bit window for decompression. If the client cannot allocate an N-bit window size then it should fail the WebSocket connection. If the server is lying and will actually use more than N bits, the client may fail to decompress messages properly. -
A server receiving a request including
client_max_window_bits=Nis being advised by the client that the client intends to use at most an N-bit window size to compress messages. As in the previous case, it can ignore this setting at long as it can allocate at least an N-bit window, and it does not need to return its ownclient_max_window_bitsvalue. It should reply with its ownclient_max_window_bitsvalue only if it wants the client to use fewer than N bits for its window size and the client has indicated that it supports doing so. If the server cannot comply for any reason it should reject the extension offer, meaning the WebSocket connection proceeds without compression. -
A client receiving a response including
client_max_window_bits=Nis being told by the server to use at most N bits to compress messages, so the server can save memory. If it cannot comply with this requirement, it should fail the WebSocket connection. If it proceeds and uses more than N bits for its window size, it may send data to the server that the server cannot then decompress.
We use the Autobahn test suite to verify our WebSocket packages' functionality. In addition to the spec, this suite is the other major indicator for compatibility that most implementations use. I ran a series of tests with Autobahn to discover which combinations of behaviour are acceptable, regardless of what the spec says.
We'll discuss the server first. The handshake works by the client sending an
HTTP request to the server, containing its desired compression parameters, and
the server then responds with the parameters it is agreeing to use. There is no
client-side acknowledgement of the server's parameters. In testing, a few
interesting results emerged in cases where the server received a request
containing server_max_window_bits=8:
- The server may respond with
server_max_window_bits=8but actually compress messages withwindowBits: 9, and the tests will pass. - If the server responds with
server_max_window_bits=8but uses any value greater than 9 for compression, the tests fail. - If the server responds with
server_max_window_bits=Nand uses any value greater than N, the tests fail. - The server may respond with
server_max_window_bits=Nwhere N is greater than 8, and use that value N to compress, and the tests will pass. The client still recognises this as acceptance of thepermessage-deflateextension offer and compresses its messages to the server. - The server may respond with no
server_max_window_bitsparameter, and usewindowBits: 15, and the tests will pass.
To summarise, Autobahn deviates from the spec and allows the server to respond
with a server_max_window_bits value greater than that requested by the client,
and the client will still activate the compression extension. The server may use
a larger compression window than requested by the client, as long as it does not
use a larger window than its own server_max_window_bits response suggests.
This includes the option for it to not send any server_max_window_bits value
at all and use the maximum window size. The one exception to this is it can
reply with server_max_window_bits=8 but actually use windowBits: 9.
However, on the client side things are more strict. Remember the handshake is
two-phase; the client speaks first and then the server. The client has no
opportunity to respond to the server's stated parameters. In testing, I observed
that the client must abide by the client_max_window_bits value sent by the
server; even if the client sends its own client_max_window_bits value to
advertise that it plans to use a larger window, if it uses a size larger than
it's told by the server, the tests fail. The only exception to this is that the
client may use windowBits: 9 if the server sends client_max_window_bits=8.
As I see it, this leaves three options:
We could just decide 8 is no longer an acceptable value for
server_max_window_bits or client_max_window_bits and reject requests using
those. On the server side this would mean accepting the WebSocket connection but
not agreeing to use the extension, while on the client side it would result in
failing the connection. This would be in violation of the RFC, and on the client
side would render some previously working clients inoperable. But this is
already the case for some implementations that don't support changing the window
size either at all or through the range supported by zlib. For example, I also
maintain the permessage-deflate-ruby library and JRuby's zlib module
does not support changing the window size to 8.
Given that historically zlib has been claiming to allow 8-bit window sizes but
really using 9-bit sizes, and apparently nobody has noticed, we could continue
doing that. We could acknowledge a request with server_max_window_bits=8 by
echoing back the same value, in compliance with the spec, but then create a
deflate context with windowBits set to 9. In theory, this would result in the
client failing to decompress, but, to date I have not received any complaints as
a result of this and the Autobahn test suite passes when windowBits was set to
8 before zlib v1.2.9.
We could decide to respond to server_max_window_bits=8 by returning
server_max_window_bits=9 and using a 9-bit window. We are now not lying and we
are conveying the necessary information to the client just like the two-byte
zlib headers would do if WebSocket used them. The problem is that this response
is not compliant with the RFC, which requires the response to have an equal or
smaller value for server_max_window_bits.
Knowing what we know about the RFC and the Autobahn tests, I believe the best
way forward is for the server respond to server_max_window_bits=8 with the
same value in its response, but then replicate the old behaviour of zlib and use
windowBits: 9 when creating its deflate stream.
This complies with the spec in terms of the header negotiation, and the
behaviour of zlib up to v1.2.8 and the Autobahn results suggest this will not
break clients that expect to receive 8-bit compressed messages and in fact
receive 9-bit ones. A similar argument applies concerning
client_max_window_bits, particularly since Autobahn is stricter with the
client than with the server, and does not allow the client to exceed the
server's requested window size.
However, to reduce the theoretical risk that a decompressor with an 8-bit window might fail when given 9-bit compressed messages, we will also replace 8 with 9 when creating a inflate stream, which is an extension to the previous behaviour of zlib. Although behaviour thus far indicates that an 8-bit inflate stream can decompress 9-bit compressed messages, I'd rather avoid the risk and give the decompressor at least as much capacity as we know it might need.
Finally, I have made a change to websocket-extensions-node to catch synchronous errors thrown by extension plugins, whereas before it was only catching async errors. When websocket-extensions catches an error, it shuts down the processing stream in both directions cleanly and reports the error back to the protocol driver so it can perform a closing handshake and clean up its state. This should mitigate any future issues caused by extensions throwing unanticipated errors.
I will also be making the above changes to permessage-deflate-ruby. Ruby is dynamically linked to zlib, so Ruby users are typically using a zlib supplied by the operating system and therefore have less control over its version and what patches it contains. Although no issue has been reported to me in the Ruby version, I am making this change to mitigate the potential issues should anyone end up running Ruby with zlib v1.2.9 or above.
These changes are being released in:
- permessage-deflate-node v0.1.6
- permessage-deflate-ruby v0.1.4
- websocket-extensions-node v0.1.2