Token API

[martinsumner]

Feature request to have a Token API:

GET tokens/<TokenID>?type=request&request_timeout=<RequestTimeoutS>&token_timeout=<TokenTimeoutS>
GET tokens/<TokenID>?type=renew&session=<EncodedSessionReference>
GET tokens/<TokenID>?type=release&session=<EncodedSessionReference>

Whereby only one token session can be active in the cluster for any one token ID at the same time

Whereby an additional header can be provided to a Riak API request

X-Riak-Session: <EncodedSessionReference>

Such that the request will only be processed if the session (granted as a result of a request to the token API) is currently active.

[martinsumner]

The implementation is as follows.

Standard Path

Each riak_kv node should have a riak_kv_token_manager. This will grant tokens on request, if no token is currently granted. If a token is currently granted it will queue the request until that grant is released.

It is possible to start in riak a riak_kv_token_session. This is a process, with a riak_client from which riak_client function calls can be made. A session will become active if and only if it can be granted a token by the local riak_kv_token_manager.

A token request can be issued for a tokenID, by starting a riak_kv_token_session. A TokenID can be a specific token {token, TokenID} or a standard object {Bucket, Key} tuple.

The riak_kv_token_session must start on a specific node given the current state of the cluster. A token request will calculate an NVal 5 primary preflist for TokenID (which will be the same as the preflist for the object, of an object key is used). If at least 3 of the nodes in the preflist are up, a riak_kv_token_session process will be spawned on the node at the head of the preflist. This session, will then seek to have a token granted by its local riak_kv_token_manager, passing the TokenID, the next two up-nodes in the preflist and a request timeout.

The riak_kv_token_manager will grant a token if:

• No other such token has granted (or any such token is released and this request becomes the next request in the queue for that TokenID), and;
• The riak_kv_token_manager can confirm via a remote call that next two nodes' riak_kv_token_manager in the Preflist have not currently granted a token (to safely handle node recovery).

On granting a token, or queueing a request, the riak_kv_token_manager will monitor the riak_kv_token_session process.

Once a token has been granted to a riak_kv_token_session the process which started the riak_kv_token_session will be returned a reference for that session. That reference can be used for forwarding riak_client requests to the session.

An inactivity timeout will be triggered on the session process. If the riak_kv_token_session terminates for any reason (including inactivity), the riak_kv_token_manager will clear any grant (or queued request) associated with the session (having detected the 'DOWN' message).

Any riak_client request may be forwarded through the riak_kv_token_session (by a process aware of the riak_kv_token_session session reference) from any node in the cluster, and succeed if and only if the grant is still active, and this will also count as activity against the grant (resetting the inactivity timeout on the session).

On completing the session, a release message SHOULD be sent to the riak_kv_token_session, closing the process and allowing for the token grant to be removed (and the next queued request to be granted).

If the session needs to be prolonged, despite being inactive, a renew message SHOULD be sent to the riak_kv_token_session before the inactivity timeout is triggered.

Failure Scenarios

Should the riak_kv_token_manager crash, all riak_kv_token_session process will be monitoring the riak_kv_token_manager that granted their token, and will crash on receipt of the 'DOWN' message.

Should the node which is currently head of the preflist stop or crash, subsequent requests will see a new head node for that tokenID. As all sessions associated with the crashed node will have been killed as part of the crash, this new head needs no prior knowledge of any previously granted tokens before granting tokens in its new role.

Should a node recover from a crash, there may have been a token granted by a node downstream in the preflist - hence why the two downstream nodes are checked before granting or queueing a request.

Extreme Edge Cases (non-exhaustive)

• The implementation assumes that target_n_val >= 5, and target_location_nval >= 3 - so that the token granting will continue to function should two nodes fail, wither because two have failed independently, or a location failure has brought down two nodes concurrently.
• In a partitioned cluster, only one side of the partition will have sufficient primary nodes to issue tokens - the solution is not partition tolerant.
• If there are inconsistent view of availability of nodes within the cluster, the guarantees may not be met - it is recommended that clusters be configured with prevent_overlapping_partitions = true.
• If a riak_kv_token_manager crashes and restarts, and a riak_kv_token_session process has an in-flight request that continues throughout the restart, then following the restart a concurrent grant made be made (ss the 'DOWN' message may not be received while the riak_kv_token_session is processing a client function).

[martinsumner]

https://github.com/nhs-riak/riak_kv/tree/nhse-d34-nhskv.i33-token

[martinsumner]

The initial implementation is to provide stronger conditional PUT by using the token mechanism within Riak to wrap the GET/PUT process in a conditional PUT in a single session.

[martinsumner]

Initial test - https://github.com/nhs-riak/riak_test/blob/nhse-d34-nhskv.i33-token/tests/verify_conditionalput_strong.erl