Single-use token used to create a snapshot reappears in the restored database and is irrevocable

[mossblaser]

Describe the bug
If a single use (e.g. -use-limit=1) token is used to create a raft snapshot, the expired token reappears when the snapshot is restored. The expired token shows a num_uses of -1 and appears to not be usable. The restored token also appears to be irrevocable and in some kind of stuck state.

Because the reappearing token (appears) not to be usable, this doesn't present an immediate security problem. It does, however, seem to be leading to some kind of "impossible" database state which might be concerning from the perspective of database integrity. It also complicates auditing of tokens in a live system as these "stuck and unusable" tokens must be manually eliminated from any checks.

To Reproduce
Steps to reproduce the behavior:

1. Create a vault cluster using raft based storage.

Sample minimal vault config (click to expand)

Setup a quick-and-dirty vault server:

$ cat << EOF > config.hcl
   storage "raft" {
     path = "data"
     node_id = "vault"
   }
   listener "tcp" {
     address = "127.0.0.1:8200"
     cluster_address = "127.0.0.1:8201"
     tls_disable = true
   }
   api_addr = "http://127.0.0.1:8200"
   cluster_addr = "https://127.0.0.1:8201"
   disable_mlock = true
EOF
$ mkdir data
$ vault server -config config.hcl

Initialise, unseal and authenticate:

$ export VAULT_ADDR="http://127.0.0.1:8200"
$ vault operator init -key-shares=1 -key-threshold=1
$ vault operator unseal <unseal key printed during init>
$ vault login <root token printed during init>

2. Create a single-use token with the ability to hit the snapshot endpoint. For example:

$ vault token create -policy=root -use-limit=1
Key                  Value
---                  -----
token                hvs.qA2zCbxgisRVnnPaS1KpIcTJ
token_accessor       LF7kIL16LczrVisluHZSSdVJ
...

3. See that the token exists:

$ vault token lookup -accessor LF7kIL16LczrVisluHZSSdVJ
Key                 Value
---                 -----
accessor            LF7kIL16LczrVisluHZSSdVJ
...

4. Create snapshot using the single-use token:

$ VAULT_TOKEN=hvs.qA2zCbxgisRVnnPaS1KpIcTJ vault operator raft snapshot save raft.snap

5. Verify that token has expired:

$ vault token lookup -accessor LF7kIL16LczrVisluHZSSdVJ
Error looking up token: Error making API request.

URL: POST http://127.0.0.1:8200/v1/auth/token/lookup-accessor
Code: 400. Errors:

* 1 error occurred:
        * invalid accessor

6. Restore snapshot:

$ operator raft snapshot restore raft.snap

7. Observe that the expired token now exists again (though it isn't usable as num_uses is -1):

$ vault token lookup -accessor LF7kIL16LczrVisluHZSSdVJ
Key                 Value
---                 -----
accessor            LF7kIL16LczrVisluHZSSdVJ
creation_time       1726147521
creation_ttl        0s
display_name        token
entity_id           n/a
expire_time         <nil>
explicit_max_ttl    0s
id                  n/a
issue_time          2024-09-12T14:25:21.445948718+01:00
meta                <nil>
num_uses            -1
orphan              false
path                auth/token/create
policies            [root]
renewable           false
ttl                 0s
type                service
$ VAULT_TOKEN=hvs.qA2zCbxgisRVnnPaS1KpIcTJ vault token lookup
Error looking up token: Error making API request.

URL: GET http://127.0.0.1:8200/v1/auth/token/lookup-self
Code: 403. Errors:

* permission denied

8. Also observe that the token cannot be revoked.

$ vault token revoke -accessor LF7kIL16LczrVisluHZSSdVJ
Error revoking token: Error making API request.

URL: POST http://127.0.0.1:8200/v1/auth/token/revoke-accessor
Code: 400. Errors:

* token not found
$ vault token lookup -accessor LF7kIL16LczrVisluHZSSdVJ
Key                 Value
---                 -----
accessor            LF7kIL16LczrVisluHZSSdVJ
...

Expected behavior

I would expect that a single-use token used to create a snapshot would not appear in the backup (I would expect that it would have been deleted as part of the checking process before the snapshot started).

I would also expect to be able to revoke the back-from-the-dead token after restoring a snapshot (even if it is in an unusable state).

Extra detail: time and use limited tokens

In case it is useful, I have observed that in the case of a single-use token also having a finite TTL, the stuck token in the restored database will eventually be removed when the TTL expires.

1. Create token with single-use limit and a short TTL:

$ vault token create -policy=root -use-limit=1 -ttl=5m
Key                  Value
---                  -----
token                hvs.zR19ilqFOKcs8YIoiu8mPEXQ
token_accessor       CByRQVJ2yDFpeRLk1Rq7Kahl
token_duration       5m
token_renewable      true
token_policies       ["root"]
identity_policies    []
policies             ["root"]

2. Create snapshot using the token and then restore from that snapshot.

$ VAULT_TOKEN=hvs.zR19ilqFOKcs8YIoiu8mPEXQ vault operator raft snapshot save raft.snap
$ vault operator raft snapshot restore raft.snap

3. Observe that the token comes back to life but is unusable and cannot be revoked, as before.

$ vault token lookup -accessor CByRQVJ2yDFpeRLk1Rq7Kahl
Key                 Value
---                 -----
accessor            CByRQVJ2yDFpeRLk1Rq7Kahl
creation_time       1726149125
creation_ttl        5m
display_name        token
entity_id           n/a
expire_time         2024-09-12T14:57:05.133931656+01:00
explicit_max_ttl    0s
id                  n/a
issue_time          2024-09-12T14:52:05.13393712+01:00
meta                <nil>
num_uses            -1
orphan              false
path                auth/token/create
policies            [root]
renewable           true
ttl                 4m8s
type                service
$ vault token revoke -accessor CByRQVJ2yDFpeRLk1Rq7Kahl
Error revoking token: Error making API request.

URL: POST http://127.0.0.1:8200/v1/auth/token/revoke-accessor
Code: 400. Errors:

* token not found
$ vault token lookup -accessor CByRQVJ2yDFpeRLk1Rq7Kahl
Key                 Value
---                 -----
accessor            CByRQVJ2yDFpeRLk1Rq7Kahl
...

4. Wait for TTL to expire and observe that the token is deleted as expected.

$ vault token lookup -accessor CByRQVJ2yDFpeRLk1Rq7Kahl

5. If restored again after the TTL has expired, notice that the expired token is also briefly present (though unusable) for a few seconds before being cleaned up.

$ vault operator raft snapshot restore raft.snap
$ vault token lookup -accessor CByRQVJ2yDFpeRLk1Rq7Kahl
Key                 Value
---                 -----
accessor            CByRQVJ2yDFpeRLk1Rq7Kahl
creation_time       1726149125
creation_ttl        5m
display_name        token
entity_id           n/a
expire_time         2024-09-12T14:57:05.133931656+01:00
explicit_max_ttl    0s
id                  n/a
issue_time          2024-09-12T14:52:05.13393712+01:00
meta                <nil>
num_uses            -1
orphan              false
path                auth/token/create
policies            [root]
renewable           false
ttl                 -50s
type                service
$ sleep 5
$ vault token lookup -accessor CByRQVJ2yDFpeRLk1Rq7Kahl
Error looking up token: Error making API request.

URL: POST http://127.0.0.1:8200/v1/auth/token/lookup-accessor
Code: 400. Errors:

* 1 error occurred:
        * invalid accessor

Environment:

Vault v1.16.3 (e92d9a5), built 2024-05-29T14:28:42Z

Ubuntu 22.04 x86_64