How do we manage the large amount of Secret information in our work? (For example, my project involves the storage of secret keys and passwords for OpenSSH, as well as the regular rotation and external invocation of this)

  • Solidified in a configuration file, stored in a server file or Database
  • stored as code on a git private repository, with strict access rights to the repository
  • Hosted on a public cloud service as a KMS (Key Management Service, mostly cloud services)
  • Challenges from the cloud: Unlocking the Cloud Operating Model (https://www.hashicorp.com/cloud-operating-model)

The generic cryptographic repository we need to implement needs to satisfy the following characteristics.

  • Cryptographic storage (taking into account complexity)
  • Generic, reducing the workload of manual Secret modification, i.e. providing RestfulAPI
  • Permission control

In this article, we would like to introduce Vault, an open source Secret management tool (password, token, private key and certificate, etc.), which is an excellent application practice for managing passwords and secret keys in code (to prevent plaintext leakage). In addition, KMS (Key Management Service, cloud services mostly) is also a better Secret management practice. vault project source code here (https://github.com/hashicorp/vault)

For such products, the following points need to be focused on.

  1. the way the Secret is stored, the supported storage backend
  2. the encryption method and algorithm of the Secret
  3. the system’s permission control, permission allocation (which people / clients can access which machine Secret)
  4. the system’s authentication method, the client uses what way to access RestfulAPI
  5. the system’s Secret storage method and expiration mechanism
  6. how to ensure the high availability of the system
  7. the QPS and concurrency performance of the system’s external interface

0x01 Vault Fundamentals

This subsection refers to the official documentation (https://www.vaultproject.io/docs/internals/architecture). The basic application scenario of vault is as follows.

Vault

Vault is generally used in the following scenarios:

  1. system users (such as operations and maintenance colleagues) write Secret data to Vault via HTTP-Vault-API, Vault command line tools, etc.
  2. Vault then stores the encrypted data to the backend
  3. external users (e.g. developers, scripts or applications) use HTTP-Vault-API, Vault command-line tools, etc. to get the Secret data associated with their own accounts only, which involves the fine-grained permissions management of Valut

The vault architecture is as follows.

vault architecture

As you can see from the architecture diagram, Vault is divided into three parts: Storage Backend, Barrier and HTTP API. The Barrier ensures that only encrypted data is written to the Storage Backend, and that encrypted data is verified and decrypted as it is read out through the Barrier.

The functions of the other major components are as follows.

  • HTTP(s) API:
  • Storage backend.
  • Token Store.
  • Auth Method.
  • Core: responsible for processing requests and response logs from the Audit brok, sending requests to all configured Audit devices
  • Policy store: responsible for managing and storing ACL Policies, with Core performing the ACL Policy checks

Vault’s data flow

Vault’s data flow

0x02 The main operation flow of Vault

Step1: Data Storage and Encryption/Decryption

Understand a few terms.

  1. Storage Backend: Vault does not store data itself, it needs to be configured with a Storage Backend. Storage Backend is not trusted and is only used to store encrypted data.

  2. Initialaztion: Vault needs to be initialized when it is first started, this step generates an Encryption key to encrypt the data, and the encrypted data can only be saved to Storage Backend.

  3. Unseal: After the Vault starts, it will enter the Sealed state because the Encryption Key is not known, and no operation can be performed until it is unsealed; the Encryption Key is protected by the Master key, and the Master key must be provided to complete the Unseal operation

The relationship between Master key and Encryption Key is shown in the following figure.

The main operation flow of Vault

Step2: Authentication && Permissions Management

After the Unseal operation is completed, the Vault can process client requests. The first time a client connects to the Vault, they need to complete authentication. Client authentication methods are as follows

  • Suitable for users: User name/password, LDAP authentication, etc. The user must be granted appropriate permissions to access the Vault.
  • Suitable for applications: Public/Private keys, Tokens or Jwt Token, etc.

This general flow is as follows.

  1. The client initiates an authentication request, which flows through the Core module and into Auth methods, which determines whether the request is valid and returns a list of associated policies (Policies). After the authentication is completed with Auth methods and the checked association policies match the authorization, the Token Store generates and manages a new Token, which is returned to the client for subsequent requests. Note that this token also has a Lease term (expiration date), and the Token is associated with a Policy that will be used to verify the request’s authority.
  2. After the request is validated, it is routed to the Secret engine module. If the Secret engine returns a Secret (automatically generated by Vault), Core registers it with the Expiration manager and appends a lease ID to it. If the client allows the lease to expire, the Expiration manager will automatically revoke the Secret Token.

Step3: Secret Engine (important)

The Secret Engine is the component of the Vault system that saves, generates or encrypts data. The Secret Engine is like a virtual file system, all read/write/delete/list operations are performed under it, and then the Secret Engine can decide for itself how to respond to requests. From the code design point of view, Secret Engine is an abstraction that provides a unified interface to the upper layer (caller), such as physical file system, database, etc., which can be used to add, delete, change and check these operations. Commonly used Engines are as follows.

  • kv: key-value storage. Can be seen as an encrypted Redis, simply store / read some static configuration / data
  • Transit Secrets Engine: Provides encryption-as-a-service function, only responsible for encryption and decryption, not storage. The main application scenario is to provide App encrypted and decrypted data, but the data is still stored in MySQL and other databases
  • Certificate Management: The most common scenario is to store the root certificate (root) in Vault, and the business certificate is issued through this Engine

You can see which Secret Engines are currently enabled in the Vault with the command vault secrets list.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

[root@VM_120_245_centos ~/vault]# vault secrets enable -path=secret_bifrost kv
Success! Enabled the kv secrets engine at: secret_bifrost/
[root@VM_120_245_centos ~/vault]# vault secrets list
Path               Type         Accessor              Description
----               ----         --------              -----------
cubbyhole/         cubbyhole    cubbyhole_e86bac2b    per-token private secret storage
identity/          identity     identity_19b16864     identity store
kv/                kv           kv_988a3c7e           n/a
secret/            kv           kv_4a27cb62           n/a
secret_bifrost/    kv           kv_0b5b6ac3           n/a
sys/               system       system_8d02021f       system endpoints used for control, policy and debugging
vault/             kv           kv_e26a68a4           n/a

The output of the command shows that there are 4 engines of type kv (key-value pair encrypted storage), loaded on the kv/, secret/, secret_bifrost/ and vault/ paths; the other engines are supported internally by Vault. You can read and write to the specified paths because they are supported by the Secret Engine; unloaded paths cannot be accessed (an error will be reported), and you cannot open the same paths at the same time.

In addition, the same Secret Engine can be loaded under different paths (multiple instances of a single Secret Engine class), and the data under each path is independent of each other.

Secret Engine

0x03 Vault Details

This section describes some details of the Vault implementation.

Shamir key sharing algorithm (shamir secret sharing)

The implementation of Shamir algorithm is given in Vault. The basic idea of this key sharing algorithm is that the distributor decomposes the secret Secret into n secret holders by a secret polynomial, where any secret at least k can be recovered, i.e., a secret cannot be kept by a single holder, but must be kept by multiple holders and can be recovered only when multiple holders are present at the same time. the secret can be recovered.

shamir secret sharing

Vault Authentication Methods

Vault supports the following authentication mechanisms.

  1. Token method Token is a built-in authentication method in Vault, which is loaded at startup and cannot be disabled. For example, Root Token is output when the server is initialized, and the user who logs in with Root Token has the highest access rights to the system. In Vault, Token is an inheritable tree structure, and this <inheritance> has two meanings.

    • The user holding the Token creates a new Token (Child Token), and by default the Child Token has the same permissions as the parent Token (if specific permissions are required)
    • When a Token is revoked, the Child Token it created, and the Child Token of the Child Token (etc.) are deleted together, which is easy to understand from a tree perspective

  2. AppRole method AppRole is a more secure authentication method provided by Vault for App applications, and is recommended. The general process of using AppRole is as follows (From official website).

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80

    #1. Create a read-only policy file template readonly
[root@VM_120_245_centos /vault]# cat readonly
# Read-only permission on secrets stored at 'secret/data/mysql/webapp'
path "secret/data/mysql/webapp" {
capabilities = ["read"]
}
#2. Load policy
[root@VM_120_245_centos /vault]# vault policy write readonly readonly<br>
Success! Uploaded policy: readonly
#3. Create a token with TTL
[root@VM_120_245_centos /vault]# vault write auth/approle/role/readonly token_policies="readonly" token_ttl=1h token_max_ttl=4h
Success! Data written to: auth/approle/role/readonly

#4. View readonly's authentication information
[root@VM_120_245_centos /vault]#  vault read auth/approle/role/readonly
Key                        Value
---                        -----
bind_secret_id             true local_secret_ids           false secret_id_bound_cidrs      <nil>
secret_id_num_uses         0
secret_id_ttl              0s
token_bound_cidrs          []
token_explicit_max_ttl     0s
token_max_ttl              4h
token_no_default_policy    false token_num_uses             0
token_period               0s
token_policies             [readonly]
token_ttl                  1h
token_type                 default

#5. Check rold-id，vault read auth/approle/role/readonly/role-id
[root@VM_120_245_centos /vault]# vault read auth/approle/role/readonly/role-id
Key        Value
---        -----
role_id    12afcbf7-33c9-d86f-e678-dc2beeb3fabd

#6. Check secret-id
[root@VM_120_245_centos /vault]# vault write -force auth/approle/role/readonly/secret-id
Key                   Value
---                   -----
secret_id             7c9e58d5-8af2-176b-8fbc-572db2f8c872
secret_id_accessor    ff5c6dca-7b5c-587d-41e2-adaae932a669
secret_id_ttl         0s

#7. Exchange token according to role-id and secret-id
[root@VM_120_245_centos /vault]#  vault write auth/approle/login role_id="12afcbf7-33c9-d86f-e678-dc2beeb3fabd"   secret_id="7c9e58d5-8af2-176b-8fbc-572db2f8c872"
Key                     Value
---                     -----
token                   s.qIcsK6N6lm4TffWWcRIRfRSQ
token_accessor          hDt3u9o9hjfHZwQ7Zisun7Vw
token_duration          1h
token_renewable         true token_policies          ["default" "readonly"]
identity_policies       []
policies                ["default" "readonly"]
token_meta_role_name    readonly

#8. Apply the new token to access the cluster
[root@VM_120_245_centos /vault]# export VAULT_TOKEN=s.qIcsK6N6lm4TffWWcRIRfRSQ
[root@VM_120_245_centos ~/vault]# vault kv get secret/data/mysql/webapp
== Data ==
Key    Value
---    -----
a      1

#9. ultra-authorized access failure (unable to write)
[root@VM_120_245_centos /vault]#  vault kv put secret/data/mysql/webapp key=abcd
Error making API request.

URL: GET http://127.0.0.1:8200/v1/sys/internal/ui/mounts/secret/data/mysql/webapp
Code: 403. Errors:

* permission denied

#9. Access failure after Token expiration
[root@VM_120_245_centos /vault]# vault kv get secret/data/mysql/webapp
Error making API request.

URL: GET http://127.0.0.1:8200/v1/sys/internal/ui/mounts/secret/data/mysql/webapp
Code: 403. Errors:

* permission denied

    From an implementation perspective, role-id and secret-id are equivalent to the application’s username and password, but in reality they are not.

    Vault wants to use this design to solve the Secret Zero problem.

  3. The Token method is easy to use, but it is designed to support its own operation and is not very secure. For real user/machine authentication scenarios, Vault officially recommends using other more mature mechanisms such as LDAP, Github, AppRole authentication methods.

Storage

Vault supports a variety of storage backends: https://github.com/hashicorp/vault/tree/master/plugins/database, production architecture backends are deployed using the HA method, such as consul/etcd/mysql clusters, etc.

Storage

0x04 Basic functions of Vault

  1. Configure Mysql as the storage backend and start Vault

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

    [root@VM_120_245_centos ~/vault]# cat vault.hcl
disable_mlock  = true ui=true storage "mysql" {
    address = "127.0.0.1:3306"
    username = "root"
    password = "xxxxxx"
    database = "vault"
    table = "vault"
}
listener "tcp" {
address     = "127.0.0.1:8200"
tls_disable = 1
}

[root@VM_120_245_centos ~/vault]# vault server -config=vault.hcl

  2. initialize vault, get 5 sub-secret keys and root Token

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

    [root@VM_120_245_centos ~/vault]# vault operator init
2021-08-20T21:34:41.973+0800 [INFO]  core: security barrier not initialized
2021-08-20T21:34:42.002+0800 [INFO]  core: security barrier initialized: stored=1 shares=5 threshold=3
2021-08-20T21:34:42.031+0800 [INFO]  core: post-unseal setup starting
2021-08-20T21:34:42.060+0800 [INFO]  core: loaded wrapping token key
2021-08-20T21:34:42.060+0800 [INFO]  core: successfully setup plugin catalog: plugin-directory=""
2021-08-20T21:34:42.060+0800 [INFO]  core: no mounts; adding default mount table
2021-08-20T21:34:42.080+0800 [INFO]  core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2021-08-20T21:34:42.081+0800 [INFO]  core: successfully mounted backend: type=system path=sys/
2021-08-20T21:34:42.081+0800 [INFO]  core: successfully mounted backend: type=identity path=identity/
2021-08-20T21:34:42.136+0800 [INFO]  core: successfully enabled credential backend: type=token path=token/
2021-08-20T21:34:42.137+0800 [INFO]  rollback: starting rollback manager
2021-08-20T21:34:42.137+0800 [INFO]  core: restoring leases
2021-08-20T21:34:42.138+0800 [INFO]  expiration: lease restore complete 2021-08-20T21:34:42.163+0800 [INFO]  identity: entities restored
2021-08-20T21:34:42.164+0800 [INFO]  identity: groups restored
2021-08-20T21:34:42.165+0800 [INFO]  core: usage gauge collection is disabled
2021-08-20T21:34:42.174+0800 [INFO]  core: post-unseal setup complete 2021-08-20T21:34:42.200+0800 [INFO]  core: root token generated
2021-08-20T21:34:42.200+0800 [INFO]  core: pre-seal teardown starting
2021-08-20T21:34:42.200+0800 [INFO]  rollback: stopping rollback manager
2021-08-20T21:34:42.200+0800 [INFO]  core: pre-seal teardown complete Unseal Key 1: xxx1
Unseal Key 2: xxx2
Unseal Key 3: xxx3
Unseal Key 4: xxx4
Unseal Key 5: xxx5

Initial Root Token: xxx-root-token

  3. Unblock vault, view unblock status

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

    [root@VM_120_245_centos ~/vault]# vault operator unseal xxx1
[root@VM_120_245_centos ~/vault]# vault operator unseal xxx2
[root@VM_120_245_centos ~/vault]# vault operator unseal xxx3

[root@VM_120_245_centos ~/vault]# vault status
Key                Value
---                -----
Seal Type          shamir
Initialized        true Sealed             true Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       95ba53e7-63e2-c998-e1b8-4df4bba20ea3
Version            1.8.1
Storage Type       mysql
HA Enabled         false

  4. Login with root Token (first time)

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

    [root@VM_120_245_centos ~/vault]# vault login root-token
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                xxx
token_accessor       xxx
token_duration       ∞
token_renewable      false token_policies       ["root"]
identity_policies    []
policies             ["root"]

  5. Open vault component to test write / read / token generation etc.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

    [root@VM_120_245_centos ~/vault]# vault secrets enable kv
2021-08-20T21:44:37.837+0800 [INFO]  core: successful mount: namespace="" path=kv/ type=kv
[root@VM_120_245_centos ~/vault]# vault kv put kv/test api_key=abc1234 api_secret=1a2b3c4d^C
[root@VM_120_245_centos ~/vault]# vault kv get kv/test
======= Data =======
Key           Value
---           -----
api_key       abc1234
api_secret    1a2b3c4d
[root@VM_120_245_centos ~/vault]# vault token create -ttl 1h
Key                  Value
---                  -----
token                s.6sfhKw2J2dFfNSMdIyWQGqiS
token_accessor       o8MjM5SuLewdJk0WSZ0oPPrF
token_duration       1h
token_renewable      true token_policies       ["root"]
identity_policies    []
policies             ["root"]

[root@VM_120_245_centos ~/vault]# export VAULT_TOKEN=s.6sfhKw2J2dFfNSMdIyWQGqiS
[root@VM_120_245_centos ~/vault]# vault kv get kv/test
======= Data =======
Key           Value
---           -----
api_key       abc1234
api_secret    1a2b3c4d

  6. Create token based on policy template Create read-only policy test-read-policy under kv/test paths

    1
2
3
4
5
6

    [root@VM_120_245_centos ~/vault]# cat limit-token.hcl
path "kv/test"{
capabilities = ["read"]
}
[root@VM_120_245_centos ~/vault]# vault policy write test-read-policy ./limit-token.hcl
Success! Uploaded policy: test-read-policy

    Create token based on policy.

    1
2
3
4
5
6
7
8
9

    [root@VM_120_245_centos ~/vault]# vault token create -policy=test-read-policy
Key                  Value
---                  -----
token                s.NMD47aWmzSUWC1bAqalQCWYw
token_accessor       gi6WVfxwJnGqMXhDVoJXI5AU
token_duration       768h
token_renewable      true token_policies       ["default" "test-read-policy"]
identity_policies    []
policies             ["default" "test-read-policy"]

    Test the operation of the token, read-only, write error, in accordance with the established policy.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

    [root@VM_120_245_centos ~/vault]# export VAULT_TOKEN=s.NMD47aWmzSUWC1bAqalQCWYw
[root@VM_120_245_centos ~/vault]# vault kv get kv/test
======= Data =======
Key           Value
---           -----
api_key       abc1234
api_secret    1a2b3c4d5e6f
[root@VM_120_245_centos ~/vault]# vault kv put  kv/test api_key=foo api_secret=bar
Error writing data to kv/test: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/kv/test
Code: 403. Errors:

* 1 error occurred:
        * permission denied

  7. View and close Secret engine

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

    [root@VM_120_245_centos ~/vault]# vault secrets list
Path          Type         Accessor              Description
----          ----         --------              -----------
cubbyhole/    cubbyhole    cubbyhole_e86bac2b    per-token private secret storage
identity/     identity     identity_19b16864     identity store
kv/           kv           kv_988a3c7e           n/a
secret/       kv           kv_a1c65202           n/a
sys/          system       system_8d02021f       system endpoints used for control, policy and debugging

[root@VM_120_245_centos ~/vault]# vault secrets disable secret/ #关闭 secret/
Success! Disabled the secrets engine (if it existed) at: secret/


[root@VM_120_245_centos ~/vault]# vault secrets list
Path          Type         Accessor              Description
----          ----         --------              -----------
cubbyhole/    cubbyhole    cubbyhole_e86bac2b    per-token private secret storage
identity/     identity     identity_19b16864     identity store
kv/           kv           kv_988a3c7e           n/a
sys/          system       system_8d02021f       system endpoints used for control, policy and debugging

  8. Turn on V2’s Secret Engine

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

    [root@VM_120_245_centos ~]# vault secrets enable -path=secretv2 -version=2 kv
Success! Enabled the kv secrets engine at: secretv2/
[root@VM_120_245_centos ~]# vault secrets list
Path               Type         Accessor              Description
----               ----         --------              -----------
bifrost_vault/     kv           kv_962069cd           n/a
cubbyhole/         cubbyhole    cubbyhole_e86bac2b    per-token private secret storage
identity/          identity     identity_19b16864     identity store
kv/                kv           kv_988a3c7e           n/a
secret/            kv           kv_4a27cb62           n/a
secret_bifrost/    kv           kv_0b5b6ac3           n/a
secretv2/          kv           kv_6e9e3b5b           n/a
sys/               system       system_8d02021f       system endpoints used for control, policy and debugging
vault/             kv           kv_e26a68a4           n/a

More use can be found in the official documentation: https://learn.hashicorp.com to learn more.