Restrict API and storage access during incident response
Emergency situations or unexpected behavior can arise when operating any software, with Vault being no exception.
Learn how to use critical Vault features with your processes and procedures for swiftly responding to incidents.
Challenge
Operators and responders require features and functionality which effectively support break-glass procedures used in incident response.
When incidents such as credential leakage, intrusion, or denial of service (DOS) attacks occur, timely mitigation is of the essence when supporting business operational goals.
Solution
Learn about and use these two important Vault features which are designed to aid operators in effectively responding to incidents.
Seal: In a typical operation, Vault runs in an unsealed state. However, an operator with sufficient privileges can seal the running Vault at any time. When sealed, the Vault server discards its in-memory key for unlocking data, and is therefore physically blocked from responding to all requests with the exception of status checks and unsealing.
API Lock: (new in v1.9.0) A namespace API lock feature was added to provide more granular access lockout. If an operator does not require that Vault be entirely sealed, they can choose instead to lock only the API on a per-namespace level, and prevent Vault from responding to all but status and unlocking requests.
Prerequisites
To perform all the steps in the hands on scenario, you need:
Vault Enterprise v1.9 or later binary installed in your system path.
jq is used to pretty print JSON output examples.
Policy requirements
Note
For the purpose of this tutorial, you can use the root
token to work with Vault. However, it is recommended that root tokens are used for just enough initial setup or in emergencies. As a best practice, use tokens with an appropriate set of policies based on your role in the organization. Unless you are running Vault locally in development mode (-dev
), it is recommended that you authenticate with Vault and acquire a token with an appropriate set of policies based on your role in the organization.
To perform all tasks demonstrated in this tutorial, your policy must include the following permissions:
If you are connecting to a non-development mode Vault server, your token must have a policy with the following permissions:
# Seal Vaultpath "sys/seal" { capabilities = [ "create", "update", "sudo" ]} # Unseal Vaultpath "sys/unseal" { capabilities = [ "create", "update", "sudo" ]} # Lock and unlock API per namespacepath "/sys/namespaces/api-lock/+/*" { capabilities = [ "create", "update", "sudo" ]}
If you are not familiar with policies, complete the policies tutorial.
Personas
The end-to-end scenario described in this tutorial involves two personas:
operator
with privileged capabilities for sealing and unsealing Vault, along with locking and unlocking API endpoints.driver
uses the username and password auth method enabled within the drivers namespace to authenticate with Vault.
Scenario introduction
As shown at left, when Vault is sealed, all users are affected and unable to use it.
Contrast this with locking the API at the namespace level as shown on the right. Although the drivers namespace within the teams namespace is locked, users in the teams and root namespaces are unaffected by the lock.
You will use terminal sessions to operate a single Vault server with Integrated Storage (Raft), and follow a series of hands-on scenarios to learn how to seal and unseal Vault from the Vault CLI, HTTP API, or web UI.
You will then learn how to lock the Vault API on a per-namespace basis from the Vault CLI or HTTP API with a username and password auth method example.
You can also clean up the scenario environment by using the provided example commands when you are finished learning.
Establish a scenario environment
Make a temporary directory to hold the files created for this scenario, and assign its path to the environment variable LEARN_VAULT.
$ mkdir -p /tmp/learn-vault/data && export LEARN_VAULT=/tmp/learn-vault
Create Vault server configuration
Create a minimal configuration for a single Vault server that uses Raft storage.
$ cat > $LEARN_VAULT/vault-server.hcl << EOFapi_addr = "http://127.0.0.1:8200"cluster_addr = "http://127.0.0.1:8201"cluster_name = "learn-vault"default_lease_ttl = "10h"disable_mlock = truemax_lease_ttl = "10h"ui = true listener "tcp" { address = "127.0.0.1:8200" tls_disable = "true"} backend "raft" { path = "/tmp/learn-vault/data" node_id = "learn-vault-1"}EOF
Start a single Vault server
Open a terminal session and start a Vault server using the configuration that you just created.
$ vault server -config $LEARN_VAULT/vault-server.hcl
The server starts uninitialized and sealed.
Initialize, unseal & authenticate
In another terminal session, export the VAULT_ADDR environment variable to address the Vault server and export the LEARN_VAULT environment variable to define the scenario directory.
$ export VAULT_ADDR=http://127.0.0.1:8200 LEARN_VAULT=/tmp/learn-vault
For the purpose of simplicity in this tutorial, initialize Vault with 1 key share and a key threshold of 1 and write the output to the file .vault-init
in the project directory.
$ vault operator init \ -key-shares=1 \ -key-threshold=1 \ | head -n3 \ | cat > $LEARN_VAULT/.vault-init
Successful execution of this command should produce no output.
Export the unseal key value as the environment variable UNSEAL_KEY.
$ export UNSEAL_KEY=$(grep 'Unseal Key 1' $LEARN_VAULT/.vault-init | awk '{print $NF}')
Unseal Vault with the Unseal Key 1 value from the .vault-init
file.
$ vault operator unseal $UNSEAL_KEY
Successful output from unsealing Vault should resemble this example:
Key Value--- -----Seal Type shamirInitialized trueSealed falseTotal Shares 1Threshold 1Version 1.9.0+entStorage Type raftCluster Name learn-vaultCluster ID fb4c6314-f2df-2cbc-aa21-9e794f44003aHA Enabled trueHA Cluster n/aHA Mode standbyActive Node Address <none>Raft Committed Index 55Raft Applied Index 55
Export the initial root token value to the environment variable ROOT_TOKEN.
$ export ROOT_TOKEN=$(grep 'Initial Root Token' $LEARN_VAULT/.vault-init | awk '{print $NF}')
login with vault login
by passing the Initial Root Token value from the .vault-init
file.
$ vault login -no-print $ROOT_TOKEN
This command should produce no output when successful. If you want to confirm that the login was successful, try a token lookup and confirm that your token policies contain root.
Note
You will use a root token in this scenario for simplicity. However, in actual production environments, root tokens should be closely guarded and used only for tightly controlled purposes. Review the documentation on root tokens for more details.
$ vault token lookup | grep policies
Successful output should contain the following.
policies [root]
You are ready for the first scenario.
Seal Vault
Vault running in an unsealed state uses the master encryption key to decrypt the encryption key kept in Vault storage that is itself used for encrypting and decrypting secrets in storage. Vault is sealed with the CLI from a terminal session in this example; the master encryption key is discarded from memory, and no further encryption or decryption from storage is possible until Vault is unsealed again.
If an incident such as intrusion or credential leakage occurs, sealing Vault in response is the heaviest tool you can use. The key used to decrypt data will be immediately discarded from memory, and Vault can no longer decrypt anything in its storage until unsealed again.
Note
A sealed Vault is effectively no longer available for normal use. You should keep this in mind particularly when using enterprise replication, as a sealed cluster will no longer be participating in replication. A sealed primary will prevent all secondaries from replicating with it, and a sealed secondary cluster will become out of sync with its primary.
An operator using a token with sufficient capabilities as described in the Policy requirements section can seal Vault with the vault
CLI, the /sys/seal HTTP API or by using, or the Web UI.
(Persona: operator)
Seal your Vault server.
$ vault operator sealSuccess! Vault is sealed.
You can check Vault status to confirm.
$ vault statusKey Value--- -----Seal Type shamirInitialized trueSealed trueTotal Shares 1Threshold 1Unseal Progress 0/1Unseal Nonce n/aVersion 1.9.0-rc1+entStorage Type inmem_transactional_haHA Enabled true
The output should show that the Sealed value is true.
You can also change to the terminal where you started the Vault server to observe the operational logging. The end of the log should resemble this example.
2021-11-14T17:23:25.131-0500 [INFO] core.cluster-listener: rpc listeners successfully shut down2021-11-14T17:23:25.131-0500 [INFO] core: cluster listeners successfully shut down2021-11-14T17:23:25.131-0500 [INFO] core: vault is sealed
Note the last line which specifically states core: vault is sealed
; this is the best signal from the logs to you as an operator that Vault is now sealed.
If you attempt to login to Vault now, you'll discover an error like the one shown in this example:
$ vault login -no-print $ROOT_TOKENError authenticating: error looking up token: Error making API request. URL: GET http://127.0.0.1:8200/v1/auth/token/lookup-selfCode: 503. Errors: * Vault is sealed
With Vault sealed, an operator must unseal it to restore normal operations.
Note
The examples in this tutorial use the Shamir's Secret Sharing based seal with an unseal key, but in typical production installations, a cloud based automatic seal is used. Keep in mind that when a Vault server using auto unseal is sealed, it will automatically unseal itself if restarted. Use caution when restarting such servers.
Use the unseal key value recorded earlier to unseal Vault in preparation for the next scenario.
(Persona: operator)
Unseal Vault.
$ vault operator unseal $UNSEAL_KEY
Change to the terminal where you started the Vault server to observe the operational logging. The end of the log should resemble this example.
2021-11-13T16:40:16.739-0500 [INFO] core: post-unseal setup complete2021-11-13T16:40:16.739-0500 [INFO] core: vault is unsealed2021-11-13T16:40:16.740-0500 [INFO] expiration: lease restore complete
Note the line core: vault is unsealed
.
Vault is now unsealed and ready for normal operations.
Lock API
Vault Enterprise edition 1.9.0 introduced a per-namespace level API lock feature that an operator can use to lock the API for any enterprise namespace.
When the API is locked for a particular namespace, it is also locked for that namespace's descendants.
In this example, the namespace teams was locked by an operator. The drivers namespace is a descendant of teams, and as a result it was also locked. This means that the userpass auth method enabled in the drivers namespace cannot be used.
When the driver user attempts authentication with the userpass method in the drivers namespace, an error about the locked status of the namespace is returned.
Vault blocks all API endpoints from responding to clients operating in a locked namespace or a descendant of a locked namespace with the exception of endpoints for status and unlocking the API lock.
Per namespace API locking provides a more granular tool for incident response that helps the operator to focus on locking one or more namespaces instead of the sealing the entire Vault. The feature is particularly helpful in multi-tenant environments where sealing Vault is too heavy a response that affects all tenants.
If you are unfamiliar with enterprise namespaces, review the Multi-tenancy with Namespaces tutorial.
Create namespaces and enable auth method
(Persona: operator)
Before you try the API lock feature in your Vault server, create two namespaces, a teams namespace and a drivers namespace that is a descendant of teams
.
Create the teams namespace.
$ vault namespace create teamsKey Value--- -----id SpQLEpath teams/
Create the drivers namespace as a descendant of the teams namespace.
$ vault namespace create -namespace=teams driversKey Value--- -----id dfTq1path teams/drivers/
Enable a username and password auth method in the drivers namespace.
$ vault auth enable -namespace teams/drivers userpass
Create a driver user within the drivers namespace.
$ vault write -namespace teams/drivers auth/userpass/users/driver \ password=p@ssw0rd
Test auth method
(Persona: driver)
Test authentication of the driver user with the userpass auth method in the teams namespace.
$ vault login \ -namespace teams/drivers \ -method=userpass \ username=driver \ password=p@ssw0rd
Successful output resembles this example.
Success! You are now authenticated. The token information displayed belowis already stored in the token helper. You do NOT need to run "vault login"again. Future Vault requests will automatically use this token.Key Value--- -----token s.fwOqtyX5wWQVx4TIaVkL1gLP.dfTq1token_accessor HjEefUHXOQy9sFWvzF0JGbcF.dfTq1token_duration 768htoken_renewable truetoken_policies ["default"]identity_policies []policies ["default"]token_meta_username driver
Lock API in drivers namespace
(Persona: operator)
After successfully testing the authentication, login with the root token again.
$ vault login -no-print $ROOT_TOKEN
Lock the API in the drivers namespace.
$ vault namespace lock teams/driversKey Value--- -----unlock_key GF1pw5hU3dDNeYmSRzHxya8T
The unlock key is returned; this key must be used to later unlock the API.
Test auth method again
(Persona: driver)
With the API locked for the drivers namespace, attempt to authenticate as the user driver with the userpass auth method.
$ vault login \ -namespace teams/drivers \ -method=userpass \ username=driver \ password=p@ssw0rd
A helpful error is returned indicating that the namespace is locked.
Error authenticating: Error making API request.Namespace: teams/drivers/URL: PUT http://127.0.0.1:8200/v1/auth/userpass/login/driverCode: 503. Errors:* 1 error occurred: * API access to this namespace has been locked by an administrator - "teams/drivers/" must be unlocked to gain access.
Unlock API in drivers namespace
(Persona: operator)
Use the unlock key previously returned to unlock the API in the drivers namespace.
$ vault namespace unlock -unlock-key GF1pw5hU3dDNeYmSRzHxya8T teams/drivers
Successful execution results in no output.
Test auth method for the last time
(Persona: driver)
Test the authentication once more to show that the unlock was successful.
With the API locked for the drivers namespace unlocked, attempt to authenticate with the userpass auth method as the user driver.
$ vault login \ -namespace teams/drivers \ -method=userpass \ username=driver \ password=p@ssw0rd
The result should be a successful login.
Cleanup
If you wish to clean up your environment after completing the tutorial, follow the steps in this section.
Unset the environment variables.
$ unset ROOT_TOKEN UNSEAL_KEY VAULT_ADDR LEARN_VAULT
You can stop the Vault server by pressing Ctrl+C where the server is running. Or, execute the following command.
$ pgrep -f vault | xargs kill
Remove the scenario environment directory.
$ rm -rf /tmp/learn-vault
Summary
You learned about two important features for incident response available in Vault. You learned about sealing an unsealing Vault, a feature available in all Vault editions.
You also learned about per namespace API locking and unlocking, a feature available only in Vault Enterprise editions.