Mirror
/
k3s
mirror of https://github.com/k3s-io/k3s.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
Page: Secrets Encryption Test Plan
Pages
Home K3s Cert Rotation SDLC Secrets Encryption Test Plan
1 Secrets Encryption Test Plan
Max edited this page 2022-01-07 15:25:30 -07:00

K3s currently has only a single command for managing secrets encryption - the --secrets-encryption flag. If this is turned on, a secrets encryption configuration document is generated using a random aescbckey key; this document is stored in the bootstrap data and shared among server nodes via the kube-apiserver --encryption-provider-config CLI flag. There are no provisions for rotating the key, disabling encryption after enabling it, etc. This feature expansion will allow for rotating the encryption key via a multi-step process in a single and multi node cluster. Ref: Design Doc

Feature

secrets-encryption keys rotation and enable/disable of encryption. The is all controlled via a new sub command k3s secrets-encrypt. Ref: Ticket

$ k3s secrets-encrypt --help
NAME:
   k3s secrets-encrypt - Control secrets encryption and keys rotation

USAGE:
   k3s secrets-encrypt command [command options] [arguments...]

COMMANDS:
   status     Print current status of secrets encryption
   enable     Enable secrets encryption
   disable    Disable secrets encryption
   prepare    Prepare for encryption keys rotation
   rotate     Rotate secrets encryption keys
   reencrypt  Reencrypt all data with new encryption key

OPTIONS:
   --help, -h  show help

Note: Stopping/killing and Restarting the service after each command, except status, is required. If not performed then the cluster goes in to bad state

Test Cases

Pre-requisite

Install k3s with `--secrets-encryption` flag

$ curl -sfL https://get.k3s.io | INSTALL_K3S_COMMIT=<commithash> INSTALL_K3S_EXEC="server --write-kubeconfig-mode 644 --secrets-encryption" sh -

OR

Install k3s using parameter `INSTALL_K3S_SKIP_START=true`. Start the service with `--secrets-encryption` flag

$ curl -sfL https://get.k3s.io | INSTALL_K3S_COMMIT=<commithash> INSTALL_K3S_EXEC="server --write-kubeconfig-mode 644" INSTALL_K3S_SKIP_START=true sh -
$ sudo k3s server secrets-encrypt

Adding secret

$ kubectl create secret generic secret1 -n default --from-literal=mykey=mydata
secret/secret1 created
$ kubectl describe secret secret1 -n default
Name:         secret1
Namespace:    default
Labels:       <none>
Annotations:  <none>
Type:  Opaque
Data
====
mykey:  6 bytes
#Test CaseStepsDocs Needed?Expected ResultPass/Fail VersionNotes /GH issues
1

secrets-encrypt status

Print current status of secrets encryption

# Check status
$ sudo k3s secrets-encrypt status
Yes

Should show the status of secrets-encryption

Encryption Status: Enabled
Current Rotation Stage: start
Server Encryption Hashes: All hashes match
Active  Key Type  Name
------  --------  ----
 *      AES-CBC   aescbckey

PASS: v1.21.8+k3s1

PASS: v1.22.5+k3s1

N/A
2

secrets-encrypt enable (Enabled by default)

Enable secrets encryption

Use flag --force --skip to properly enable

# Check status
$ sudo k3s secrets-encrypt status
# Enable secrets encryption
$ sudo k3s secrets-encrypt enable
# Check status
$ sudo k3s secrets-encrypt status
# Stop/Kill and Restart k3s service
$ sudo systemctl stop k3s && sudo systemctl restart k3s
Yes

Console output shows

secrets-encryption enabled

Status should show the correct state of secrets-encryption

Encryption Status: Enabled
Current Rotation Stage: start
Server Encryption Hashes: All hashes match
Active  Key Type  Name
------  --------  ----
 *      AES-CBC   aescbckey

PASS: v1.21.8+k3s1

PASS: v1.22.5+k3s1

N/A
3

secrets-encrypt disable

Disable secrets encryption

Use flag --force --skip to properly disable

# Check status, ensure it is enabled
$ sudo k3s secrets-encrypt status
# Disable secrets encryption
$ sudo k3s secrets-encrypt disable
# Check status
$ sudo k3s secrets-encrypt status
# Stop/Kill and Restart k3s service
$ sudo systemctl stop k3s && sudo systemctl restart k3s
# Deploy another secret and ensure it is not encrypted
$ kubectl create secret generic secret2 -n default --from-literal=mykey=mydata
Yes

Console output shows

secrets-encryption disabled

Status should show the correct state of secrets-encryption

Encryption Status: Disabled
Current Rotation Stage: start
Server Encryption Hashes: All hashes match
Active  Key Type  Name
------  --------  ----
 *      AES-CBC   aescbckey

Use ETCDCTL to show secrets being encrypted/not encrypted. Change the below command to point to secret1, which should still show k8s:enc:aescbc:v1: and not have the data of the secret, meaning it is encrypted. secret2 will not show that, and will show the secret data, meaning it is not encrypted

sudo ETCDCTL_API=3 etcdctl --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --endpoints https://127.0.0.1:2379 --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt get /registry/secrets/default/secret2 | hexdump -C

PASS: v1.21.8+k3s1

PASS: v1.22.5+k3s1


4

secrets-encrypt prepare

Prepare for encryption keys rotation

# Check status, ensure its enabled
$ sudo k3s secrets-encrypt status
# Prepare secrets encryption
$ sudo k3s secrets-encrypt prepare
# Check status
$ sudo k3s secrets-encrypt status
# Stop/Kill and Restart k3s service
$ sudo systemctl stop k3s && sudo systemctl restart k3s
Yes

Console output shows

prepare completed successfully

Status should show the correct state of secrets-encryption along with new key

Encryption Status: Enabled
Current Rotation Stage: prepare
Server Encryption Hashes: All hashes match
Active  Key Type  Name
------  --------  ----
 *      AES-CBC   aescbckey
        AES-CBC   aescbckey-2021-12-08T21:34:03Z

PASS: v1.21.8+k3s1

PASS: v1.22.5+k3s1


5

secrets-encrypt rotate

Rotate secrets encryption keys

# Check status, ensure its enabled
$ sudo k3s secrets-encrypt status
# Prepare secrets encryption
$ sudo k3s secrets-encrypt prepare
# Check status, ensure it is in prepare stage
$ sudo k3s secrets-encrypt status
# Stop/Kill and Restart k3s service
$ sudo systemctl stop k3s && sudo systemctl restart k3s
# Rotate secrets encryption
$ sudo k3s secrets-encrypt rotate
# Check status
$ sudo k3s secrets-encrypt status
# Stop/Kill and Restart k3s service
$ sudo systemctl stop k3s && sudo systemctl restart k3s
Yes

Console output shows

rotate completed successfully

Status should show the correct state of secrets-encryption with new key as active

Encryption Status: Enabled
Current Rotation Stage: rotate
Server Encryption Hashes: All hashes match
Active  Key Type  Name
------  --------  ----
 *      AES-CBC   aescbckey-2021-12-08T21:34:03Z
        AES-CBC   aescbckey

PASS: v1.21.8+k3s1

PASS: v1.22.5+k3s1


6

secrets-encrypt reencrypt

Reencrypt all data with new encryption key

Use flag -f or --force

# Check status, ensure its enabled
$ sudo k3s secrets-encrypt status
# Prepare secrets encryption
$ sudo k3s secrets-encrypt prepare
# Check status, ensure it is in prepare stage
$ sudo k3s secrets-encrypt status
# Stop/Kill and Restart k3s service
$ sudo systemctl stop k3s && sudo systemctl restart k3s
# Rotate secrets encryption
$ sudo k3s secrets-encrypt rotate
# Check status, ensure it is in rotate stage
$ sudo k3s secrets-encrypt status
# Stop/Kill and Restart k3s service
$ sudo systemctl stop k3s && sudo systemctl restart k3s
# Re-encrypt secrets encryption
$ sudo k3s secrets-encrypt reencrypt
# Check status, ensure it is in reencrypt_active stage
$ sudo k3s secrets-encrypt status
# Check status until it is in reencrypt_finished stage
$ sudo k3s secrets-encrypt status
# Stop/Kill and Restart k3s service
$ sudo systemctl stop k3s && sudo systemctl restart k3s
Yes

Console output shows

reencryption started

Status should show the correct state of secrets-encryption with new key as active

Encryption Status: Enabled
Current Rotation Stage: reencrypt_active
Server Encryption Hashes: All hashes match
Active  Key Type  Name
------  --------  ----
 *      AES-CBC   aescbckey-2021-12-08T21:34:03Z
        AES-CBC   aescbckey

Status shows correct state of secrets-encryption with new key as active and old key as removed

Encryption Status: Enabled
Current Rotation Stage: reencrypt_finished
Server Encryption Hashes: All hashes match
Active  Key Type  Name
------  --------  ----
 *      AES-CBC   aescbckey-2021-12-08T21:34:03Z

PASS: v1.21.8+k3s1

PASS: v1.22.5+k3s1


Multi-node cluster (HA)

With embedded etcd

#Test CaseStepsExpected ResultPass/Fail VersionNotes/GH issue
1Perform secrets encryption rotation on multi node cluster with
3 servers, 1 agent 
# Start up 3 K3s server
$ k3s server --secrets-encryption
# Check status on all the servers, ensure all shows the same result
$ sudo k3s secrets-encrypt status 
# Select ONE server (S1 going forward) to perform the rotate on (doing any stage on any server is supported)
# Perform secrets-encrypt prepare on S1
$ sudo k3s secrets-encrypt prepare
# Stop/Kill and Restart k3s service on S1
$ sudo systemctl stop k3s && sudo systemctl restart k3s
# Once S1 is back up, restart the other two servers
$ sudo systemctl stop k3s && sudo systemctl restart k3s
# Check status on all the servers, ensure all shows the same result
$ sudo k3s secrets-encrypt status 
# Perform secrets-encrypt rotate on S1
$ sudo k3s secrets-encrypt rotate
# Stop/Kill and Restart k3s service on S1
$ sudo systemctl stop k3s && sudo systemctl restart k3s
# Once S1 is back up, restart the other two servers
$ sudo systemctl stop k3s && sudo systemctl restart k3s
# Check status on all the servers, ensure all shows the same result
$ sudo k3s secrets-encrypt status 
# Perform secrets-encrypt reencrypt on S1
$ sudo k3s secrets-encrypt reencrypt
# Check status on all the servers
$ sudo k3s secrets-encrypt status 
# Stop/Kill and Restart k3s service on S1
$ sudo systemctl stop k3s && sudo systemctl restart k3s
# Once S1 is back up, restart the other two servers
$ sudo systemctl stop k3s && sudo systemctl restart k3s
# Check status on all the servers, ensure all shows the same result
$ sudo k3s secrets-encrypt status

PASS: v1.21.8+k3s1

PASS: v1.22.5+k3s1


2Ensure all server nodes perform reconciliation on restart.
  • Start 3 k3s servers (1 cluster-init, 2 --server)
  • On server 2 run k3s secrets-encrypt prepare
  • Check that the status has updated only on server 2 k3s secrets-encrypt status
  • Kill all 3 servers
  • Restart only servers 2 and 3.
  • Check that server 3 now has the same status as server 2 k3s secrets-encrypt status

PASS: v1.21.8+k3s1

PASS: v1.22.5+k3s1

Detailed steps to be added later

With external DB

#Test CaseStepsExpected ResultPass/Fail VersionNotes/GH issue
1Perform secrets encryption rotation on multi node cluster with
3 servers, 1 agent 
# Start up 3 K3s server joined to external DB (Ex: mySQL)
$ k3s server --secrets-encryption --datastore-endpoint "mysql://root:mysql@tcp(<server_ip>:3306)/k3s"
# Check status on all the servers, ensure all shows the same result
$ sudo k3s secrets-encrypt status 
# Select ONE server (S1 going forward) to perform the rotate on (doing any stage on any server is supported)
# Perform secrets-encrypt prepare on S1
$ sudo k3s secrets-encrypt prepare
# Stop/Kill and Restart k3s service on S1
$ sudo systemctl stop k3s && sudo systemctl restart k3s
# Once S1 is back up, restart the other two servers
$ sudo systemctl stop k3s && sudo systemctl restart k3s
# Check status on all the servers, ensure all shows the same result
$ sudo k3s secrets-encrypt status 
# Perform secrets-encrypt rotate on S1
$ sudo k3s secrets-encrypt rotate
# Stop/Kill and Restart k3s service on S1
$ sudo systemctl stop k3s && sudo systemctl restart k3s
# Once S1 is back up, restart the other two servers
$ sudo systemctl stop k3s && sudo systemctl restart k3s
# Check status on all the servers, ensure all shows the same result
$ sudo k3s secrets-encrypt status 
# Perform secrets-encrypt reencrypt on S1
$ sudo k3s secrets-encrypt reencrypt
# Check status on all the servers
$ sudo k3s secrets-encrypt status 
# Stop/Kill and Restart k3s service on S1
$ sudo systemctl stop k3s && sudo systemctl restart k3s
# Once S1 is back up, restart the other two servers
$ sudo systemctl stop k3s && sudo systemctl restart k3s
# Check status on all the servers, ensure all shows the same result
$ sudo k3s secrets-encrypt status

PASS: v1.21.8+k3s1

PASS: v1.22.5+k3s1


Additional Scenarios

#Test ScenariosStepsExpected ResultPass/Fail VersionNotes/GH issues
1Perform commands without restarting k3s
# Prepare secrets encryption
$ sudo k3s secrets-encrypt prepare
# Rotate secrets encryption
$ sudo k3s secrets-encrypt rotate
# Re-encrypt secrets encryption
$ sudo k3s secrets-encrypt reencrypt
# Stop/Kill and Restart k3s service
$ sudo systemctl stop k3s && sudo systemctl restart k3s

The system does not restart and goes in to infinite loop with the error below

Dec 08 21:39:09 ip-172-31-41-171 k3s[5579]: E1208 21:39:09.263197    5579 cacher.go:420] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/default-token-f6bkz": no matching key was found for the provided AES transformer; reinitializing...
Dec 08 21:39:09 ip-172-31-41-171 k3s[5579]: E1208 21:39:09.941337    5579 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/default-token-f6bkz": no matching key was found for the provided AES transformer

PASS: v1.21.8+k3s1

PASS: v1.22.5+k3s1

This action breaks the cluster so don't try this! This happens because all the secrets gets locked and the key gets thrown away
2

Perform same command repeatedly / multiple times

On running `sudo k3s secrets-encrypt enable`

Console shows `secrets-encryption enabled` without any errors

PASS: v1.21.8+k3s1

PASS: v1.22.5+k3s1


On running `sudo k3s secrets-encrypt disable`Console shows `secrets-encryption enabled` without any errors

PASS: v1.21.8+k3s1

PASS: v1.22.5+k3s1


On running `sudo k3s secrets-encrypt prepare`Console shows `FATA[0000] https://127.0.0.1:6443/v1-k3s/encrypt/config: 400 Bad Request error secrets-encrypt-90363: see server logs for more info` after showing `prepare completed successfully` on the first command run

PASS: v1.21.8+k3s1

PASS: v1.22.5+k3s1


On running `sudo k3s secrets-encrypt rotate`Console shows `FATA[0000] https://127.0.0.1:6443/v1-k3s/encrypt/config: 400 Bad Request error secrets-encrypt-45155: see server logs for more info` after showing `rotate completed successfully` on the first command run

PASS: v1.21.8+k3s1

PASS: v1.22.5+k3s1


On running `sudo k3s secrets-encrypt reencrypt`

Console shows `reencryption started` with no errors but the system gets stuck with rotation stage as `reencrypt_request` and spits out errors in the log 

Dec 10 18:28:15 ip-172-31-32-180 k3s[2146]: time="2021-12-10T18:28:15Z" level=error msg="error syncing 'ip-172-31-32-180': handler reencrypt-controller: invalid hash: 8d0f2a2ad73ce974c4cc39874ebcb3b31d142774aed2c4771e4b4205c94a028f found on node ip-172-31-32-180, requeuing"

PASS: v1.21.8+k3s1

PASS: v1.22.5+k3s1


3

Perform each stage on different servers in HA cluster



PASS: v1.21.8+k3s1

PASS: v1.22.5+k3s1


4

Perform commands out of sync

# Rotate secrets encryption
$ sudo k3s secrets-encrypt rotate
# Prepare secrets encryption
$ sudo k3s secrets-encrypt prepare
# Disable secrets encryption
$ sudo k3s secrets-encrypt disable
# Enable secrets encryption
$ sudo k3s secrets-encrypt enable

Console shows

FATA[0000] https://127.0.0.1:6443/v1-k3s/encrypt/config: 400 Bad Request error secrets-encrypt-5012: see server logs for more info

PASS: v1.21.8+k3s1

PASS: v1.22.5+k3s1


5

Perform re-encrypt with 1000+ secrets stored in the DB





6Perform commands on agent node instead of server

Install k3s with 1 server and 1 agent node
On the agent node, ran

sudo k3s secrets-encrypt status

Console shows

FATA[0000] open /var/lib/rancher/k3s/server/token: no such file or directory

PASS: v1.21.8+k3s1

PASS: v1.22.5+k3s1


    </div>

Test Plans