k3s/docs/adrs/flannel-options.md

Record architecture decisions

Date: 2023-2-16

Status

Discussing

Context

Previous work

A previous PR was introduced by Manuel but got bogged down in discussion and was never accepted.

Flannel upstream

Flannel is the default and only supported CNI plugin in k3s. Flannel runs in each node and, on a high level, does the following:

1 - Generates a general configuration based on flannel passed flags, subnet, and backend configuration (normally passed as configmap) 2 - Based on the general configuration, it deploys the required infrastructure (e.g. vxlan VTEP) 3 - Based on the general configuration, it yields a flannel CNI configuration file that will be used when creating/deleting pods. Note that CNI plugins read the config file each time Kubelet requests their service.

Kubernetes relevant flannel flags are:

--iface (string): interface to use (IP or name) for inter-host communication. Can be specified multiple times to check each option in order. Returns the first match found. --iface-can-reach (string): detect interface to use (IP or name) for inter-host communication based on which will be used for provided IP. This is exactly the interface to use of command 'ip route get ' --iface-regex (string): regex expression to match the first interface to use (IP or name) for inter-host communication --ip-masq (bool): setup IP masquerade rule for traffic destined outside of overlay networki (applies to both IPv4 and IPv6) --iptables-forward-rules (bool): Overrides default rule of FORWARD chain to ACCEPT --public-ip (string): IP accessible by other nodes for inter-host communication --public-ipv6 (string): IPv6 accessible by other nodes for inter-host communication

Kubernetes relevant flannel subnet and backend configuration parameters:

Network (string): IPv4 network in CIDR format to use for the entire flannel network (Mandatory) IPv6Network (string): IPv6 network in CIDR format to use for the entire flannel network. (Mandatory if EnableIPv6 is true) EnableIPv6 (bool): Enables ipv6 support Backend (dictionary): Type of backend to use and specific configurations for that backend.

K3s using Flannel

K3s prepares both flannel flags, subnet, and backend configuration before starting flannel. Some flags and config parameters can be configured using k3s flags, some are hardcoded, and some are not supported in the current K3s:

Flag	Supported by K3s	K3s flag
--iface (string)	Yes	--flannel-iface
--iface-can-reach (string)	No**	N/A
--iface-regex (string)	No**	N/A
--ip-masq (bool)	Yes/Hardcoded	ipv4 hardcoded to true. ipv6 configurable by k3s server flag --flannel-ipv6-masq
--iptables-forward-rules (bool)	Hardcoded	True
--public-ip (string)	Hardcoded	--external-ip when it is of IPv4 nature if k3s server flag --flannel-external-ip is true
--public-ipv6 (string)	hardcoded	--external-ip when it is of IPv6 nature if k3s server flag --flannel-external-ip is true

** Does not provide much value as flannel in k3s can configure the interface per node

Regarding subnet and backend configuration:

Network and IPv6Network are set by reading the assigned podCIDR for the Kubernetes node. EnableIPv6 is set based on what the user passed as k3s server --cluster-cidr. Backend uses vxlan as default, although this can be changed using k3s server flag --flannel-backend

Something important to note is that k3s allows the user to override the whole subnet and backend configuration, by using the k3s agent flag --flannel-conf and --flanel-cni-conf. Overriding the whole subnet and backend configuration is useful if user wants to add specific backend configurations, e.g. VNI for vxlan. Users can currently do this by using <=option1=val1,option2=val2> when selecting the backend, but it seldom used and complicates things, thus it was deprecated in favor of --flannel-conf.

To wrap up the context, k3s includes the following flannel options:

Agent Flag	Type	Description
--flannel-iface	string	Overrides the default flannel interface. This interface is used to forward encapsulated traffic in inter-node communication. It matches the flannel flag --iface
--flannel-conf	string	Path that points to a file containing the flannel subnet&backend config. Used directly by flannel and contains clusterCIDR, backend type, and other flannel config.
--flannel-cni-conf	string	Path that point to a flannel CNI config file. This is used by kubelet to configure the network when a new pod is created. The naming is opaque as the flag in flannel is called cni-conf, but every flannel flag in k3s has the flannel prefix, thus flannel-cni-conf.

Server Flag	Type	Description
--flannel-backend	string	Sets the encapsulation technology that will allow inter-node traffic to work. It matches part of the Backend parameter in the subnet&backend configuration
--flannel-ipv6-masq	bool	Enables masquerading traffic for IPv6. Note that IPv4 traffic is always masqueraded. It matches the ipv6 part of the --ip-masq flag
--flannel-external-ip	bool	Enables using node external IP addresses for flannel traffic. It sort of matches on --public-ip and --public-ipv6 flag. In this case, k3s selects the external-ips for those flags

Design suggestion

The suggestion is to include a string slice flag that would consolidate all flannel bool server flags into just one. Benefits:

• Simplify user's life by having fewer flags
• Use it for potential new flannel toggles

Proposal

Starting in v1.26, introduce a new flannel-opt flag that includes flannel server options. The redundant flags are deprecated and removed in a few releases.

We could reduce it to 2:

• flannel-backend (string)
• flannel-opt ([]string)

Flannel-opt would have the following values:

Value	Old Flag
ipv6-masq	--flannel-ipv6-masq
external-ip	--flannel-external-ip

Similar to how we handle --kubelet-arg, both comma separated lists and repeated args would be accepted.

Examples of usage:
--flannel-opt=ipv6-masq,external-ip (assumes true)
--flannel-opt=ipv6-masq=true
--flannel-opt=ipv6-masq=true,external-ip=false
--flannel-opt=ipv6-masq --flannel-opt=external-ip --flannel-opt=ipv6-masq --flannel-opt=external-ip=false

Alternatives

The naming of the flag could be flannel-arg, but I don't want users to assume that we pass flags to flannel directly.

Decision

Consequences