跳转至

什么是 Sriov-network-operator

目前使用 sriov 的方式比较复杂繁琐，需要管理员完全手动配置，如手动确认网卡是否支持 SRIOV、配置 PF 和 VF 等，参考 sriov。社区开源 Sriov-network-operator， 旨在降低使用 sriov-cni 的难度。sriov-operator 整合 sriov-cni 和 sriov-device-plugin 两个项目， 完全使用 CRD 的方式统一使用和配置 sriov，包括组件本身和节点上的必要配置，极大地降低了使用难度。

组件

[root@controller1 ~]# kubectl  get po -n sriov-network-operator -o wide
NAME                                                              READY   STATUS      RESTARTS   AGE    IP               NODE          NOMINATED NODE   READINESS GATES
sriov-device-plugin-kw8rc                                         1/1     Running     0          62s    10.5.212.132     worker1       <none>           <none>
sriov-network-config-daemon-nhmws                                 3/3     Running     0          76m    10.5.212.132     worker1       <none>           <none>
sriov-network-operator-6955b75d8c-gmpcc                           1/1     Running     0          67s    10.233.73.233    controller1   <none>           <none>
  • sriov-operator：控制层面组件，监听 CRs 变化，安装和配置 sriov-cni 和 sriov-device-plugin 组件
  • sriov-network-config-daemon：与节点交互，用于开启节点网卡的 SR-IOV 功能和配置 VFs。内置 srivo-cni, 将 sriov-cni 的二进制文件拷贝至主机的 /opt/cni/bin 目录下
  • sriov-device-plugin：发现主机上的 VFs，并宣告给 kubelet

CRD

SriovNetworkNodeState： SriovNetworkNodeState 发现主机上支持 SR-IOV 功能的网卡，并且写入到 status 中

[root@controller1 ~]# kubectl get SriovNetworkNodeState -n sriov-network-operator worker1 -o yaml

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodeState
metadata:
  creationTimestamp: "2023-06-25T07:01:04Z"
  generation: 4
  name: worker1
  namespace: sriov-network-operator
  ownerReferences:
    - apiVersion: sriovnetwork.openshift.io/v1
      blockOwnerDeletion: true
      controller: true
      kind: SriovNetworkNodePolicy
      name: default
      uid: 111e692f-cc3c-40da-aa28-de3a7f8f7c0e
  resourceVersion: "11353566"
  uid: d1bef95a-82c5-4a5c-8eb1-0ff7744eff0f
spec:
  dpConfigVersion: "11351926"
status:
  interfaces:
    - deviceID: "1017"
      driver: mlx5_core
      linkSpeed: 10000 Mb/s
      linkType: ETH
      mac: 04:3f:72:d0:d2:86
      mtu: 1500
      name: enp4s0f0np0
      pciAddress: "0000:04:00.0"
      totalvfs: 8
      vendor: 15b3
    - deviceID: "1017"
      driver: mlx5_core
      linkSpeed: 10000 Mb/s
      linkType: ETH
      mac: 04:3f:72:d0:d2:87
      mtu: 1500
      name: enp4s0f1np1
      pciAddress: "0000:04:00.1"
      totalvfs: 8
      vendor: 15b3
  syncStatus: Succeeded

上面信息说明: 在 worker1 节点上的接口 enp4s0f0np0 和 enp4s0f1np1 具有 SR-IOV 功能，我们可以基于它们配置 VFs，供 Pod 使用。

SriovNetworkNodePolicy： 用于配置 VFs 的数量和安装 sriov-device-plugin 组件

[root@controller1 ~]# kubectl get sriovnetworknodepolicies.sriovnetwork.openshift.io -n sriov-network-operator policy1 -o yaml

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"sriovnetwork.openshift.io/v1","kind":"SriovNetworkNodePolicy","metadata":{"annotations":{},"name":"policy1","namespace":"sriov-network-operator"},"spec":{"deviceType":"netdevice","nicSelector":{"pfNames":["enp4s0f0np0"],"vendor":"15b3"},"nodeSelector":{"kubernetes.io/os":"linux"},"numVfs":4,"resourceName":"sriov_netdevice"}}
  creationTimestamp: "2023-06-25T07:01:28Z"
  generation: 3
  name: policy1
  namespace: sriov-network-operator
  resourceVersion: "11350025"
  uid: b0513a9c-8c64-421d-97cc-d780fd7e8cec
spec:
  deviceType: netdevice
  nicSelector:
    pfNames: # (1)!
      - enp4s0f0np0
  nodeSelector: # (2)!
    kubernetes.io/hostname: 10-20-1-240 # (3)!
  numVfs: 4 # (4)!
  resourceName: sriov_netdevice
  1. PF 的列表，创建 CR 后将基于列表中的 PF 创建指定数量的 VFs
  2. 此 Policy 在哪些节点生效。注：会安装 sriov-device-plugin 组件到指定节点
  3. 只作用于 10-20-1-240 这个节点
  4. 渴望的 VFs 数量