Date: Fri, 29 Mar 2024 04:43:56 +0000 (UTC) Message-ID: <153452650.29642.1711687436014@aws-us-west-2-lfn-confluence-1.web.codeaurora.org> Subject: Exported From Confluence MIME-Version: 1.0 Content-Type: multipart/related; boundary="----=_Part_29641_1531106743.1711687436014" ------=_Part_29641_1531106743.1711687436014 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Location: file:///C:/exported.html
This document is divided into two sections. The first sect= ion is focusing on the baremetal node installer & the second secti= on on the Kubernetes/software installers.
This section describes how to set up and configure a bare metal pr= ovisioner for RI-2. These investigations could be used as an input t= o the wiki of RI-2.
Provisioning is the operation of installing OS on a given infrastructure= before the software stack can be installed on them. For the purpose of the= se investigations, a former OPNFV baremetal provisioner XCI, now referred t= o as Cloud Infra Automation Framework and hosted by Nordix Labs [1] is used= .
This framework uses Bifrost for provisioning virtual and baremetal nodes= and supports online and offline deployments. Bifrost is a set of Ansible p= laybooks that automates the task of deploying a base image onto a set of kn= own hardware using ironic [2].
The lab is hosted in OPNFV and is setup in accordance with the lab princ= iples/requirements defined in CNTT RI-2 Chapter 3.
Before initiating a deployment, two configuration templates, referred to= as POD Descriptor File (PDF) and Installer Descriptor File (IDF) in OPNFV = terminology need to be defined. Both PDF and IDF files are modelled as yaml= schema.
A PDF is a hardware configuration template that includes hardware charac= teristics of the jumphost node & the set of compute/controller nodes. F= or each node, the following characteristics should be defined:
IDF extends the PDF with the network information required by the install= er. All the networks along with possible VLAN, DNS and gateway information = should be defined here.
After ensuring that the lab and provisioner requirements are met, genera= te SSH keypair, add user to the sudo group & have passwordless sudo ena= bled. After this the deployment can be initiated by cloning the repo, navig= ating to the engine directory & running the deploy command
git clone https://gerrit.nordix.org/infra/engine.= git
cd engine/engine
./deploy.sh -o <OStype>-p file:///<pdf.yaml> -i file:///<= idf.yaml> -l provision
Currently, Ubuntu 18.04 & CentOS 7.8 are supported (def= ault Ubuntu 18.04). Support for other system can be added as well depending on the requirements.
The engine supports offline deployment as well. <Steps to follow><= /p>
After the successful completion of the deployment, one need= s to setup host networking for K8s, etc. to be able to run software p= rovisioning tooling from CNF Testbed or Intel=E2=80=99s BMRA playbooks to c= onfigure and install k8s (& other plugins) on the provisioned nodes (e.g. creating VLAN=E2=80=99s, setting up internet connectivity, etc. = =E2=80=93 left to the choice of operator/vendor).
CNF Testbed tooling
After provisioning nodes & configuring networking, one can run software provisioning tooling from CNF Testbed to setup and c= onfigure k8s.
Intel BMRA tooling
After provisioning nodes & configuring networking, down= load Intel container kit repo & update the inventory, etc with your des= ired configuration (refer to Prepare the BMRA software section below for mo= re details).
Then run the following command to provision k8s & other plugins/feat= ures
sudo docker run --rm -v $(pwd):/bmra -v ~/.ssh/:/root/.ssh/ -ti bmra-ins= tall:centos ansible-playbook -i /bmra/inventory.ini /bmra/playbooks/cluster= .yml
The above BMRA installation works if you have provisioned your nodes usi= ng the BM provisioner described above. For pre-provisioned nodes, refer to = the section below.
The following is based on configuration and installation outside of OPNF= V Intel Lab.
The OS used for Jump, Master and Worker nodes is CentOS 7.8.2003= (3.10.0-957.12.2.el7.x86_64)
Prior to installing BMRA, log on the worker node and check the hardware = configuration. This information is used when configuring BMRA later.
Start by installing pciutils, which is used by Ansible and needed when g= athering information:
$ yum install pciutils
CPU:
$ lscpu
Interfaces / PFs:
$ ip a
$ lspci | grep Eth
$ lspci -ns <bus:device.function>
Start by installing tools needed for running the BMRA playbook:
$ yum update
$ yum =
install git epel-release python36 python-netaddr
$ yum=
install python-pip
$ pip install ansible=3D=3D2.7.16 jmesp=
ath
$ pip install jinja2 --upgrade
Prepare the BMRA software:
$ git clone https://github.com/intel/container-experience-kits.git
$ cd container-experience-kits/
$ git checkout &=
lt;tag>
- v1.4.1 (If using Kubernetes 1.16)
- v1.2.1 (If using =
Kubernetes 1.14)
$ git submodule update --init
$ cp -r examples/group_va=
rs examples/host_vars .
Update inventory.ini
to match the the hardware an=
d size of deployment. A minimal setup can look as follows:
[all]
master1 ansi=
ble_host=3D<master_ip> ip=3D<master_ip>
node1 a=
nsible_host=3D<worker_ip> ip=3D<worker_ip>
[kube-master]
maste=
r1
[etcd]
master1
[kube-node]
node1=
code>
[k8s-cluster:children]
kube-node
[calico-rr]
Update host_vars/node1.yml
(and any additional fi=
les depending on inventory.ini
):
sriov_enabled
- Change to true if VFs should be crea=
ted during installationsriov_nics
- Update with interface names (PF) from t=
he node. Number of VFs and driver can be changed toovpp_enabled
& ovs_dpdk_enabled
=
- Disable one or both depending on need for a vSwitch. Using both mig=
ht cause issues.force_nic_drivers_update
- Set to false if SSH connec=
tion to machine uses interface with i40e or i40evf driver (otherwise connec=
tion to node is likely to be broken)install_ddp_packages
- Set to false if DDP (Dynamic Device Personalization) should not be i=
nstalled isolcpus
- Change according to HW and need for isolat=
ed cores/treads. Also relevant for CMK configuration (see below)sst_bf_configuration_enabled
- Consider setting this=
to false unless platform and HW supports itUpdate group_vars/all.yml
according to hardware a=
nd capabilities:
cmk_hosts_list
- Update according to inven=
tory.ini
, e.g. only node1 for the above examplecmk_shared_num_cores
& cmk_exclusive_=
num_cores
- Update according to available cores/threads and th=
e number of isolated cores
sriovdp_config_data
- Update according to the =
sriov_nics
host_vars config (see above)
qat_dp_enabled
- Change to false if HW doesn't suppo=
rt QAT, either through chipset of PCI addongpu_dp_enabled
- Change to false if not supported on=
HWtas_enabled
- Can be set to false if not neededtas_create_policy
- Set to false if TAS is not enable=
d cluster_name
- Can be changed to something more spec=
ific than "cluster.local"Once the necessary files have been updated, BMRA can be installed
tmux
or screen=
code> session to prevent installation issues due to disconnects
ansible-playbook -i inventory.ini playboo=
ks/cluster.yml
Once installation is complete, you can decide if you will access the clu= ster from the master node, of from the jump/installer node.
kubectl get nodes
scp <master_ip>:.ku=
be/config kubeconfig
kubectl get nodes