4. NFV automated testing evaluation index model

Contributions: 

• Lei Huang - Index model content , cut it down to less pages
• Beth Cohen - Optimize content, cut it down to less pages
• Others?

1.Personnel capacity: Personnel capacity constitutes the basis of NFV automated testing. Testers are required to have the overall grasp of NFV system architecture, familiarity with the tested network elements and interfaces, and corresponding specifications (in the initial stage, 5G core network main network elements such as AMF/SMF/UPF ), the testing capabilities of open source and commercial tools required for NFV testing, and the development capabilities of automated scripting languages (Restful API, Python, etc.).

A reasonable test framework design and professional testers with corresponding capabilities can form the following ability matrix. According to product and project planning, you can find a suitable project starting point in the corresponding matrix in the figure below, and later use industry ecosystem tools, solutions or endogenous development momentum, expand to other quadrant dimensions.

2. Basic environmental capabilities: NFV testing is inseparable from a resource-rich, stable and reliable basic platform. The flexibility and dynamic scalability required by NFV testing need to be considered during the preparation of the basic environment.

Based on the current status of the NFV business, with the openstack cloud platform as the main infrastructure, it is necessary to consider a server and switch networking structure that is as consistent as possible with the existing network environment, in order to be able to fully verify the problems found in the test process before the business goes online , To avoid the deviation of the test results caused by the different hardware between the test environment and the existing network environment.

Considering the crossover of multiple software and hardware combinations tested in the NFV laboratory phase, it is necessary to plan different hardware devices in different resource pools, including server configuration and network switching equipment configuration, and to traverse through automated means when necessary. The difference in test results under this combination of software and hardware even exposes potential problems caused by compatibility. A topology management mechanism needs to be introduced between resource pools composed of various hardware to improve the efficiency of automatic execution and the ability of automatic traversal execution under multiple combinations of software and hardware.

3. Network element and test tool capabilities: NFV testing different network elements requires the testing capabilities of different tools. From a functional perspective, some open source tools can meet some basic functional testing requirements. However, due to the peculiarities of NFV services, it still requires the use of the industry for many years. Some commercial test tools can meet the stability and accuracy that the test tools can provide in the performance test scenarios of large-scale, large number of users and large traffic; the function test part of some network elements can be considered as open source tools + commercial tools to carry out.

The deployment capability of tools is also a key issue that needs to be considered in the design of NFV automated test framework. Commercial test tools usually adopt dedicated hardware form due to the consideration of stability and high performance, which contradicts the software and elastic characteristics of NFV testing to a certain extent. At this time, dynamic topology management tools are needed to achieve the rapid networking topology construction capabilities required for temporary test environment construction.

Software-based tools must have "cloud native" features, and at least can be deployed flexibly through virtual machine forms. Currently, virtualized test tools usually have little problems with hardware compatibility, so you can consider taking a part from the basic test resource pool. Server resources are used for the deployment of cloud-based test tools, and can even be deployed together with the tested network elements in a resource pool during automated testing to avoid network performance bottlenecks in a "multi-cloud" environment.

4.Test tool and system ability to support cloud environment: Cloud deployment of test tools is usually not a problem, but the network planning between the NFV test framework and the tested environment may introduce some additional complexity. The test scheduling execution module, test execution machine, test machine instance, there is a certain degree of system connectivity between the license management or test machine controller, etc. and the tested system or network element, and the large amount of test execution pressure formed by the cross-combination of various software and hardware will inevitably require the NFV test framework to have the ability to execute multiple tests in parallel. Planning resources such as network hardware, topology, IP, VLAN, etc. in advance is a primary condition for the smooth execution of NFV automated test execution.