2. Analysis of NFV testing status

• Contributions: 
  • Saad Ullah Sheikh -  Saad contribution-section 3. Analysis of NFV Testing Survey
  • Lei Huang - 2.1- 2.3 analysis of NFV testing status, include testing scenario, process, challenges
  • Beth Cohen - optimize white paper
  • Others?

2.1 NFV testing scenario

Note: NFV testing objects, NFV testing types, main scenarios of traditional NFV testing, e.g. network access test, acceptance test, operation and maintenance test, etc.

2.1.1       Acceptance testing scenario

Acceptance testing is used to verify whether a system can achieve expected effect, including patch testing, acceptance testing in production, etc. Acceptance testing is sometimes called delivery testing, or ATP (Acceptance Test Procedure), UAT (User Acceptance Test), etc. It refers to a test process in which relevant users test the system and decide whether to accept the system according to the test results. The purpose of acceptance testing is to ensure that the system is ready to work as expected.

In NFV field, acceptance testing is mainly led by service providers and operated by vendors. The testing objects include but not limited to hardware, VIM+Hypervisor, MANO, VNF, etc. The content of acceptance testing includes but not limited to functional testing, performance testing, reliability testing, abnormal testing, etc. The number of test cases for acceptance testing varies from dozens to hundreds depending on test objects; the time for acceptance testing is usually controlled within 2 to 3 weeks for a test object.

Before the acceptance testing, vendors usually also conduct relevant test. This test generally covers all acceptance test cases (except for the acceptance test cases temporarily added by service providers).

2.1.2 Centralized procurement testing scenario

Centralized procurement testing is used to verify whether the system conform with the technical requirements of centralized procurement, and the test results are used as technical basis for procurement bidding evaluation. Service providers organize vendors to conduct testing according to the pre-developed test plan before procurement bidding. The centralized procurement testing method is similar to acceptance testing, which includes-

1. Test basis: including technology and test specifications of system under test, and the evaluation criteria of test results.
2. Test objects and topology: including the system under test and surrounding network elements.
3. Testing tool: instruments and tools used for testing.
4. Services and service models: including the various service processes involved, their proportions, and service volume.
5. Test case: including test objectives, preset conditions, test steps and expected results.

2.1.3 Network access testing scenario

The network access testing is a type of test to verify whether the new system and its new release meet the requirements of existing network. The result of network access testing is the basis for issuing network access license. The test method of network access testing is similar to centralized procurement testing and acceptance testing, so the section won’t repeat the content here.

2.2 NFV testing process

Note: Summarize the current test process of each test scenario.

The current NFV testing process include test topology design, test environment deployment, test execution, test result analysis and certification, in which-

1. Test topology design: test topology that needs to be designed before executing test sequence, usually include the the component under test, test equipment, instrument, surrounding components and the network (wired or wireless) connection between them, etc.
2. Test environment deployment: test environment deployment includes hardware environment and software environment. Hardware environment refers to the required servers, clients, network connection equipment, etc.; software environment refers to the environment composed of the operating system, database and other application when the software under test is running.
3. Test execution: utilize a series of operation procedures for the elements in the test environment to simulate specific scenarios such as service processes, information interactions or equipment failures that occur in the actual environment to verify whether the system under test or function under test meets the expected specific functions, performance, reliability, interface compliance and other requirements.
4. Test result analysis and certification: analyze test execution results and and locate problems, then put the tested objects that have passed test certification into the certification market.

2.3 Challenges in NFV testing

Note:According to the survey analysis, describe some of the main NFV testing challenges.

After the introducing of NFV, the resource pool tests and network element tests are usually conducted separately. And as pairing tests are needed between resource pools and network elements, they increase test types and frequency.

In addition, due to the requirement of new functions of the network or service and the upgrade of software itself, the upgrade cycle of NFV software-based network elements is shortened to 2~3 weeks or 1~2 months compared with the half-year upgrade cycle of traditional physical network elements. Which means upgrade cycle is shortened.

To sum up, service providers have the requirement of frequently upgrade NFV. Currently, some major limiting factors make this requirement impossible to achieve, including:

1. Service provider’s test environment is in short supply
2. Long approval process for access test
3. Insufficient stability of the production environment
4. Personnel, management, and equipment testing costs are too high
5. and many other such factors

At the same time, from the perspective of equipment, instrument, environment, personnel, etc., there are many difficulties include diversification of network elements and interfaces, complex instruments, problem location, large test capacity, large demand for personnel, and high skill requirements, etc. Specifically, we use the 5G core test as an example, the testing challenges include-