• Contributions: 
    • Saad Ullah Sheikh -  Saad contribution-section 6.Test Case automation and intelligence using ONAP ; CNCF conformance
    • Beth Cohen -Optimize content, cut it down to less pages(using diagrams or other methods)
    • Lei Huang- 3.1-3.3 automated testing status, industry status, requirements; will cut it down to less pages(using diagrams or other methods)
    • Cecilia Corbi - 3.2.1 Research status of NFV automated testing in standard organizations - ENI?
    • Others?

3.1  NFV automated testing status

Note:Analyze the current status of NFV automated testing.

At present, the degree of automation of NFV testing by service providers are still relatively low, and there are a lot of cross-departmental and cross-organizational communication, manual errors are inevitable. The problems that still need to be resolved include:

  1. Automatic configuration and deployment of test environment
  2. Integration of test scripts from different vendors
  3. Automatic control of test process
  4. Integration of test tools/test instruments from different vendors and traceability of test results, etc.

At the same time, operators have currently introduced DevOps tools. All operators hope to achieve automatic network element code construction, automatic integration, automatic network element life cycle and service testing, automatic network element deployment and online full-process automatic closed loop, thereby speeding up the network elements development, testing and the speed of online process.

Regarding the DevOps cooperation model, the current model between service providers and VNF vendors is usually that VNF vendors provide VNF software packages and operators implement CI/CD in their own DevOps environment. There is a requirements about how to load the VNF software package into the operator's DevOps environment automatically.

3.2  Research status of NFV automated testing in various organizations

Note: Investigate existing research content, progress and status of automated testing in various industry organizations, including standard organizations, open source communities, etc.

3.2.1 Research status of NFV automated testing in standard organizations

Note: Research status of NFV automated testing in various standard organizations, including ETSI NFV TST, ZSM, 3GPP, ENI, etc.

  • ETSI NFV TST

Note: Briefly introduce the main work content of the ETSI NFV TST working group, and provide relevant information.

(Contribution- CMCC, others?)

  • ETSI ISG ZSM

Note: Briefly introduce the main work content of the ETSI ISG ZSM, and provide relevant information.

(Contribution- CMCC, others?)

  • 3GPP SA5

Note: Briefly introduce the main work content of the 3GPP SA5, and provide relevant information.

(Contribution- CMCC, others?)

  • Other contributions?...

3.2.2 Research status of NFV automated testing in open source communities

Note: Research status of NFV automated testing in various open source communities, including ONAP, OPNFV, CDF, CNCF conformance, etc.

(One contribution from Saad- 6.Test Case automation and intelligence using ONAP)

  • ONAP

In ONAP Guilin release, China Mobile led automated testing requirement. This requirement focuses on the establishment of a general automated testing platform by introducing automated testing tools and processes in each testing phases.

The requirement analyzed that in traditional network element, system or equipment network access test, there are a large number of cross-department and cross-organization communications. And the manual errors are inevitable, the knowledge in test field cannot be solidified. The cost of each test is high and the test cycle is always long.

After introducing NFV, because network element software and hardware equipment are layered decoupled, the introduction of a large number of open source components as well as the frequent upgrade of the software itself, make network access test become more complicated and frequent. Testing has become a bottleneck during the introduction and iteration of new technologies. Therefore, it is urgent to introduce automated test tools and processes, including test topology auto design, test environment auto deploy, test task auto execution, and test result auto analysis & certification. Thereby reducing labor costs, improving test efficiency and test accuracy.

By enhancing each module of ONAP, the overall framework of automated testing is realized respectively, including-

  1. Enhance SDC to implement the auto design of test topology
  2. Enhance VF-C to implement test environment auto deploy
  3. Enhance VTP to implement test task auto execution
  4. Enhance VTP and integrate with OVP to implement test result auto analysis & certification

After enabling this automated testing framework, CSPs can leverage it to accelerate time to deployment for new network services, improve interoperability and software quality, and reduce in-house testing effort and reduce costs. Vendor can leverage it to deduce the cost of VNF developing, and improve time to revenue for new product offerings, achieve greater alignment with customer requirements from service provider, and demonstrate product quality through open ecosystem testing. Instrument manufacturers can integrate their test tools and test capabilities into this platform, and demonstrate product quality through open ecosystem testing. Integrators can refer to the open source implementation of this platform and provide solutions for their commercial products. At the same time, the realization of this requirement can be combined with OVP's existing processes to accelerate the testing and certification of OVP NFV commercial products.

  • OPNFV

Note: In OPNFV, briefly introduce projects related to automated testing, including Functest, Dovetail, Airship, CIRV, Xtesting, etc.

(Contribution- CMCC, others?)

  • Openstack ZUUL

Note: Briefly introduce Openstack ZUUL project.

(Contribution- CMCC, others?)

  • CDF

Note: Briefly introduce CDF(Continuous Delivery Foundation).

(Contribution- CMCC, others?)

  • CNCF conformance

(Contribution- STC, Saad)

  • Other contributions?...

3.3 NFV automated testing requirements

Note: According to current situation of automated testing, further analyze the requirements of automated testing from the perspective of test scenarios or test objects.

The test is carried out in the way of "test framework + test instrument/tool + test object", including preparation of environment and deployment, configuration, preparation and execution of service parameters, test cases, as well as test process observation, and provision of test reports. In general, NFV automated testing requirements should include-

  1. Test environment: NFV automated testing should support automated deployment processes including network configuration and network element instantiation, etc.
  2. Test configuration: Including the test framework/tool and service configuration of the tested object, it should support the unified distribution of the configuration during the automated test processes.
  3. Test execution: Automated execution of test tasks, providing a flexible automated test framework.
  4. Test process observation: The test framework/tool should support real-time monitoring of the test processes to facilitate understanding of the execution of test cases.
  5. Test report: The test framework/tool should support test data aggregation and provide customized test reports.
  6. Test result analysis: The test framework/tool should support automated analysis and certification of test results and automatic release of certified objects.
  7. Testing Tool- appeal to Restful API:
    1. The specified test suite can be loaded.
    2. Test suites/test cases can be executed regularly or in real time.
    3. Observation points can be set to observe the test process in real time.
    4. Can obtain the original test log and set which indicators to obtain to support customized test reports.
  8. Network element: It should support the issuance of remote configuration files and provide interfaces to facilitate access to key indicators of network elements.
  9. Network element life cycle testing and service function testing need to support automated deployment of network element, integration of test scripts from different vendors, and automated control of test procedures. At the same time, test tool/instrument integration of different vendors and test results must be traceable.
  10. DevOps integration requirements: the current model between service providers and VNF vendors is usually that VNF vendors provide VNF software packages and operators implement CI/CD in their own DevOps environment. There is a requirements about how to load the VNF software package into the operator's DevOps environment automatically.

(Others?...)

  • No labels