Overview

This page is intended to capture discussions and work relating to the longer term strategy and direction of testing within the LFN programs.

Discussion:

conformance vs compliance.

Standard vs Specs.

Testing Scopes

Note, the following is a work in progress.

Compliance (to a specification or standard):

Definition:

• A "device" is compliant if it meets the requirements defined within a specification, i.e. a power plug is required to have 3 prongs and would be consider compliant if it indeed has 3 prongs in the correct locations and orientations.  

VNF Compliance

• Conformance to HEAT,  TOSCA, or HELM Templates, as defined by ONAP Requirements.

• Conformance to ONAP VNFRqmts documentation which will integrate other SDO requirements (ie, 5G, BBF, ETSI,...)
• Conformance to ONAP interfaces.

• Documentation of VNF is available.

NFVI/VIM Compliance:

• ETSI ISG NFV interface standards (or a subset of those requirements)
• Conformance to well accepted API definitions and operation (i.e. OpenStack or K8s APIs)
• Open Questions:
  • Should the compliance tests apply to both software and hardware?
    • Hardware example: NUMA compliance, DPDK support, ODP compliance
  • Support DPDK and/or FDIO
  • Address additional specification (e.g. GSMA profiles)
• We need a standard to comply against.
  • what is it? GSMA profile. 

Validation:

Definition:

• A "device" is validated when it operated according to its intended purpose or design, i.e. a power plug is able to connect to a socket that is also known to be compliant and passes electricity, both have been validated (it really works).

VNF Validation:

• • On-boarding: Validate the VNF Artifact(s) is able to be "consumed" by an ONAP instance.

    • Open questions: 1) What are the requirements defined for "On-boarding?" What is the requirement for the "ONAP instance?" 3) Is there a requirement for backwards compatibility? 4) Besides ONAP are there other organizations defining this?

    • Open question: 5) ONAP instance: does it mean MANO, or does it include VNFM.

  • Instantiation: Validate the VNF can be "launched" by an ONAP instance on top of an NFVI.

    • Open questions: 1) What is the requirements for ONAP Instance? 2) What is the requirements for the NFVI? 3) What are the requirements for the "VNF launch?"

  • Validate various life-cycle operations are possible with the VNF running on the ONAP Instance.

    • Open questions: 1) What the required "life-cycle operations?" 2) What the requirements for each "operation?" 3) What are the requirements for the "ONAP instance?"

    • Life Cycle Operations  of VNF (long term vision) - my version of FCAPS:
    • Fault Management
    • Capacity Management
    • Billing/Accounting Management
    • Performance Management
    • Security Management 
    • Open question: does it relate to MANO, or does it include VNFM (generic vs specific).:
• NFVI Validation:
  • Instantiate NFVI/VIM
  • Life Cycle of NFVI/VIM - Healthcheck?

Performance:

• NFVI Performance

  • Capacity: of RAM, CPU, storage

  • Stability:

    • Chaos monkey tests?
  • data throughput, CPU performance, RAM/Storage access performance?
  • What is my reference implementation of NFVI? GSMA Profiles (B, N, and C). ETSI ISG NFV has a performance working group having specs. Led by Al Morton, AT&T. Also I think Pierre Lynch, IXIA, leads the testing spec group also.
  • What are the metrics we need to measure? (align with ETSI, GSMA defined metrics).
    • 1 test per metric?
• VNF Performance.
  • huge dependency on NFVI. need reference NFVI to compare against. (GSMA Profiles).
  • what are the metrics we need to measure? (align with ETSI, GSMA defined metrics).
    • 1 test per metric?
  • VNF Reference Implementation. 

Security:

• Open question: Where do these requirements come from? Who is responsible for the test cases?

Slide presented/created in Paris 2019 F2f