Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

This is an attempt to map the ecosystem Open Source projects in the OVP Phase 2 umbrella and their relationships.  The goal is to link the source of “what to validate” (i.e. requirements) to either testing specifications (i.e. Certifications) or testing implementations.  A project being in a box only implies it is potentially aligned to provide the functionality – not that it already has it or we confirmed alignment.

Additionally, we are analyzing areas where there is ambiguity or lack of clarity, and recommending alternative approaches to either eliminate the issue or make the scope clearer. 

Mapping (Current Categories)

Category

Sub Category

Requirements

Certifications

Conformance

Test Impl./Tools

Notes

Cloud Platform Conformance

Performance/NF

???

???

???

Doesn’t appear to be in RA-2


Functional

RA-2

RC-2

???

Are there tools to specifically validate RA-2?


Cloud Native

RA-2

RC-2

CNCF Software Conformance (K8S compatibility only)

CNF Conformance

CNF Conformance has defined limited tests against the K8S instance itself (CNI and K8S APIs)

CNF On-boarding

CNF Compliance

CNF Conformance

ONAP VNF Requirements

???

RC-2

CNF Conformance

CNF Conformance is not specific to CNTT nor part of LFN

ONAP CNF support is limited and ONAP is not currently a requirement for a CNF to utilize an RI based on RA-2. 


CNF LCM

CNF Conformance

???

RC-2

CNF Conformance


CNF Conformance

Performance

???

???

???

Should this all include Non-Functional as we did with Cloud Platform Conformance?  Unclear what we are testing against here.


Functional

???

???

RC-2

???

Need better definition of what this would encompass.  Are we checking that a NF does the function (ex: firewall) that it supposed to perform?


Cloud Native

CNF Conformance

???

RC-2

CNF Conformance

CIS Benchmark

...


General Questions

...

/Comments/Concerns on Current Categorization

  • Cloud Platform Performance - Do we envision that there will be specific requirements related to the performance of NFVI expressed as some minimum level of performance? UIf so where do we envision such expectations being documented? Alternatively, do we see this primarily as a benchmarking exercise where reference CNFs and tests will be run to produce results? If so, is this really in the scope of OVP or should benchmarking be a separate activity?
  • CNF Onboarding and CNF Conformance don't appear to be separate concerns.  They are still functions of the CNF's compliance with CNTT and general Cloud Native/K8S requirements.  Given the current architecture of RA-2, there is also not a separate onboarding activity that is separate from instantiation - this is a difference from the VNF badge with ONAP.  Recommend collapsing these to a single category.
  • Functional vs. Cloud Native - This distinction is not clear, and there are likely to be overlaps between the 2 categories.  Working with the assumption that CNTT will be the source of requirements, if there is going to be a breakdown it must be very clear what the distinction is or align to a breakdown in the source requirements (i.e. RA-2).  Recommend collapsing these categories.
  • CNF Performance - This type of testing is often very specific to the function of the CNF.  We're not clear that a general purpose framework could be created nor requirements could be defined.  We recommend at a minimum this is outside the MVP scope, and it may be outside of OVP scope as well.

Other Open Questions/Concerns

  • No project in the LFN umbrella is currently defining requirements for what a CNF needs to comply with. The only project identified is the CNF Conformance project which is part of CNCF TUG. Given that a CNF will need to be compatible with a given CNTT RA and RI, it would seem there needs to be a project somewhere in CNTT to manage those requirements and potentially tests.  If there are more general " Cloud Native " requirements that aren't specific to the CNTT, then we could create a requireement requirement to comply with those external requirements and test suites.
  • There was mention of using ONAP as a vehicle to define CNF onboarding requirements, but this presents some challenges:
    • ONAP’s support of CNFs is fledgling and still under development. As such, no requirements currently exist for CNFs in ONAP and they are likely go under significant change as support is built out.
    • Additionally, ONAP is not required for a CNF to utilize an RA-2 based implementation.
    • Suggestion: We need a place within CNTT to define the requirements of a CNF to be compatible with RA-2. ONAP may define additional requirements to onboard a CNF into ONAP for orchestration and management, but that would be a separate and optional set of requirements that would only be applicable when using ONAP.  This may impact how the badges are defined as well.
    Testing Category Feedback
  • The 2 categories of CNF Conformance and CNF Onboarding seem to significantly overlap. Onboarding artifacts would seem to be just another aspect of CNF Conformance rather than a category itself. 
  • In Cloud Platform Compliance, the distinction between the categories Cloud Native and Functional are not clear to me. Is a distinction necessary? RA-2 already has requirements such as the implementation must be compatible with K8S software conformance tests. If we are going to have sub categories of compliance, then I suggest we better align them with RA-2 categories (ex: Capabilities, Architecture, etc.)
  • Under CNF Conformance, it's unclear what Functional means.  Does it mean the CNF performs the function it is designed to do (e.g. firewall) or does it mean it is compatible with specific lifecycle functions common to all CNFs
  • Suggested Categorization:
  • RC-2 does not plan to handle certification of the CNFs, but rather the NFVIs.  However, the diagram on the OVP 2.0 Boot Strap page implies that RC-2 does play a role.  We either need to change the scope of RC-2 or correct the diagram.  
    • Answer: Per discussion with RC-2 lead, we agreed that some level of CNF requirements can be included in RC-2 (this would someone align with the approach for RC-1).  However, unlike RC-1 there are no external CNF-specific requirements within LFN to point to.  In RC-1, they were pointed to ONAP testing and requirements specific to the VNF.

Suggested Revised Categorization

  • Cloud Platform Compliance
    • Performance/Non-Functional - Does the NFVI meet specific performance benchmarks using reference CNFs , and does it handle specific non-functional requirements
      • Note: This assumes that we envision there will be specific performance requirements

...

      • .  If this is not true, then this category should be revisited.
    • Functional - Does the NVFI support the required capabilities as defined by

...

    • RA-2

      ...

          • Note: If further sub-categorization is required, then it should be tied to categorization of requirements in CNTT
      • CNF Conformance
        • Artifact Compliance (images, descriptors, charts, etc.) -

      ...

        • Does the CNFs charts, configurations, descriptors, images, etc. meet CNF requirements as defined in CNTT?
        • Functional -

      ...

        • Can the CNF perform specific general purpose LCM activities on a RA-2 compliant NFVI.
          • Note 1: This is not testing the the Network Function behavior itself (i.e. it operates properly as a firewall) which is out of scope of OVP
          • Note 2: If further sub-categorization is required (security, , then it should be tied to categorization of requirements in CNTT

      Mapping (Proposed Categories)

      Category

      Sub Category

      Requirements

      Conformance

      Test Impl./Tools

      Notes

      Cloud Platform Conformance

      Performance/Non-Functional

      ???

      ???

      ???

      Where are non-functional requirements documented?

      Are requirements for performance expected?  Should benchmarking against ref. CNFs be it's own track?


      Functional

      RA-2

      RC-2

      CNCF Software Conformance (K8S compatibility only)

      CNF Conformance

      Current tools are not currently linked to RA-2, but CNF Conformance is analyzing alignment.

      Are there other tools/projects that will test specific requirements?

      CNF Conformance primary focus so far has been on testing the CNF itself, although I see some tests specific to the NFVI.  Do we see this suite testing both NFVI and CNFs?

      CNF Conformance

      Artifact Compliance (images, descriptors, charts, etc.)

      CNF Conformance

      RC-2

      CNF Conformance

      We would still need additional requirements specific to RA-2, and potentially more general purpose telco requirements.  Where would those be documented?  RA-2, RC-2, or somewhere else?


      Functional

      ???

      RC-2

      ???

      This testing would not cover the functional behavior of the CNF (e.g. is it a firewall), but rather can the CNF handle standard LCM operations or utilize capabilities of RA-2 based NFVI properly.

      There is no place in CNTT or any LFN project where such requirements are defined for CNFs