Status: DRAFT

Overview

This is an attempt to map the ecosystem Open Source projects in the OVP Phase 2 umbrella and their relationships.  The goal is to link the source of “what to validate” (i.e. requirements) to either testing specifications (i.e. Certifications) or testing implementations.  A project being in a box only implies it is potentially aligned to provide the functionality – not that it already has it or we confirmed alignment.

Additionally, we are analyzing areas where there is ambiguity or lack of clarity, and recommending alternative approaches to either eliminate the issue or make the scope clearer. 

Mapping (Current Categories)

Category

Sub Category

Requirements

Conformance

Test Impl./Tools

Notes

Cloud Platform Conformance

Performance/NF

???

???

???

Doesn’t appear to be in RA-2


Functional

RA-2

RC-2

???

Are there tools to specifically validate RA-2?


Cloud Native

RA-2

RC-2

CNCF Software Conformance (K8S compatibility only)

CNF Conformance

CNF Conformance has defined limited tests against the K8S instance itself (CNI and K8S APIs)

CNF On-boarding

CNF Compliance

CNF Conformance

ONAP VNF Requirements

RC-2

CNF Conformance

CNF Conformance is not specific to CNTT nor part of LFN

ONAP CNF support is limited and ONAP is not currently a requirement for a CNF to utilize an RI based on RA-2. 


CNF LCM

CNF Conformance

RC-2

CNF Conformance


CNF Conformance

Performance

???

???

???

Should this all include Non-Functional as we did with Cloud Platform Conformance?  Unclear what we are testing against here.


Functional

???

RC-2

???

Need better definition of what this would encompass.  Are we checking that a NF does the function (ex: firewall) that it supposed to perform?


Cloud Native

CNF Conformance

RC-2

CNF Conformance

CIS Benchmark


General Questions/Comments/Concerns on Current Categorization

  • Cloud Platform Performance - Do we envision that there will be specific requirements related to the performance of NFVI expressed as some minimum level of performance? UIf so where do we envision such expectations being documented? Alternatively, do we see this primarily as a benchmarking exercise where reference CNFs and tests will be run to produce results? If so, is this really in the scope of OVP or should benchmarking be a separate activity?
  • CNF Onboarding and CNF Conformance don't appear to be separate concerns.  They are still functions of the CNF's compliance with CNTT and general Cloud Native/K8S requirements.  Given the current architecture of RA-2, there is also not a separate onboarding activity that is separate from instantiation - this is a difference from the VNF badge with ONAP.  Recommend collapsing these to a single category.
  • Functional vs. Cloud Native - This distinction is not clear, and there are likely to be overlaps between the 2 categories.  Working with the assumption that CNTT will be the source of requirements, if there is going to be a breakdown it must be very clear what the distinction is or align to a breakdown in the source requirements (i.e. RA-2).  Recommend collapsing these categories.
  • CNF Performance - This type of testing is often very specific to the function of the CNF.  We're not clear that a general purpose framework could be created nor requirements could be defined.  We recommend at a minimum this is outside the MVP scope, and it may be outside of OVP scope as well.

Other Open Questions/Concerns

  • No project in the LFN umbrella is currently defining requirements for what a CNF needs to comply with. The only project identified is the CNF Conformance project which is part of CNCF TUG. Given that a CNF will need to be compatible with a given CNTT RA and RI, it would seem there needs to be a project somewhere in CNTT to manage those requirements and potentially tests.  If there are more general Cloud Native requirements that aren't specific to the CNTT, then we could create a requirement to comply with those external requirements and test suites.
  • There was mention of using ONAP as a vehicle to define CNF onboarding requirements, but this presents some challenges:
    • ONAP’s support of CNFs is fledgling and still under development. As such, no requirements currently exist for CNFs in ONAP and they are likely go under significant change as support is built out.
    • Additionally, ONAP is not required for a CNF to utilize an RA-2 based implementation.
    • Suggestion: We need a place within CNTT to define the requirements of a CNF to be compatible with RA-2. ONAP may define additional requirements to onboard a CNF into ONAP for orchestration and management, but that would be a separate and optional set of requirements that would only be applicable when using ONAP.  This may impact how the badges are defined as well.
  • RC-2 does not plan to handle certification of the CNFs, but rather the NFVIs.  However, the diagram on the OVP 2.0 Boot Strap page implies that RC-2 does play a role.  We either need to change the scope of RC-2 or correct the diagram.  
    • Answer: Per discussion with RC-2 lead, we agreed that some level of CNF requirements can be included in RC-2 (this would someone align with the approach for RC-1).  However, unlike RC-1 there are no external CNF-specific requirements within LFN to point to.  In RC-1, they were pointed to ONAP testing and requirements specific to the VNF.

Suggested Revised Categorization

  • Cloud Platform Compliance
    • Performance/Non-Functional - Does the NFVI meet specific performance benchmarks using reference CNFs , and does it handle specific non-functional requirements
      • Note: This assumes that we envision there will be specific performance requirements.  If this is not true, then this category should be revisited.
    • Functional - Does the NVFI support the required capabilities as defined by RA-2
      • Note: If further sub-categorization is required, then it should be tied to categorization of requirements in CNTT
  • CNF Conformance
    • Artifact Compliance (images, descriptors, charts, etc.) - Does the CNFs charts, configurations, descriptors, images, etc. meet CNF requirements as defined in CNTT?
    • Functional - Can the CNF perform specific general purpose LCM activities on a RA-2 compliant NFVI.
      • Note 1: This is not testing the the Network Function behavior itself (i.e. it operates properly as a firewall) which is out of scope of OVP
      • Note 2: If further sub-categorization is required (security, , then it should be tied to categorization of requirements in CNTT

Mapping (Proposed Categories)

Category

Sub Category

Requirements

Conformance

Test Impl./Tools

Notes

Cloud Platform Conformance

Performance/Non-Functional

???

???

???

Where are non-functional requirements documented?

Are requirements for performance expected?  Should benchmarking against ref. CNFs be it's own track?


Functional

RA-2

RC-2

CNCF Software Conformance (K8S compatibility only)

CNF Conformance

Current tools are not currently linked to RA-2, but CNF Conformance is analyzing alignment.

Are there other tools/projects that will test specific requirements?

CNF Conformance primary focus so far has been on testing the CNF itself, although I see some tests specific to the NFVI.  Do we see this suite testing both NFVI and CNFs?

CNF Conformance

Artifact Compliance (images, descriptors, charts, etc.)

CNF Conformance

RC-2

CNF Conformance

We would still need additional requirements specific to RA-2, and potentially more general purpose telco requirements.  Where would those be documented?  RA-2, RC-2, or somewhere else?


Functional

???

RC-2

???

This testing would not cover the functional behavior of the CNF (e.g. is it a firewall), but rather can the CNF handle standard LCM operations or utilize capabilities of RA-2 based NFVI properly.

There is no place in CNTT or any LFN project where such requirements are defined for CNFs

  • No labels

4 Comments

  1. Bill Mulligan

    Open Questions/Issues

    • RC-2 does not plan to handle certification of the CNFs, but rather the NFVIs.  However, the diagram on the OVP 2.0 Boot Strap page implies that RC-2 does play a role.  We either need to change the scope of RC-2 or correct the diagram.  
      • From Rabi - Although RC2 will mainly focus on the NFVI, Conformance of CNFs running on top of the NFVI is also covered. this will however look at how CNFs interact with the infrastructure (using the standard defined interfaces). Although the scope of the RC2 will mainly be infrastructure, it will also cover CNF conformance as well (to be limited to how CNFs are consuming infra resources). so it might make sense to clarify this to the RC2 Scope
    • CNF Conformance currently has tests for CNFs and the Kubernetes platform itself (ex: CNI tests, K8S API tests). CNF Providers and NFVI providers are different use cases. Should we split these tests?
      • Aren't these all individual tests right now anyways so they can be split up and recombined as needed?
    • No project in the LFN umbrella is currently defining requirements for what a CNF needs to comply with. The only project identified is the CNF Conformance project which is part of CNCF TUG. Given that a CNF will need to be compatible with a given CNTT RA and RI, it would seem there needs to be a project somewhere in CNTT to manage those requirements and potentially tests.  If there are more general "Cloud Native" requirements that aren't specific to the CNTT, then we could create a requireement to comply with those external requirements and test suites.
      • I agree and think this should be brought back to the larger group. Should this be part of LNF or does it lie outside its scope?
    • There was mention of using ONAP as a vehicle to define CNF onboarding requirements, but this presents some challenges:
      • ONAP’s support of CNFs is fledgling and still under development. As such, no requirements currently exist for CNFs in ONAP and they are likely go under significant change as support is built out.
      • Additionally, ONAP is not required for a CNF to utilize an RA-2 based implementation.
      • Suggestion: We need a place within CNTT to define the requirements of a CNF to be compatible with RA-2. ONAP may define additional requirements to onboard a CNF into ONAP for orchestration and management, but that would be a separate and optional set of requirements that would only be applicable when using ONAP.  This may impact how the badges are defined as well.
        • See the above response from Rabi. It seems RA2 could define how CNF should be compatible with the infrastructure while RC2 could be used for verification. This approach seems similar to CNF testbed that tests both infrastructure and NFs.
    • Testing Category Feedback
      • The 2 categories of CNF Conformance and CNF Onboarding seem to significantly overlap. Onboarding artifacts would seem to be just another aspect of CNF Conformance rather than a category itself. 
        • I agree. Can they be combined into one category?
      • In Cloud Platform Compliance, the distinction between the categories Cloud Native and Functional are not clear to me. Is a distinction necessary? RA-2 already has requirements such as the implementation must be compatible with K8S software conformance tests. If we are going to have sub categories of compliance, then I suggest we better align them with RA-2 categories (ex: Capabilities, Architecture, etc.)
        • I agree. I don't think we actually need sub categories for cloud native and functional. However, I do think performance should be left as its own category. 
      • Under CNF Conformance, it's unclear what Functional means.  Does it mean the CNF performs the function it is designed to do (e.g. firewall) or does it mean it is compatible with specific lifecycle functions common to all CNFs
        • I think it is the latter the it functions like a CNF should. With this in mind then this is the same as for infrastructure above that functional and cloud native should just be combined into one category. 
      • Suggested Categorization:
        • Cloud Platform Compliance
          • Performance/Non-Functional - Does the NFVI meet specific performance benchmarks, and handle specific non-functional requirements
          • RA-2 Capabilities Alignment - Does the NFVI include all mandatory capabilities defined in RA-2
          • RA-2 Architecture Alignment - Does the NFVI meet the mandatory architecture requirements defined in RA-2
            • Do capabilities and architecture need to be separate? See above
        • CNF Conformance
          • Artifact Compliance (images, descriptors, charts, etc.) - Do the charts, configurations, descriptors, images, etc. meet CNF requirements
          • Functional - need a better definition of the scope of this
          • Performance - not sure what would be validated here.  It wouldn't seem to be a matter of conformance.   Maybe CNF Benchmarking is something that should be in it's own category.

    TO DO:

    Clarify RC2 Scope regarding CNFs


    1. Trevor Lovett

      Thanks, Bill.  Re: the scope of RC-2, I've updated the mapping table to increase the scope of RC-2.  Please review and let me know if that is a better reflection.  I did review RC-1, and it does have sections dedicated specifically to VNF verification/validation, so I assume that will be carried over to RC-2.  The actual requirements of the tests performed are very high-level, and ultimately refer back to ONAP requirements for the bulk of the heavy lifting.  So while RC-2 will cover CNF testing, I think we'll still have to solve for what CNF requirements are we going to validate and where those are defined.

      Re: the CNF Conformance tests, they are individual, but I think we will have to ensure the individual tests are organized into a testing profile for easy selection and execution.  That could be separate projects or simply separate testing profiles that are selected at run time.  

      I've only spent a little time looking at CNF testbed.  If you're more familiar with it, maybe you could speak a bit about it on our next working session so we understand how it fits into the picture.

      As for the categorizations, I'm open to other ways to do it.  What I laid out was mostly a straw-man proposal aimed at simplifying the current structure. My hope is we can align on a proposal as a work stream, and take it back to the larger group. I'd like to dedicate some time to this on our next meeting, then potentially bring it to the next CVC call.

  2. Bill Mulligan

    For the Cloud Platform Compliance, I think the Performance/Non-Functional category lies outside the scope of CNTT and OVP. Performance seems to be more between the vendor and operator and I think it would get hard for any of them to actually publish numbers even if we came up with tests.


    For CNF Conformance, if we want to define conformance, there is currently a massive gap in RA2 because it is not addressed there at all. The conformance can be done in RC2, but the requirements need to come from somewhere so this is a gap that we need to bring back to RA2.

    1. Trevor Lovett

      I would tend to agree with you on Performance and Non-Functional.  Let's discuss on our weekly call how to address this.

      For CNF Conformance, I had a typo in the table, CNF Functional requirements should have been listed as an open instead of listing RA-2.  I've updated it.