Skip to end of metadata
Go to start of metadata

This is a work in progress to define the minimum viable/valuable product (MVP) for a program to validate the life-cycle of a VNF running "in/with" ONAP.

The MVP definition should not discourage anyone from contributing to other projects or efforts related to VNF testing, but should help guide developer priorities for test frameworks.

Requirements & Deliverables 

  1. The goal of the MVP is to specify a suite of VNF validation tests and associated test infrastructure that can be developed within the time-frame of the E-release of ONAP.
  2. Tests will focus on on-boarding and instantiation of the VNF in/within ONAP.
  3. Tests will focus on HEAT and TOSCA based VNFs.
  4. Test will leverage the SDNC and APPC, VFC controllers.
  5. Tests will use the the VIM that is provided with ONAP i.e. a generic version of OpenStack. (a future program update / release may migrate to other VIMs).
  6. VNFs are validated against the specific release of ONAP (i.e. VNF is validated against ONAP E-release or F-release).
  7. VNFs must also past the current VNF criteria defined by ONAP for the initial release of VNF compliance testing.
  8. Testing uses existing interfaces into ONAP for stimulus / response to "drive" the tests (i.e. avoid creating new requirements / interfaces for the ONAP at large project).

What is needed / Work to do

  1. ONAP release must be readily deploy-able, to allow a test framework to run on the deployment and test a VNF.  
  2. ONAP deployment must be reproducible to ensure VNF tests are conducted in a uniform environment.
  3. Definition of life cycle requirements (i.e. what the test cases validate)
    1. Both HEAT & TOSCA 
      1. What does a mean (formally for a test case requirement) to "instantiate" (or startup) a VNF?

        1. from Victor, Definition of VNF onboarding for TOSCA based VNF.

          • Verify the VNF package by using VNFSDK compliance check test cases.
          • upload a VNF package to ONAP system
          • verify the VNF Package already exist on the ONAP system.
        2. from Victor, Definition of VNF instantiation for TOSCA based VNF.

          • Trigger an "instantiate VNF" operation to Controller
          • verify that the requested grant for the "instantiate VNF" operation has been approved by the controller
          • VNF related software images have been successfully added to the image repository managed by the VIM or already exist on the VIM system(Currently, ONAP is manually doing this.)
          • Verify that the requested virtualised resources have been allocated by the VIM according to the VNFD
          • Verify that virtualised resource allocation constraints have been met by querying the VIM
          • Verify that any existing virtualised resources have not been affected by the allocation of the new virtualised resources by querying the VIM
          • Verify that the VNF instance resources are visible on the controller
      2. Does instantiation include configuration?
        1. Victor: both heat and Tosca template could include the inject-file/user-data(cloud-init) as the day0 configuration file of a VNF.  
        2. Trevor: I would argue against configuration for the MVP phase.  There are a variety of options to execution post-instantiation configuration management (Ansible, Chef, etc.).   Since there are multiple options for a compliant VNF to utilize it would not make sense to offer only a subset of options, and offering any option would involve setup, configuration, and execution of the configuration mgmt option itself. 
      3. Does instantiation include health check?
        1. Victor: Could be yes, we could use the query interface (query the VNF detail info) to implement the health-check func.
      4. HEAT
        1. Trevor Lovett & Ryan Hallahan will help with this.
        2. The following is a DRAFT - needs further review and input
        3. Prerequisite Decision Points 
          1. Are we planning for decentralized (VNF Provider must setup their own ONAP instance and peform their own testing) or centralized testing (VNF Provider is submitting their VNF and associated settings for testing)  model?  Assuming the former, but need confirmation as the later has additional automation and other concerns to care for (network creation, image registration, etc.)
        4. Assumptions 
          1. Test scripts will utilize a default license model in SDC
          2. Compliance with test suite assumes the VNF has already has compliant packaging certified
          3. If utilizing full automated testing, then assume the Service model will only include a single VF for instantiation.  It would likely be too difficult to automatically create a multi-VF service in the first phase, and not necessary for the scope of testing
          4. Test Input will include the preload information in some TBD file format that can be loaded into SDNC.
        5. Implementation Options 
          1. Within ONAP today, the Integration Testsuite generally has all the building blocks to automate and execute the steps below.   It would need some modification to handle a more arbitrary VNF as it is not used in this way today, but based on preliminary analysis this looks achievable. 
          2. Orange Python Framework - This is another option that can be considered.  It looks to handle the instantiation use case, and potentially others.  
        6. Potential Test Flow
          1. Intialize Vendor and Category Information
          2. Create the VSP in SDC
          3. Upload Heat Archive
          4. (Optional) Assign any Unassigned Files to Artifacts
          5. Validate the VSP and ensure now Errors exist (warnings are OK)
          6. Assign the Vendor License Model to the VSP (assumes a single VLM for testing purposes)
          7. Create the Virtual Function
            1. Import the VSP (find using Name or ID from prior steps)
            2. Set name of VF (auto-assign or make input into test script), contact and other required fields
          8. Create Service
            1. Set Name (auto-assign based on VSP or make input into test script)
            2. Assign required or optional fields based on test script input
            3. Assign VF to the Servce Model
          9. Distribute the Service Model and validate successful Distribution
          10. Submit Preloads to SDNC
          11. Trigger Instantiation of Base Module from VID (NOTE: Need to see how we handle multi-module VNFs - presumably we can query this information and instantiate each individully)
          12. Verify sucessfull instantiation
          13. Health-check TBD - needs further discussion
      5. TOSCA
        1. Victor Gao will help with this.
        2. from Victor, Definition of VNF instantiation for TOSCA based VNF.

          • Trigger an "instantiate VNF" operation to Controller
          • verify that the requested grant for the "instantiate VNF" operation has been approved by the controller
          • VNF related software images have been successfully added to the image repository managed by the VIM or already exist on the VIM system(Currently, ONAP is manually doing this.)
          • Verify that the requested virtualised resources have been allocated by the VIM according to the VNFD
          • Verify that virtualised resource allocation constraints have been met by querying the VIM
          • Verify that any existing virtualised resources have not been affected by the allocation of the new virtualised resources by querying the VIM
          • Verify that the VNF instance resources are visible on the controller
    2. Test case description template for specifying VNF validation test purpose, implementation steps and pass/fail criteria.
    3. Definition of the set of ONAP components and their configuration required for the testing (ONAP profile used for testing).
    4. Definition of test infrastructure requirements needed for testing (i.e. hardware with compute / network / storage and pod requirements to run testing).
    5. Test tooling that drives testing through existing ONAP interfaces.
      1. HEAT
      2. TOSCA

What is needed / Work to do Matrix

Test Life Cycle

#Step (Common)HEAT SpecificTOSCA Specific
1

Initialize Vendor and Category Information

via SDCN/A- already in SOL001 VNF Descriptor
2

Create the VSP in SDC

via SDCvia SDC
3

Upload Archive

Heat ArchiveETSI SOL004 CSAR File
4

(Optional) Assign any Unassigned Files to Artifacts

via SDCError or warning since package should match manifest
5

Validate the VSP and ensure no Errors exist (warnings are OK)

via SDCvia VNFSDK
6

Assign the Vendor License Model to the VSP (assumes a single VLM for testing purposes)

via SDCvia SDC
7Create Virtual Function - Import the VSP (find using Name or ID from prior steps)via SDCN/A- already in SOL001 VNF Descriptor
8

Create Virtual Function - Set name of VF (auto-assign or make input into test script), contact and other required fields

via SDCN/A- already in SOL001 VNF Descriptor
9Create Service - Set Name (auto-assign based on VSP or make input into test script)via SDCvia SDC
10

Create Service - Assign required or optional fields based on test script input

via SDCvia SDC
11

Create Service - Assign VF/VNF to the Service Model

via SDCvia SDC
12

Distribute the Service Model and validate successful Distribution

via SDC → DMaaPvia SDC → DMaaP
13

Submit Preloads

via SDNCvia SDNC
14

Trigger Instantiation of Base Module from VID (NOTE: Need to see how we handle multi-module VNFs - presumably we can query this information and instantiate each individually)

via VID

via VID to SO & SOL003 adapter or VFC & SOL003 adapter

15

Verify successful instantiation

Verify Heat Stack Create Successful

Ping Ports on OAM network

Verify VNF created successfully

Ping ports on OAM network

Items to do 

Based on a review with the ONAP Integration team the test suite Robot scripts provide the majority of the building blocks to perform the automation required for Heat-based VNFs this effort.   There is still work to adapt the existing scripts to handle a generic VNF vs. the predefined demo VNFs currently used.  The amount of effort on a per function basis as laid out in the table is not known at this time, but the overall effort does look to be achievable in the El Alto time frame.

#itemHEAT specificsexist?resources neededTOSCA specificsexist?resources needed
1Update VNFREQTS for LCM definitionRequirements for VNF "life-cycle" will be the same for HEAT / TOSCA.70%VNFREQTS TeamRequirements for VNF "life-cycle" will be the same for HEAT / TOSCA.50%VNFREQTS Team
2Automation Script(s) to on-board VSPIntegration TestSuiteYesContributions to Integration project  by VVP team.Victor: Investigating to reuse the existing scripts.~80%VNFSDK Team
3Automation Script(s) to Create VFIntegration TestSuiteYesContributions to Integration project  by VVP team.Victor: Investigating to reuse the existing scripts.~80%VNFSDK Team
4Automation Script(s) to Create ServiceIntegration TestSuiteYesContributions to Integration project  by VVP team.Victor: Investigating to reuse the existing scripts.~80%VNFSDK Team
5Automation Script(s) to Submit PreloadsIntegration TestSuiteYesContributions to Integration project  by VVP team.SDN-C Specific Operation: TOSCA could be ignoredN/A
7Automation Script(s) to Instantiate VNFIntegration TestSuiteYesContributions to Integration project  by VVP team.Need to develop new scriptsNoVNFSDK Team
8Automation Script(s) to Healthcheck VNFN/A - not planned for this phaseN/A
Nice to haveN/A
9Clean up after test(s)Implemented directly in TestSuite
TestSuite TeamImplemented directly in TestSuite
TestSuite Team


Open Questions

  1. Can the test requirements or definitions (procedures) by pulled from, or reuse, the ETSI TST-0007
    1. What are the integration and testing interfaces that are currently available, i.e. used by the integration team / gating team?

Definition of Done / Success Measures

  1. Tests can readily be run, with high level of repeatability.
    1. Level of complexity is manageable by end users (i.e. ease of ONAP deployment + test cases).

EUAG Feedback

Please place your feedback here, as needed.

  1. Feedback from CMCC: The VNFD we are using in our company are all TOSCA-based. Also, we are using VFC for VNF LCM. We suggest to update the 3rd item and the 4th item in "Requirements & Deliverables" to  "Tests will focus on HEAT based VNFs and TOSCA based VNFs" and "Test will leverage the SDNC, APPC and VFC controllers".
    1. Verizon feedback: As mentioned in the comment below we we are using TOSCA based VNF-D as well. We would like to extend the requirements to include SOL004/SOL001 VNFs using the SDC→SO→SOL003 Adapter → External VNFM→VNF path or the SDC → SO → SOL005 Adapter → VF-C → VNF path.
    2. China Telecom Feedback:We have both HEAT based and TOSCA based VNFDs in the DEMO running on our testbed. It will be great if TOSCA based VNFs could be added in the Requirement, providing the resources are available.
    3. ChinaUnicom Feedback: Our company also has the strong demands for TOSCA based VNF and we hope that it could be included into the scope.

Timeline

DateDeliverable
April 19, 2019MVP agreed by the CVC.
April 23, 2019Presentation of MVP to LFN EUAG during teleconference
End AprilMVP agreed / finalized (feature freeze)
Late April / Early MayMeetings with development/technical teams to determine what currently exists and what needs to be proposed as new work.
Late MayDevelopment plans finalized with technical teams.
June 13, 2019ONAP E-release M1
June - JulyVNF requirements created for life-cycle
July 18, 2019ONAP E-release M2/M3
JulyTest case development, per requirements set
AugustTest tooling development
August 29, 2019ONAP E-release
SeptemberBeta testing from E-release, requirements frozen / completed, test case and tooling bug fixes only
OctoberBeta conclusions, first VNFs publicly listed as passing the validation testing

External Resources

https://wiki.onap.org/display/DW/OVP+LCM+Support

Testing framework comparison

  • No labels

9 Comments

  1. For information, the OOM gating (gating of the ONAP installer - it means check of patchset before merge) already integrates some VNF testing as part of the non regression tests (https://wiki.onap.org/display/DW/OOM+Gating).

    It is up&running since February and you can see the results in OOM gerrit pages e.g. https://gerrit.onap.org/r/#/c/83149/ (search for ONAP JobDeployer).

    Please note that the VNF tests are run only if the core healthcheck tests are OK, so they are not systematically triggered, especially on Master branch)

    These VNF tests are leveraging xtesting framework developped  in OPNFV.

    It means that the way the ONAP VNF tests are designed, integrated in CI, launched and managing results (same API, same DB result data model) is similar than Functest (Cedric Ollivier made a presentation on that at ONS 2019 and during the OPNFV plugfest in Paris).


    Regarding the requirements:

    RequirementsCurrent situation
    The goal of the MVP is to develop validation testing within the time-frame of the E-release of ONAPtests already running since Casablanca, integrated in gating for Dublin
    Tests will focus on on-boarding and instantiation of the VNF in/within ONAP.

    End to End test cases include the onboarding, the check of the model distribution, the instantiation and the check that the VNF has been properly created on the targeted cloud

    it includes also the cleaning of the resource (currently a bug in Casablanca preventing full resources to be cleaned, announced to be fixed in Dublin)

    Tests will focus on HEAT based VNFs.

    The tests are based on a python framework https://gitlab.com/Orange-OpenSource/lfn/onap/onap-tests integrated in xtesting.

    See https://wiki.onap.org/pages/viewpage.action?pageId=6593670&preview=%2F6593670%2F54722733%2Fonap_tests.pdf for details.

    It may consume any heat template.The tool has been designed to simplify the onboarding/instantiation steps for VNF providers.

    basic_vm: a simple VM

    freeradius_nbi: simple VM on which we install a freeradius server - the goal here was to test the instantiation through the NBI module

    clearwater_ims: the traditionnal vIMS use cases (already tested in OPNFV/Functest with cloudify and heat) - here we just test the onboarding/deployment (signaling tests not integrated yet)

    vfw: the canonical vFW that can also be tested through the official Robot scripts

    Test will leverage the SDNC and APPC controllers.

    Today the tests leverage SDNC (for the post configuration via SDNC preloads), No test with APPC.

    Tests will use the the VIM that is provided with ONAP.That is the usual way (easier for CI) even if it is possible to configure the tool to use any Cloud platform 
    NFs are validated against the specific release of ONAP (i.e. VNF is validated against ONAP E-release or F-release).

    That is the case today. Based on xtesting, we can manage the upper constraints. Today we have a casablanca and a master docker file to test the ONAP platform accordingly

    See https://cloud.docker.com/repository/docker/morganrol/xtesting-onap-vnf/tags

    VNFs must also past the compliance testing (i.e. testing defined in the initial release of VNF testing).

    In ONAP test repository, we added a linter for that.

    All the heat templates stored in the repository are systematically tested with VVP (version a bit old need to be updated with the official docker).

    In other words, if you try to commit a non VVP compliant heat file, you will get notified (and your pull request may be rejected).

    Testing uses existing interfaces into ONAP for stimulus / response to "drive" the test.No sure to undestand this point


    And regarding the What is needed / Work to do section



    Definition of life cycle requirements (i.e. what the test cases validate)
    1. Does instantiation include configuration?
    2. Does instantiation include health check?

    see the previous pdf (P8 & 11)

    Instantiation includes configuration at 2 levels

    • infrastructure configuration (using shade)
    • VNF configuration (SDNC preload)

    Note for the moment we use only the ONAP VNF API, evolutions are planned to support also the new GR API

    Instantiation doe not include healthcheck but it should be the case. Leveraging VNF abstraction provided in xtesting, we should follow the rule: deploy the orchestrator/deploy the VNF/test the VNF

    for the IMS it is planned to add the signaling test section (as it is already the case in OPNFV/functest for heait/ims and cloudify/ims)

    Test case description template for VNF validation work

    We are using the same format than the one defined in OPNFV and storing the results the same way

    see http://testresults.opnfv.org/onap/api/v1/projects/functest/cases

    Definition of the set of ONAP components required for the testing (ONAP profile used for testing).

    For the gating we are deploying a full ONAP, which is pretty big.

    But it is possible to deploy different flavors core/small/medium/full (healthcheck are defined for these different categories)

    For our End to End tests today, core should be possible (no tests with policies and loop yet)

    Definition of requirements needed for testing (i.e. hardware / pod requirements to run testing).

    relevant questions not adressed for the moment.

    On test case definition, it could be done through tags and/or installer/scenario constraints but it supposes that we have choices on target clouds for testing, which is not the case today.


    Please note that there are several test projects in ONAP, this End to End testing is related to the integration + OOM Gating project

    It is not not linked to VTP, VNFSDK, OTP, some discussions were planned during the ONS, see also JIRA discussion https://wiki.onap.org/pages/viewpage.action?pageId=43386304


    Cedric Ollivier Eric Debeau

    1. Regarding, Test projects alignment in ONAP:

      Today we have VVP and VNFSDK as apporved projects under ONAP, which handles the VNF related testing and in dublin, we have unified the tools and made it available over VNF Test platform (VTP) which is currently integrated in SDC for VSP compliance check before on-boarding the VNF and dovetail for running the test cases in ONAP for OVP. so I believe we have unified the testing tools in ONAP in dublin release. 

      Regarding LCM support from ONAP VNF Test Platform (VTP)

      In dublin release, ONAP community is working on to enable all required CLI for doing end-end testing and is progressing well. so we will have all required commands to do the operation on  ONAP, which will be super set of LCM operations, which are mentioned here.  And VNF Test Platform (VTP) works very well with the ONAP commands with approx. zero development effort, i.e, these commands could be directly used in the VTP to perform all LCM operations over ONAP. Please refer the available commands for the ONAP till today, here https://github.com/onap/cli/tree/master/products/onap-dublin/features


      1. Hi Rabi Abdel  Lincoln Lavoie  Victor Gao

        I have captured the relization of the LCM thru the VTP and documented here, https://wiki.onap.org/display/DW/OVP+LCM+Support This covers the heat and tosca based ones and samples are provided https://github.com/onap/integration/blob/master/test/hpa_automation . IMHO, These automation could be levaraged for LCM testing.


  2. Morgan Richomme - This looks really good.  Would it be possible to line a review of this framework on the Victor's joint VVP/VNFSDK/CVC call on Monday's.  I'd certainly benefit from an overview of this, and what your thoughts are on how this could be applied to externally submitted VNFs through the compliance program (it looks very configurable already).  


    cc - Victor Gao Steven Wright

    1. sure, we can plan a dedicated topic for the next meeting.

  3. My company is invested in deploying ETSI NFV SOL004/SOL001 TOSCA VNFs.  If this effort gets pushed to the Frankfurt release, we would like to extend the requirements to include SOL004/SOL001 VNFs using the SDC→SO→SOL003 Adapter → External VNFM→VNF.  How can we help enable that path?

    An alternative path would be to use the VF-C VNFM so that an external VNFM would not need to be deployed first, ie SDC → SO → SOL005 Adapter → VF-C → VNF.  However, I don't think that this path will work in the general case, since the VNFs that we have deployed have vendor specific VNFM dependencies.

    1. AFAIK, VF-C also includes third party VNFM adapters, for vender specfic VNFM requirement, I think another path can be SDC → SO → SOL005 Adapter → VF-C →External VNFM→VNF

  4. Regarding HEAT vs TOSCA, i think there is a need to confirm if there is/will be equivalent definition of LCM of TOSCA based VNFs as there is for HEAT today within VNFRQTS, equivalent automation/test scripts to onboard, deploy, and test TOSCA based VNFs as there is today for HEAT based ones on time for El-Alto release and what resources are available/committed for this.

    Not we were happy about it, but based on the discussions we had during ONS and the f2f EUAG session, the assumption was there isn't/will not be an equivalent support, therefore, the thinking was to use what we have today for MVP.

     My understanding is that to add TOSCA, we need the following: 

    • VNFREQTS will need to have explicit support for TOSCA based testing in their LCM definitions (although some of existing definitions apply to both, there are more tailored toward HEAT based VNFs today).
    • Automation scripts for TOSCA based VNFs onboarding/deployment are needed: scripts that is able to automatically onboard TOSCA based VNFs onto ONAP and deploy them into OpenStack.
    • VNF test script to configure/health check/communicate with VNFs are available. (pass/fail criteria)

    Let's discuss today!

  5. ETSI NFV is currently finalizing maintenance related specifications. There should be constraints from EM - VNFM - VIM to have graceful maintenance operations. I am building the Fenix project to have the generic rolling infrastructure maintenance in interaction with EM/VNFM and implementing the ETSI standard. However, this standard needs support up in the mano in VNFM and EM (And Fenix with the orchestration). So it has an impact on validating VNF when implemented. There is already a couple of people working with Tacker to look into EM - VNFM  and further to VIM (Fenix) interface.