This is a work in progress to define the minimum viable/valuable product (MVP) for a program to validate the life-cycle of a VNF running "in/with" ONAP.
The MVP definition should not discourage anyone from contributing to other projects or efforts related to VNF testing, but should help guide developer priorities for test frameworks.
Requirements & Deliverables
- The goal of the MVP is to specify a suite of VNF validation tests and associated test infrastructure that can be developed within the time-frame of the E-release of ONAP.
- Tests will focus on on-boarding and instantiation of the VNF in/within ONAP.
- Tests will focus on HEAT and TOSCA based VNFs.
- Test will leverage the SDNC and APPC, VFC controllers.
- Tests will use the the VIM that is provided with ONAP i.e. a generic version of OpenStack. (a future program update / release may migrate to other VIMs).
- VNFs are validated against the specific release of ONAP (i.e. VNF is validated against ONAP E-release or F-release).
- VNFs must also past the current VNF criteria defined by ONAP for the initial release of VNF compliance testing.
- Testing uses existing interfaces into ONAP for stimulus / response to "drive" the tests (i.e. avoid creating new requirements / interfaces for the ONAP at large project).
What is needed / Work to do
- ONAP release must be readily deploy-able, to allow a test framework to run on the deployment and test a VNF.
- ONAP deployment must be reproducible to ensure VNF tests are conducted in a uniform environment.
- Definition of life cycle requirements (i.e. what the test cases validate)
- Both HEAT & TOSCA
What does a mean (formally for a test case requirement) to "instantiate" (or startup) a VNF?
from Victor, Definition of VNF onboarding for TOSCA based VNF.
- Verify the VNF package by using VNFSDK compliance check test cases.
- upload a VNF package to ONAP system
- verify the VNF Package already exist on the ONAP system.
from Victor, Definition of VNF instantiation for TOSCA based VNF.
- Trigger an "instantiate VNF" operation to Controller
- verify that the requested grant for the "instantiate VNF" operation has been approved by the controller
- VNF related software images have been successfully added to the image repository managed by the VIM or already exist on the VIM system(Currently, ONAP is manually doing this.)
- Verify that the requested virtualised resources have been allocated by the VIM according to the VNFD
- Verify that virtualised resource allocation constraints have been met by querying the VIM
- Verify that any existing virtualised resources have not been affected by the allocation of the new virtualised resources by querying the VIM
- Verify that the VNF instance resources are visible on the controller
- Does instantiation include configuration?
- Victor: both heat and Tosca template could include the inject-file/user-data(cloud-init) as the day0 configuration file of a VNF.
- Trevor: I would argue against configuration for the MVP phase. There are a variety of options to execution post-instantiation configuration management (Ansible, Chef, etc.). Since there are multiple options for a compliant VNF to utilize it would not make sense to offer only a subset of options, and offering any option would involve setup, configuration, and execution of the configuration mgmt option itself.
- Does instantiation include health check?
- Victor: Could be yes, we could use the query interface (query the VNF detail info) to implement the health-check func.
- HEAT
- Trevor Lovett & Ryan Hallahan will help with this.
- The following is a DRAFT - needs further review and input
- Prerequisite Decision Points
- Are we planning for decentralized (VNF Provider must setup their own ONAP instance and peform their own testing) or centralized testing (VNF Provider is submitting their VNF and associated settings for testing) model? Assuming the former, but need confirmation as the later has additional automation and other concerns to care for (network creation, image registration, etc.)
- Assumptions
- Test scripts will utilize a default license model in SDC
- Compliance with test suite assumes the VNF has already has compliant packaging certified
- If utilizing full automated testing, then assume the Service model will only include a single VF for instantiation. It would likely be too difficult to automatically create a multi-VF service in the first phase, and not necessary for the scope of testing
- Test Input will include the preload information in some TBD file format that can be loaded into SDNC.
- Implementation Options
- Within ONAP today, the Integration Testsuite generally has all the building blocks to automate and execute the steps below. It would need some modification to handle a more arbitrary VNF as it is not used in this way today, but based on preliminary analysis this looks achievable.
- Orange Python Framework - This is another option that can be considered. It looks to handle the instantiation use case, and potentially others.
- Potential Test Flow
- Intialize Vendor and Category Information
- Create the VSP in SDC
- Upload Heat Archive
- (Optional) Assign any Unassigned Files to Artifacts
- Validate the VSP and ensure now Errors exist (warnings are OK)
- Assign the Vendor License Model to the VSP (assumes a single VLM for testing purposes)
- Create the Virtual Function
- Import the VSP (find using Name or ID from prior steps)
- Set name of VF (auto-assign or make input into test script), contact and other required fields
- Create Service
- Set Name (auto-assign based on VSP or make input into test script)
- Assign required or optional fields based on test script input
- Assign VF to the Servce Model
- Distribute the Service Model and validate successful Distribution
- Submit Preloads to SDNC
- Trigger Instantiation of Base Module from VID (NOTE: Need to see how we handle multi-module VNFs - presumably we can query this information and instantiate each individully)
- Verify sucessfull instantiation
- Health-check TBD - needs further discussion
- TOSCA
- Weitao Gao will help with this.
from Victor, Definition of VNF instantiation for TOSCA based VNF.
- Trigger an "instantiate VNF" operation to Controller
- verify that the requested grant for the "instantiate VNF" operation has been approved by the controller
- VNF related software images have been successfully added to the image repository managed by the VIM or already exist on the VIM system(Currently, ONAP is manually doing this.)
- Verify that the requested virtualised resources have been allocated by the VIM according to the VNFD
- Verify that virtualised resource allocation constraints have been met by querying the VIM
- Verify that any existing virtualised resources have not been affected by the allocation of the new virtualised resources by querying the VIM
- Verify that the VNF instance resources are visible on the controller
- Test case description template for specifying VNF validation test purpose, implementation steps and pass/fail criteria.
- Definition of the set of ONAP components and their configuration required for the testing (ONAP profile used for testing).
- Definition of test infrastructure requirements needed for testing (i.e. hardware with compute / network / storage and pod requirements to run testing).
- Test tooling that drives testing through existing ONAP interfaces.
- HEAT
- TOSCA
- Both HEAT & TOSCA
What is needed / Work to do Matrix
Test Life Cycle
# | Step (Common) | HEAT Specific | TOSCA Specific |
---|---|---|---|
1 | Initialize Vendor and Category Information | via SDC | N/A- already in SOL001 VNF Descriptor |
2 | Create the VSP in SDC | via SDC | via SDC |
3 | Upload Archive | Heat Archive | ETSI SOL004 CSAR File |
4 | (Optional) Assign any Unassigned Files to Artifacts | via SDC | Error or warning since package should match manifest |
5 | Validate the VSP and ensure no Errors exist (warnings are OK) | via SDC | via VNFSDK |
6 | Assign the Vendor License Model to the VSP (assumes a single VLM for testing purposes) | via SDC | via SDC |
7 | Create Virtual Function - Import the VSP (find using Name or ID from prior steps) | via SDC | N/A- already in SOL001 VNF Descriptor |
8 | Create Virtual Function - Set name of VF (auto-assign or make input into test script), contact and other required fields | via SDC | N/A- already in SOL001 VNF Descriptor |
9 | Create Service - Set Name (auto-assign based on VSP or make input into test script) | via SDC | via SDC |
10 | Create Service - Assign required or optional fields based on test script input | via SDC | via SDC |
11 | Create Service - Assign VF/VNF to the Service Model | via SDC | via SDC |
12 | Distribute the Service Model and validate successful Distribution | via SDC → DMaaP | via SDC → DMaaP |
13 | Submit Preloads | via SDNC | via SDNC |
14 | Trigger Instantiation of Base Module from VID (NOTE: Need to see how we handle multi-module VNFs - presumably we can query this information and instantiate each individually) | via VID | via VID to SO & SOL003 adapter or VFC & SOL003 adapter |
15 | Verify successful instantiation | Verify Heat Stack Create Successful Ping Ports on OAM network | Verify VNF created successfully Ping ports on OAM network |
Items to do
Based on a review with the ONAP Integration team the test suite Robot scripts provide the majority of the building blocks to perform the automation required for Heat-based VNFs this effort. There is still work to adapt the existing scripts to handle a generic VNF vs. the predefined demo VNFs currently used. The amount of effort on a per function basis as laid out in the table is not known at this time, but the overall effort does look to be achievable in the El Alto time frame.
# | item | HEAT specifics | exist? | resources needed | TOSCA specifics | exist? | resources needed |
---|---|---|---|---|---|---|---|
1 | Update VNFREQTS for LCM definition | Requirements for VNF "life-cycle" will be the same for HEAT / TOSCA. | 70% | VNFREQTS Team | Requirements for VNF "life-cycle" will be the same for HEAT / TOSCA. | 50% | VNFREQTS Team |
2 | Automation Script(s) to on-board VSP | Integration TestSuite | Yes | Contributions to Integration project by VVP team. | Victor: Investigating to reuse the existing scripts. | ~80% | VNFSDK Team |
3 | Automation Script(s) to Create VF | Integration TestSuite | Yes | Contributions to Integration project by VVP team. | Victor: Investigating to reuse the existing scripts. | ~80% | VNFSDK Team |
4 | Automation Script(s) to Create Service | Integration TestSuite | Yes | Contributions to Integration project by VVP team. | Victor: Investigating to reuse the existing scripts. | ~80% | VNFSDK Team |
5 | Automation Script(s) to Submit Preloads | Integration TestSuite | Yes | Contributions to Integration project by VVP team. | SDN-C Specific Operation: TOSCA could be ignored | N/A | |
7 | Automation Script(s) to Instantiate VNF | Integration TestSuite | Yes | Contributions to Integration project by VVP team. | Need to develop new scripts | No | VNFSDK Team |
8 | Automation Script(s) to Healthcheck VNF | N/A - not planned for this phase | N/A | Nice to have | N/A | ||
9 | Clean up after test(s) | Implemented directly in TestSuite | TestSuite Team | Implemented directly in TestSuite | TestSuite Team |
Open Questions
- Can the test requirements or definitions (procedures) by pulled from, or reuse, the ETSI TST-0007
- What are the integration and testing interfaces that are currently available, i.e. used by the integration team / gating team?
Definition of Done / Success Measures
- Tests can readily be run, with high level of repeatability.
- Level of complexity is manageable by end users (i.e. ease of ONAP deployment + test cases).
EUAG Feedback
Please place your feedback here, as needed.
- Feedback from CMCC: The VNFD we are using in our company are all TOSCA-based. Also, we are using VFC for VNF LCM. We suggest to update the 3rd item and the 4th item in "Requirements & Deliverables" to "Tests will focus on HEAT based VNFs and TOSCA based VNFs" and "Test will leverage the SDNC, APPC and VFC controllers".
- Verizon feedback: As mentioned in the comment below we we are using TOSCA based VNF-D as well. We would like to extend the requirements to include SOL004/SOL001 VNFs using the SDC→SO→SOL003 Adapter → External VNFM→VNF path or the SDC → SO → SOL005 Adapter → VF-C → VNF path.
- China Telecom Feedback:We have both HEAT based and TOSCA based VNFDs in the DEMO running on our testbed. It will be great if TOSCA based VNFs could be added in the Requirement, providing the resources are available.
- ChinaUnicom Feedback: Our company also has the strong demands for TOSCA based VNF and we hope that it could be included into the scope.
Timeline
Date | Deliverable |
---|---|
April 19, 2019 | MVP agreed by the CVC. |
April 23, 2019 | Presentation of MVP to LFN EUAG during teleconference |
End April | MVP agreed / finalized (feature freeze) |
Late April / Early May | Meetings with development/technical teams to determine what currently exists and what needs to be proposed as new work. |
Late May | Development plans finalized with technical teams. |
June 13, 2019 | ONAP E-release M1 |
June - July | VNF requirements created for life-cycle |
July 18, 2019 | ONAP E-release M2/M3 |
July | Test case development, per requirements set |
August | Test tooling development |
August 29, 2019 | ONAP E-release |
September | Beta testing from E-release, requirements frozen / completed, test case and tooling bug fixes only |
October | Beta conclusions, first VNFs publicly listed as passing the validation testing |
9 Comments
Morgan Richomme
For information, the OOM gating (gating of the ONAP installer - it means check of patchset before merge) already integrates some VNF testing as part of the non regression tests (https://wiki.onap.org/display/DW/OOM+Gating).
It is up&running since February and you can see the results in OOM gerrit pages e.g. https://gerrit.onap.org/r/#/c/83149/ (search for ONAP JobDeployer).
Please note that the VNF tests are run only if the core healthcheck tests are OK, so they are not systematically triggered, especially on Master branch)
These VNF tests are leveraging xtesting framework developped in OPNFV.
It means that the way the ONAP VNF tests are designed, integrated in CI, launched and managing results (same API, same DB result data model) is similar than Functest (Cedric Ollivier made a presentation on that at ONS 2019 and during the OPNFV plugfest in Paris).
Regarding the requirements:
End to End test cases include the onboarding, the check of the model distribution, the instantiation and the check that the VNF has been properly created on the targeted cloud
it includes also the cleaning of the resource (currently a bug in Casablanca preventing full resources to be cleaned, announced to be fixed in Dublin)
The tests are based on a python framework https://gitlab.com/Orange-OpenSource/lfn/onap/onap-tests integrated in xtesting.
See https://wiki.onap.org/pages/viewpage.action?pageId=6593670&preview=%2F6593670%2F54722733%2Fonap_tests.pdf for details.
It may consume any heat template.The tool has been designed to simplify the onboarding/instantiation steps for VNF providers.
basic_vm: a simple VM
freeradius_nbi: simple VM on which we install a freeradius server - the goal here was to test the instantiation through the NBI module
clearwater_ims: the traditionnal vIMS use cases (already tested in OPNFV/Functest with cloudify and heat) - here we just test the onboarding/deployment (signaling tests not integrated yet)
vfw: the canonical vFW that can also be tested through the official Robot scripts
Today the tests leverage SDNC (for the post configuration via SDNC preloads), No test with APPC.
That is the case today. Based on xtesting, we can manage the upper constraints. Today we have a casablanca and a master docker file to test the ONAP platform accordingly
See https://cloud.docker.com/repository/docker/morganrol/xtesting-onap-vnf/tags
In ONAP test repository, we added a linter for that.
All the heat templates stored in the repository are systematically tested with VVP (version a bit old need to be updated with the official docker).
In other words, if you try to commit a non VVP compliant heat file, you will get notified (and your pull request may be rejected).
And regarding the What is needed / Work to do section
see the previous pdf (P8 & 11)
Instantiation includes configuration at 2 levels
Note for the moment we use only the ONAP VNF API, evolutions are planned to support also the new GR API
Instantiation doe not include healthcheck but it should be the case. Leveraging VNF abstraction provided in xtesting, we should follow the rule: deploy the orchestrator/deploy the VNF/test the VNF
for the IMS it is planned to add the signaling test section (as it is already the case in OPNFV/functest for heait/ims and cloudify/ims)
We are using the same format than the one defined in OPNFV and storing the results the same way
see http://testresults.opnfv.org/onap/api/v1/projects/functest/cases
For the gating we are deploying a full ONAP, which is pretty big.
But it is possible to deploy different flavors core/small/medium/full (healthcheck are defined for these different categories)
For our End to End tests today, core should be possible (no tests with policies and loop yet)
relevant questions not adressed for the moment.
On test case definition, it could be done through tags and/or installer/scenario constraints but it supposes that we have choices on target clouds for testing, which is not the case today.
Please note that there are several test projects in ONAP, this End to End testing is related to the integration + OOM Gating project
It is not not linked to VTP, VNFSDK, OTP, some discussions were planned during the ONS, see also JIRA discussion https://wiki.onap.org/pages/viewpage.action?pageId=43386304
Cedric Ollivier Eric Debeau
Kanagaraj Manickam
Regarding, Test projects alignment in ONAP:
Today we have VVP and VNFSDK as apporved projects under ONAP, which handles the VNF related testing and in dublin, we have unified the tools and made it available over VNF Test platform (VTP) which is currently integrated in SDC for VSP compliance check before on-boarding the VNF and dovetail for running the test cases in ONAP for OVP. so I believe we have unified the testing tools in ONAP in dublin release.
Regarding LCM support from ONAP VNF Test Platform (VTP)
In dublin release, ONAP community is working on to enable all required CLI for doing end-end testing and is progressing well. so we will have all required commands to do the operation on ONAP, which will be super set of LCM operations, which are mentioned here. And VNF Test Platform (VTP) works very well with the ONAP commands with approx. zero development effort, i.e, these commands could be directly used in the VTP to perform all LCM operations over ONAP. Please refer the available commands for the ONAP till today, here https://github.com/onap/cli/tree/master/products/onap-dublin/features.
Kanagaraj Manickam
Hi Rabi Abdel Lincoln Lavoie Weitao Gao
I have captured the relization of the LCM thru the VTP and documented here, https://wiki.onap.org/display/DW/OVP+LCM+Support This covers the heat and tosca based ones and samples are provided https://github.com/onap/integration/blob/master/test/hpa_automation . IMHO, These automation could be levaraged for LCM testing.
Trevor Lovett
Morgan Richomme - This looks really good. Would it be possible to line a review of this framework on the Victor's joint VVP/VNFSDK/CVC call on Monday's. I'd certainly benefit from an overview of this, and what your thoughts are on how this could be applied to externally submitted VNFs through the compliance program (it looks very configurable already).
cc - Weitao Gao Steven Wright
Morgan Richomme
sure, we can plan a dedicated topic for the next meeting.
Fernando Oliveira
My company is invested in deploying ETSI NFV SOL004/SOL001 TOSCA VNFs. If this effort gets pushed to the Frankfurt release, we would like to extend the requirements to include SOL004/SOL001 VNFs using the SDC→SO→SOL003 Adapter → External VNFM→VNF. How can we help enable that path?
An alternative path would be to use the VF-C VNFM so that an external VNFM would not need to be deployed first, ie SDC → SO → SOL005 Adapter → VF-C → VNF. However, I don't think that this path will work in the general case, since the VNFs that we have deployed have vendor specific VNFM dependencies.
shitao li
AFAIK, VF-C also includes third party VNFM adapters, for vender specfic VNFM requirement, I think another path can be SDC → SO → SOL005 Adapter → VF-C →External VNFM→VNF
Rabi Abdel
Regarding HEAT vs TOSCA, i think there is a need to confirm if there is/will be equivalent definition of LCM of TOSCA based VNFs as there is for HEAT today within VNFRQTS, equivalent automation/test scripts to onboard, deploy, and test TOSCA based VNFs as there is today for HEAT based ones on time for El-Alto release and what resources are available/committed for this.
Not we were happy about it, but based on the discussions we had during ONS and the f2f EUAG session, the assumption was there isn't/will not be an equivalent support, therefore, the thinking was to use what we have today for MVP.
My understanding is that to add TOSCA, we need the following:
• VNFREQTS will need to have explicit support for TOSCA based testing in their LCM definitions (although some of existing definitions apply to both, there are more tailored toward HEAT based VNFs today).
• Automation scripts for TOSCA based VNFs onboarding/deployment are needed: scripts that is able to automatically onboard TOSCA based VNFs onto ONAP and deploy them into OpenStack.
• VNF test script to configure/health check/communicate with VNFs are available. (pass/fail criteria)
Let's discuss today!
Tomi Juvonen
ETSI NFV is currently finalizing maintenance related specifications. There should be constraints from EM - VNFM - VIM to have graceful maintenance operations. I am building the Fenix project to have the generic rolling infrastructure maintenance in interaction with EM/VNFM and implementing the ETSI standard. However, this standard needs support up in the mano in VNFM and EM (And Fenix with the orchestration). So it has an impact on validating VNF when implemented. There is already a couple of people working with Tacker to look into EM - VNFM and further to VIM (Fenix) interface.