Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.

titleTable of Contents

Table of Content Zone

Table of Contents



The intention of this page is to outline what CNTT "success" looks like by mid-April @ ONES-NA 2020, along with high level tracking items that need to be completed.

Success is defined as:

  1. TBD - documentation created ? test suites defined ? test suites executed ?
  2. TBD
  3. TBD


To-Do (In General)

This is a comprehensive list of work that can be done. A subset of the list below will go into  Baldy (and will be saved in the release notes for Baldy).

A: To be included in Baldy Release.

B: Nice to include in Baldy Release.

C: Low Priority for Baldy Release.

Reference Model

Mark Shostak to add general to-do list in here.


Target Date

( // )

0A: ms RM cleanup (ch content complete 80%, issue closure)

  • B: 1.x Various: May need to be rationalized w/ new Tech Steering document

  • A: 1.9 Roadmap: This section needs perpetual maintenance (strongly consider moving the roadmap out of the RM, altogether)

  • A: RA Compliance/traceability methodology 

  • A: Still needs a crisp executive summary identifying the problem statement, and the two key methodologies employed to address them (VNF Abstraction and a common NFVI)

RM-Ch01 - Intro

  • A: RA Compliance/traceability methodology 
    • Needs to be Chapter 8 content (move to)
  • B: Needs a community aligned list of workloads the platform is intended to support and their priority, which can then be used as a basis for resource allocation and for weighting “suggestions” (notably “The public cloud does it this way, so we should too.”; understood, but we need an objective basis. Right now, it becomes a subjective debate)
  • A: Align and execute on Prague proposal to park the Compute Intensive flavor, and supplement it by removing "over-subscription?" from the Basic flavour
  • C: Flesh out quantitative details about supported workloads. (nice to have)

RM-Ch02 - VNF Requirements & Analysis
  • A: Needs tighter coupling w/ Container paradigm (i.e. decoupling from VM-based VNF)
    • flavours for example might not be applicable to containers(CPU and memory, disc we want to find a way for containers) - possibly Ch04
    • harmonisation of template in Model.
  • C: Needs a general review. Given the experience we’ve gained, can be made more succinct and more usable by a VNF developer
  • C: Need to clearly explain slicing (and contrast to Tenant) from CNTT perspective
  • B: Template attributes should be vetted w/ underlying capabilities, rationalized for duplication (i.e. CNTT value add, vs. what’s already in HEAT) and expanded to integrate Container paradigms
  • B: Data Model - Create actual schema(s) and data model(s) for CNTT-specific information (potentially in an appendix)
  • A: ms Shared w/ RM-Ch4: Need to define virtual networking strategy and related attributes/parameters (workload level)

RM-Ch03 - Modelling

Pankaj Goyal Flavours/Flavors are not mentioned in RM Ch03. The term "virtual resource" is used and that includes resources that result from both hardware (server) and Operating System virtualisation. Typically, VNFs and CNFs are  mentioned together; there is an exception that should be corrected.

  • A: Integrate more Container paradigms/support
  • C: ms Flesh out Generic Fabric Model (Fabric, Underlay and Overlay level (i.e. Infra, not workload)
  • C: Research and enhance storage performance extensions using performance per unit of storage model (i.e. IOPs/GB)
  • A) Hardware Profiles and Performance.
    • Objectives/
    • Guidelines
    • What level of granularity.
    • Validation/Auditing.
    • Proposals or white paper on how we plan to achieve deterministic performance from the reference model w/o having to specify hardware to down to a very granular level (e.g. w/o SKUs, clock speeds, etc.)
  • A: Clarify diagrams and language around T-Shirts, Flavors and IT relationships
  • A: ms Policies for Non-Conforming Technologies // assumes supporting frameworks are ratified
    • SRIOV policy
    • SmartNIC policy*
    • Live Migration policy
    • *includes Tomas’ concept of driving SmartNIC industry to standard, cloud-friendly ABI
  • A: Shared w/ RM-Ch3: Need to define virtual networking strategy and related attributes/parameters (workload level).
    • 4.2.2: need to consider aggregate network bandwidth instead of per Interface bandwidth.
  • TBD: VNF Evolution / VNF Profile Generations
  • TBD: Simplify VNF Profile Naming

RM-Ch04 - Infrastructure Capabilities, Measurements and Catalogue

Pankaj Goyal suggest that the White Paper be not priority A.

Generic Fabric Model → Focus on terminology for Baldy.

  • A: Integrate cloud native concepts w/ sw profile (i.e. how abstraction and workload portability is maintained)
  • A: Update s/w profile, pending parking of Compute Intensive IT
    • Rationalize s/w profile to IT mapping after removal of third IT
    • Audit chapter alignment w/ RM-Ch4
  • A: Incorporate VNF Profile Generations/Evolution (once ratified), designing and describing linkage/relationship to h/w profiles

RM-Ch05 - Feature set and Requirements from Infrastructure
  • B: Flesh out initial Enabler Services

RM-Ch06 - External Interfaces (APIs and Interfaces)
  • A: ms Worthy of review by SMEs for suggestions of areas to incorporate and/or refine
19 Feb 2020

RM-Ch07 - Security

Added Security Requirements PR#1118

  • Cleanup to Ch08
    • A: Generic  qualification model (i.e. NFVI and VNF)
      • Badging types (requirements in RC)
      • Methodology to support N-3 VNF Rls Profiles
    • A: Overview of badging methodology
      • Base-lining testbed before VNF qualification
      • VNF testing without prior knowledge
      • Plumbing testbed for supplier testing of VNF functionality, after VNF badging.
    • A: ms Overview of novel VNF-related concepts
      • Reference VNF (aka Golden VNF)
      • NFVI characterization, and correction coefficients for normalizing performance
    • A: What is out of scope for CNTT (e.g., VNF functional testing, performance, scalability, HA, etc.)

RM-Ch08 - Compliance, Verification, and Certification

Should the RM content be VNF/CNF agnostic?

  • C: Create Generic Installer Model

RM-Ch09 - Infrastructure Operations and Lifecycle Management
  • C: Needs a strategy/purpose review and new commitment from original or new contributors to continue on it.

RM-Appendix-A - VNF Guidelines

Reference Architecture 1

Pankaj Goyal to add general to-do list in here.


Target Date

( // )


A: RA-1 cleanup (ch content complete 80%, issue closure)



A: Update OpenStack version based on newer version. (let continue discussion TSC call)

TBD (Baraque release)

Overall: Needs a TSC/Governance decision on criteria and then selection. Should not be Baldy release

Criteria to be discussed in vF2F April 21st. New version of RA-1 for selected OpenStack release will be in scope for Baraque release.

3A: Update RA-1 Ch01: Introduction content

Develop after most Chapters greater than 60% complete

Issue #1081

  • A: Remove 1.6 as this will be moved into Overall Roadmap.
  • A: Remove CI (as this has been decided to be parked for a bit and picked up later).
March 15, 2020Ch01 Issue #1081: PR#1169

A: Update RA-1 Ch02: Requirements

Align RM and RA-1 Requirements.

Added to Backlog as RM completion TBDCh02:  A document provided to RM WSLs with a request for clean-up RM reqts (2020-01-27)

A: Create Traceability entries in RA-1 Ch02. If Gaps and variances assign for completion/remediation7 Feb 2020Ch02: (2020-02-07) Completed 2.3.1 - 2.3.5 (Scope of Baldy release)
5A: Finalise RA-1 Ch03: NFVI + VIM Architecture

Topology: Cover DC/Edge and SLA driven.
Ch03: Issue #638 (content developed to be added to Github)

Review Networking Section and suggest Improvements

Ch03: Move to RM?

Create Traceability entries in RA-1 Ch02. If Gaps and variances assign for completion/remediation7 Feb 2020Ch03: Done

Remove 3.6 as Traceability is in Ch02.7 Feb 2020Done
6A: Finalise Ch04: NFVI + VIM Component Level Architecture

Add Hardware acceleration10 Feb 2020

Ch04: Cyborg Added

Create Traceability entries in RA-1 Ch02. If Gaps and variances assign for completion/remediation(7 Feb 2020)Ch04: PR #1046 Completed

Remove 4.7 as Traceability is in Ch02(14 Feb 2020)Ch04: completed as part of PR #1065
7A: Finalise RA-1 Ch05: APIs and Interfaces12 Feb 2020Ch05: completed

Qualify APIs that are actually utilised.12 Feb 2020Ch05: Already mentioned APIs and Microversion capabilities that are required

Add explanation on micro-versions10 Feb 2020Ch05: PR#1006 completed

Create Traceability entries in RA-1 Ch02. If Gaps and variances assign for completion/remediation(7 Feb 2020)Ch05: PR #1046 Completed

B: Create content for RA-1 Ch06: Security: Create security requirements list

Ch06: Created RM Ch07 Consolidated Security Requirements++: currently under review

Content being added – based on RM Ch07 requirements

Create Traceability entries in RA-1 Ch02. If Gaps and variances assign for completion/remediation

B: Create updated ToC and Content for RA-1 Ch07: Operations and Life Cycle Management

March 31, 2020Ch07:  Content CreatedTOC done

Develop content including Logging and MonitoringMarch 31, 2020Ch07: LMA content added; under review

Create Traceability entries in RA-1 Ch02. If Gaps and variances assign for completion/remediation

10B: Update RA-1 Ch08: Gaps, Innovation, and Development
Ch08: Done

Adopt Open Source (Tungsten Fabric) SDN APIs, map back to Requirements and push for adoption by OSTK as a Neutron extension
Ch08: now part of RM Networking FG

Discovery: review ONAP A&AI. Support for specific use cases: packet acceleration (DPDK, SR-IOV, …)

Capture Prague Etherpad Items

Reference Implementation 1

Qiao Fu to add general to-do list in here.


Target Date

( // )

1A: RI-1 cleanup (ch content complete 80%, issue closure)

2C: Resolve Continuous Software Deployment Errors
(MikeF add) - Not clear what that is. → MikeF:  Automatic deployment of OpenStack via the CICD Pipeline using AirShip fails.  Manual intervention is needed to resolve issues with erroneous port 80 references, incorrect NOVA versions, and end-point URLs which do not work.  (to name a few).  
3C: Document & Perform Repeatable CICD Deployment
(MikeF add) - some of it need to be documented in cookbook and some of it needs to be taken care of by OPNFV.
4A: Document how to Perform Repeatable Compliance Validations (post deployment)
(MikeF add) - some of it need to be documented in cookbook and some of it needs to be taken care of in OPNFV.

C: Close Gaps in Cookbook

[RC1 Ch9 - Snezka F2F] RC Cookbook Enhancements #945 

[RI1 Ch07] User Manual initial content #408


  • Common Failure codes, description, and resolutions
  • Test Framework Clean-Up Utility, Process, and Support Expectations

Process / Version Mgmt

  • Process for version control
  • Process for bug fixes and patches
  • Containers Tagged
  • Playbooks grab the tagged version

(MikeF add)
6C: Conduct Friendly Cookbook Trial

(MikeF add) - That is not for RI team to worry about.  → MikeF:  How/who will own this item as there needs to be a friendly trial of the cookbook with feedback/improvement?  This is about lab procurement, h/w and s/w validations, software installs, etc.  All pertaining to RI WS. 

Rabi: Michael Fix (AT&T) i will send you an invite to the weekly adoption meeting where we are getting those vendors for the trials. I think it should be handled in there and I think you should get involved as per your role in RI & OPNFV

7A: Select Vendor Candidates for RI installs
(MikeF add) - That is not for RI team to worry about.  → MikeF:  Can RI WS handle the handoff to the appropriate WS?  The ask is for parallel/multiple RI installs.  Who will identify & support these candidates?
8A: Identify Target Labs for RI installs
(MikeF add) - That is not for RI team to worry about.
9B: Outline Trial Partner Expectations & Establish Contact
(MikeF add) → → MikeF:  Can RI WS handle the handoff to the appropriate WS?  The ask is for parallel/multiple RI installs.  Who will identify & support these candidates?

B:  Descriptor File - finalize & perform PoC

Refer to: 

  • [RI 1 Core]: Work with OPNFV Infra WG for the evolvment of PDF to fit into need for CNTT RI #526
  • [RI 1 Labs] Tooling to Generate a PDF matching Airship Internal Data Structure #525

(MikeF add) - That is OPNFV issue not CNTT. need to discuss who should own this. off course CNTT team needs to be part of the discussion, but someone from OPNFV has to take ownership of this and lead that discussion.   → MikeF: Agree.  RI WS is managing this today.  Need handoff from RI WS to OPNFV.  Should keep this open through the handoff.
  • A: 1.3 scope: clarify scope as per Prague discussion.
  • A: 1.4 remove Roadmap (this will be /should be covered in the overall CNTT Roadmap).

  • A: 2.2 Remove RA Req as it is not necessary.
  • A: 2.3 put req into a table with reference numbers.
  • A: 2.4 Diagram is not relevant: This should be removed or moved to Chapter 4 (Lab requirement) and improve diagram as it is low quality. Do we need an example section?

  • A: 3.1 clarify the intention of the chapter (to be able to create a PDF/IDF from the content of the chapter).
  • A: It will be better to simplify the presentation of the content here (metadata driven approach will be recommended)
    • To make it simple to create PDF/IDF from the chapter.
  • A: 3.4 it mixes requirement, with architecture with state. (need to clean up and remove any duplications). Remove any reference to functest or certification or test cases, this should be all about the state of NFVI.

  • C: Need a Topology Diagram.
  • B: Need Networking/Switching Requirements.

  • B: 5.2 Add more installer general requirement (e.g. the need for it to be opensource, the result of it needs to be opensource).
  • A: 5.3: needs to agree if this is something we need to have in CNTT. (ongoing discussion in OPNFV about it)


A: Create a Cookbook for Labs (how to access labs, types of labs available, etc)


A:  lot of clean-ups needed.

  • Not mix labs with installation (Chapter 6 deals with labs)
  • 7.2, 7.3, 7.4 will need to move to Chapter 6 since they are lab related.
  • 7.6, 7.7 (clarify difference between deployment validation and development validation).
  • This should be sanity check and not extensive testing.


C: Need some initial content (are there no gaps? )



18B: Descriptor File - finalize & perform PoC
Work with OPNFV to agree on approach for handling Descriptor Fromat for various installers.
19A: Ensure RI-1 lab is installed / available for tests.

Confirm that the CI/CD Scripting works (Natural bi-product of lab install success) 

20A: Endure that funkiest is able to repeatedly validate the installation of RI.

21A: RI-1 passes the RC-1 sanity check
  • RI supports RC
  • Define pass criteria for sanity check

Reference Conformance 1

Michael Fix (AT&T) to add general to-do list in here.


Target Date

( // )


A: Type of conformance:

  • Installation Validation: Sanity Test (APIs),
  • Conformance against CNTT Specs:
    • Feature-set and Configuration (Functional)
    • Performance, etc

  • Define pass, compliance, verification, validation
  • Updates to: Ch01: Introduction

A:  Define Conformance

A:  Replace Certification w/ Conformance

B: 1.7 Results Collation & Presentation - write section on where/how results will be normalized, collated, and presented

C: 1.8 Governance - expand to include LCM, define partnerships and expectations from these partners as to what info or support is exchanged, or provided

C:  [RC1 Ch01] Provide Verification Process, including Life Cycle Management #159

  • Updates to: Ch01: Introduction

A: (General) Finalize Test tooling/Framework
  • Updates to: (NFVI) Ch02: NFVI Testing Framework Requirements

A:  Replace Certification w/ Conformance

A:  Finalize Test Hardness/Framework

B. 2.6 Entry & Exit Criteria - review for completeness, and reach consensus from the community that criteria satisfies objectives for CNTT (as intake to testing, and delivery to telcos)

B. Test Results - community review and consensus needed on collation and portal requirements; need to identify and document portal, or dashboard, used for results presentation

B. 2.7.4 Badging - need alignment with OVP / CVC on badging steps and expectations; need alignment to add Xtesting as an alternative to Dovetail 

  • Updates to: (NFVI) Ch02: NFVI Testing Framework Requirements

A: (General) Finalize TC Req 

  • Performance is not required for April
  • Updates to: (NFVI) Ch03: NFVI Test Cases Requirements

A:  Replace Certification w/ Conformance

C: 3.5 Software & Hardware Reference - content needs to be written to identify requirements for testing Software Configuration/Profiles, and Hardware Configuration/Profiles.  

C: 3.6 Options & Extensions - content needs to be written to create requirements for testing/evaluating Options and Extensions available and configured.  

C: 3.8.2 Resiliency Measurements - need to be written

C: 3.9 Test Cases - remove (move) to Ch 4

  • Updates to: (NFVI) Ch03: NFVI Test Cases Requirements

A: Cleanup & Finalise NFVI Testing Cookbook

(General) Test suite is created

  • Updates to: (NFVI) Ch04: NFVI Testing Cookbook
  • Identify and writeup missing TCs
  • Performance is not required for Baldy
  • Supporting OvS-DPDK in RI-1
  • Test suite details

A: Mapping Test Cases to CNTT Req

  • Updates to: (NFVI) Ch05:  NFVI Test Cases and Traceability to CNTT Requirements

A:  [RC1 Ch04] Define the test cases for exposed infrastructure capabilities #774 - This issue is to capture the tests the exposed infrastructure capabilities as defined in RM §4.1.2.

A:  Finalize TC Req Mapping & Close TC Gaps

A:  Port needed/missing Test Cases (also with RC1 Dev)

  • port YardStick testcases to Xtesting
  • port Bottlenecks to Xtesting
  • port StorPerf testcases to Xtesting
    port NFVbench testcases to Xtesting
  • update and integrate heat-tempest-plugin in Functest Heat API testing
  • integrate KloudBuster in Functest disk benchmarking
  • add tempest-stress in Functest stress testing
  • port VTP test cases to Xtesting

B: 5.3 Traceability Matrix - write introductory explaining this section defines the mapping, or traceability of RM/RA-1 requirements to test cases

C: Review the applicability for, &/or Create content for Test Case Traceability of the following:

5.3.8 Tenants
5.3.9 LCM
5.3.10 Assurance
5.3.11 Security
5.3.13 Resilience
5.3.14 Bare-metal validations

  • Updates to: (NFVI) Ch05:  NFVI Test Cases and Traceability to CNTT Requirements

C: (General) RC-1 cleanup (ch content complete 80%, issue closure)

C: (General) Create Tools & Perform Hardware & Manifest Verifications
Part of long-term RC program

C: (General) Perform Empirical Validations (against prototype VNFs)
Part of long-term RC program

B: (General) Collect & Normalize  Results
Needed for RC program to simplify reviews & badging

A:  Replace Certification w/ Conformance

C: 6.2.2. Prototype VNFs - Identify reference VNFs per Family Types to be used for Empirical Data Collection and evaluation against 'real' VNFs

B.3 Badging Requirements - 6.3.1 Badging Scope - reach consensus with community, and OVP/CVC on badging framework defined

C: Expand (elaborate) the following which has no/limited content today:

  • 6.4.11 User & System Interfaces - lacks context, lists only UI and Programming Interfacce

  • 6.4.12 Deliverables - needs content to describe by Docker and Standalone Installation Scripts are needed and pertinent for VNF Frameworks

(VNF) Ch06: VNF Testing Framework Requirements

A:  Replace Certification w/ Conformance

C: 7.5 Interaction Type - Describe the types of Interactions: Extended Topology, Complex (Akraino), Functional, HA, Fault, Interoperability

(VNF) Ch07: VNF Test Cases Requirements

B: Chapter needs community review and inputs - integrated NFVI framework (Section 8.3) is missing context, or reference to prior NFVI E2E Framework chapters.  Clean up needed.

e.g. Content examples to add 

  • Identify Framework Needs, Goals, and Dependencies
  • Define Opensource Integration (OPNFV, OVP, Functest, CVC, others)
  • Provide Automation Toolchain (list, topology, flow)

Missing content needs to be reviewed/vetted for inclusion, and if needed, write content:

8.2.2. Yardstick - purpose, adoption &/or use of project, define why important

8.2.3 Bottlenecks  - purpose, adoption &/or use of project, define importance

(Dev) Ch08: VNF Testing Cookbook

C: All Chapter Content needs to be written:

  • Introduction - Provide an overview of the purpose for the VNF TC Traceability to RM Requirements chapter
  • RM/RA1 Requirements - define requirements
  • Test Case Traceability - tracing test cases to requirements

(VNF) Ch09: VNF Test Cases and Traceability to CNTT Requirements

A: 9.3 TC Mapping to Requirements needs explicit reference to RM/RA-1 requirements mapping.  Very general mappings at present, with not explicit requirement traceability, or reference to NFCN traceability in prior chapter.

(Dev) Ch09: NFVI Tests Traceability to TC Requirements

B: Content needs to be expanded:

10.2 OpenStack Release Comparisons - need to upload/add existing content regarding detail / comparison of OpenStack releases based on Pike baseline for CNTT RI-1 (e.g. Ocata, Pike, Queens, Stein, etc) 

10.5 Framework Gaps - VTP is referenced as framework gap, but need to also include:

  • Hardware (baremetal) Validations
  • Software (manifest) Validations
  • VVP heat artifact/template validations

(Dev) Ch10: Gap analysis & Development:


A: (General) RI-1 passes the RC-1 test suite execution (For sanity and APIs)

Reference Architecture 2

Tom Kivlin to add general to-do list in here.


Target Date

( // )


A: Chapter completeness:

  • Chapter 1 bogometer at 100% (stable scope/principles, etc.)
  • Chapter 2 bogometer at 100% (stable requirements)
  • Chapter 3 bogometer at 80% 
  • Chapter 4 bogometer at 80%
  • Chapter 5 bogometer at 60%
  • Chapter 6 bogometer at 60%
  • Chapter 7 bogometer at 60%
  • Chapter 8 bogometer at 80%

Align on RI2 on which chapters/sections are needed as a priority.

Chapter 1

  • A: Generic clean up, change bogometer, remove roadmap section to Technical document
  • A: Clarify principles around cluster scope (per VNF, etc.), what RA2 specifies and what software vendors can specify


Chapter 2

  • A: Complete section 2.3
  • A: Close all backlog issues re. ch2


Chapter 3

  • A: TSC decided to remove Compute Intensive - remove from chapter
  • A: Clearer link between Kubernetes / containerisation and sw/hw profiles
  • B: Content addition to empty sections


Chapter 4

  • B: Way more detail required
  • B: Detailing the constraints that make component decisions (e.g. kernel modules/versions/etc.)
  • B: More detail on Kubernetes feature flags
  • B: 4.7 - check on requirements driving ASM/NSM


Chapter 5

  • B: Add content


Chapter 6

  • B: Restructure and content review - focus on testable specification rather than guidance


Chapter 7

  • B: Add content


Chapter 8

  • B: Content needed in 8.3 - this defines the scope/requirements for any development effort


Appendix A

  • B: Consider title change to reflect the focus being on how to move from VNF-only to VNF and CNF support from infrastructure
  • C: Consider moving to a separate document, or even the CNCF TUG


Picks for Baldy

Picks for Baldy will go in Pull Request here: