Minutes

Minutes of June 23 7am EDT session "Joint Topic: Performance"

Definitions of Performance

• CNTT_vF2F_June_Performance-edits TCAMv02.pptx
• Work plan - use the next CVC calls (July 6) to begin documenting the agreed terms for performance
  • Then map how these terms apply within the context of the CNTT RC definitions.
• Performance Measurement - 

(SUT = System Under Test)

Calibration:

The process of checking and/or adjusting a stimulus generation or measurement device with a known reference value, to improve the overall quality of the measured results.  Calibration may be very simple, such as a comparison of the configured traffic generator sending rate and measured rate using a simple SUT (loop-back cable between interfaces), such that that the known reference value is the published nominal interface rate.

Known Reference Value:

A measured or established result or outcome for comparison with new measurements. For example, the reference value or expected outcome of a Functional Test is "PASS". The reference value or expected outcome of a Performance Measurement or Benchmarking test may be the value measured for the previous SUT release, or the published value or theoretical limit of a simple SUT.

Functional Testing:

The main object of functional testing is the verification of a specific stimulus / response behavior within the SUT, including causation.  These tests generally result in a binary outcome, i.e. pass / fail.  For example, verification of an "API call" and its associated response, such as the instantiation of a VM (or container) and verification of the VM's existence (causation), or the ability to activate a specific feature of the SUT (i.e. SR-IOV).

Performance Measurement: 

The procedure or set of operations having the objective of determining a Measured Value or Measurement Result of an infrastructure in operation according to a defined metric.

In the context of telemetry telemetry, Performance Measurements reflect data generated and collected within the cloud infrastructure, that reflects a performance aspect of the cloud infrastructure. For example, a count of frames or packets traversing an interface per unit of time, memory usage information, other resource usage and availability, etc. This data may be instantaneous or accumulated, and made available (i.e. exposed) based on permissions and contexts (e.g., workload vs. infra).

Other Performance Measurements are designed to assess the efficiency of SUT Functions, such as the time to successfully instantiate one or more VMs or containers, or the percentage of instantiation Function failures in a set of many attempts.

Still other Performance Measurements are conducted under controlled conditions using Calibrated test systems, such that the measured results are more likely to comparable with other such measurements.

Performance Testing:

The main objective of performance testing is to understand if the System Under Test is able to achieve the expected performance, through conducting a series of Performance Measurements, comparing those results against a specific (Benchmark / Threshold) Reference value.

It requires a set of performance testing tools (open source) that help with the dimensioning of a solution by an operator.

Testing results may be useful to compare infrastructure capabilites between the System Under Test (SUT) and a CNTT reference implementation of RA-2. Performance testing for the purpose of comparing between different commercial implementations is not a goal of CNTT. Performance testing relies on well established benchmark specifications.

Benchmarking:

Benchmarking assessments do not define acceptance criteria or numerical performance requirement. Benchmark testing and Conformance testing intersect when a specific requirement in the software specification is very important, such as a frequently-executed function. Correct execution with the expected result constitutes conformance. The completion time for a single conforming execution, or the number of conforming executions per second are potential Benchmarks. Benchmarks tests assess a key aspect of the computing environment in its role as the infrastructure for (cloud-native) network functions, using Calibrated test systems and controlled conditions. The benchmarks (and related metrics) have been agreed by the Industry and documented in publications of an accredited standards body. As a result, benchmarks are a sub-set of all possible performance metrics; they are selected measurements which are more important than others. Examples benchmarks include data rate, latency, and loss ratio of various components of the environment, expressed in quantitative units to allow direct comparison between different systems treated as a black box (vendor-independence). Because the demands on a particular system may vary from deployment to deployment, Benchmarking assessments do not define acceptance criteria or numerical performance requirements.

Calibration:

The process of adjusting a measurement device, or its outputs, to improve the overall quality of the data.  

Functional Testing:

Benchmarking assessment methods do not define acceptance criteria or numerical performance requirement.

Benchmark testing and Conformance testing intersect when a specific requirement in the software specification is very important, such as a frequently-executed function. Correct execution with the expected result constitutes conformance. The completion time for a single conforming execution, or the number of conforming executions per second are potential Benchmark metrics, and sources of known reference valuesThe main objected of functional testing is the verification of a specific stimulus / response behavior within the SUT, including causation.  These tests generally result in a binary outcome, i.e. pass / fall.  For example, verification of an "API call" and its associated response, the instantiation of a VM and verification of its existence, or the availability of a specific feature of the SUT (i.e. SR-IOV).

• Threshold - 
• Functional Tests
• Measurement Performance of functions
• Exception Threshold 
• Network
• Computation
  • CPU
  • GPU
• Life cycle
  • Time to create VM / container
  • Time to delete VM / container
  • Maximum number of VMs / containers