Overview

The purpose of this page at this time is to capture requirements related to observability of the EMCO services (https://gitlab.com/groups/project-emco/-/epics/7).

Front-ending the services with Istio provides a useful set of metrics and tracing, and adding the Prometheus library provided collectors to each service expands that with other fundamental metrics. The open question is what additional metrics and tracing will be useful to EMCO operators.

Metrics

The following items are based on Prometheus recommendations for instrumentation.

Queries, errors, and latency

Both client and server side are provided by Istio. https://istio.io/latest/docs/reference/config/metrics/

Istio metrics can be customized to include other attributes from Envoy such as subject field of peer certificate. https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/advanced/attributes

Example PromQL

ServiceTypePromQLNotes

HTTP/gRPC*

*The request_protocol label can be used to distinguish among HTTP and gRPC.

Queriessum(irate(istio_requests_total{reporter="destination",destination_workload=~"services-orchestrator"}[5m]))inbound
sum(irate(istio_requests_total{reporter="source",source_workload="services-orchestrator"}[5m])) by (destination_workload)outbound
Errorssum(irate(istio_requests_total{reporter="destination",destination_workload=~"services-orchestrator",response_code!~"5.*"}[5m])) / sum(irate(istio_requests_total{reporter="destination",destination_workload=~"services-orchestrator"}[5m]))inbound
sum(irate(istio_requests_total{reporter="source",source_workload=~"services-orchestrator",response_code!~"5.*"}[5m])) by (destination_workload) / sum(irate(istio_requests_total{reporter="source",source_workload=~"services-orchestrator"}[5m])) by (destination_workload)outbound
Latencyhistogram_quantile(0.90, sum(irate(istio_request_duration_milliseconds_bucket{reporter="destination",destination_workload="services-orchestrator"}[1m])) by (le)) / 1000P90
Saturation

Queries, errors, and latencies of resources external to process (network, disk, IPC, etc.)

The prometheus golang library provides builtin collectors for various process and golang metrics: https://pkg.go.dev/github.com/prometheus/client_golang@v1.12.2/prometheus/collectors. A list of metrics provided by cAdvisor is at https://github.com/google/cadvisor/blob/master/docs/storage/prometheus.md. Additional K8s specific metrics can be enabled with the https://github.com/kubernetes/kube-state-metrics project.

Example PromQL

Note: some of these require that kube-state-metrics is also deployed.

Pod ResourceTypePromQL
CPUUtilizationsum(rate(container_cpu_usage_seconds_total{namespace="emco"}[5m])) by (pod)
Saturationsum(rate(container_cpu_cfs_throttled_seconds_total{namespace="emco"}[5m])) by (pod)
Errors
MemoryUtilizationsum(container_memory_working_set_bytes{namespace="emco"}) by (pod)
Saturationsum(container_memory_working_set_bytes{namespace="emco"}) by (pod) / sum(kube_pod_container_resource_limits{namespace="emco",resource="memory",unit="byte"}) by (pod)
Errors
DiskUtilizationsum(irate(container_fs_reads_bytes_total{namespace="emco"}[5m])) by (pod, device)
sum(irate(container_fs_writes_bytes_total{namespace="emco"}[5m])) by (pod)
Saturation
Errors
NetworkUtilizationsum(rate(container_network_receive_bytes_total{namespace="emco"}[1m])) by (pod)
sum(rate(container_network_transmit_bytes_total{namespace="emco"}[1m])) by (pod)
Saturation
Errorssum(container_network_receive_errors_total{namespace="emco"}) by (pod)
sum(container_network_transmit_errors_total{namespace="emco"}) by (pod)

Internal errors and latency

Internal errors should be counted.  It also desirable to measure success to calculate ratio.

Totals of info/error/warning logs

Unsure if this is a useful metric.

Any general statistics

This bucket includes EMCO specific information such as number of projects, errors and latency of deployment intent group instantiation, etc. Also consider any cache or threadpool metrics.

Preliminary guidelines:

  • Distinguish between resources and actions. 
  • Action metrics will record requests, errors, and latency similar to general network requests.
  • Resource metrics will record creation, deletion, and possible modification.  
  • Metrics will be labeled with project, composite-app, deployment intent group, etc.

For rsync specifically, measure health/reachability of target clusters.

Also, keep in mind this cautionary note from the Prometheus project:

CAUTION: Remember that every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values.

However note that well-known projects such as Istio and kube-state-metrics appear to disregard this, so further investigation may be needed on the motivations behind this note.

Preliminary metrics

This section contains some of the considerations of the guidelines above applied to the orchestrator service.

The actions of a service can be identified from the gRPC requests and HTTP lifecycle requests:

ServiceAction
orchestrator

approve

instantiate
migrate
rollback
stop
terminate
update
StatusRegister
StatusDeregister

The requests, errors, and latency can be modeled after Istio's istio_requests_total and istio_request_duration_milliseconds, with an additional action name label.

The resources of a service can be identified from the HTTP resources.  The initial labels can be the URL parameters.

ServiceResourceLabels
orchestrator









controllername


projectname
compositeAppversion, name, project
appname, composite_app_version, composite_app, project
dependencyname, app, composite_app_version, composite_app, project
compositeProfilename, composite_app_version, composite_app, project
appProfilename, composite_profile, composite_app_version, composite_app, project
deploymentIntentGroupname, composite_app_version, composite_app, project
genericPlacementIntentname, deployment_intent_group, composite_app_version, composite_app, project
genericAppPlacementIntentname, generic_placement_intent, deployment_intent_group, composite_app_version, composite_app, project
groupIntentname, deployment_intent_group, composite_app_version, composite_app_name, project
dcm

emco_logical_cloud_resource

project, name, namespace, status
clm


emco_cluster_provider_resource

name

emco_cluster_resource

name, clusterprovider
ncm


emco_cluster_network_resource

clusterprovider, cluster, name, cnitype

emco_cluster_provider_network_resource

clusterprovider, cluster, name, cnitype, nettype, vlanid, providerinterfacename, logicalinterfacename, vlannodeselector
dtc

emco_dig_traffic_group_intent_resource

name, project, composite_app, composite_app_version, dig




emco_dig_inbound_intent_resource

name, project, composite_app, composite_app_version, dig, traffic_group_intent,

spec_app, app_label, serviceName, externalName, port, protocol, externalSupport, serviceMesh, sidecarProxy, tlsType

emco_dig_inbound_intent_client_resource

name project, composite_app, composite_app_version, dig, traffic_group_intent, inbound_intent, spec_app, app_label, serviceName

emco_dig_inbound_intent_client_access_point_resource

name, project, composite_app, composite_app_version, dig, traffic_group_intent, inbound_intent, client_name, action

ovnaction

emco_network_controller_intent_resource

name, project, composite_app, composite_app_version, dig

emco_workload_intent_resource

name, project, composite_app, composite_app_version, dig, network_controller_intent, app_label, workload_resource, type

emco_workload_interface_intent_resource

name, project, composite_app, composite_app_version, dig, network_controller_intent, workload_intent

interface, network_name, default_gateway, ip_address, mac_address

The metrics for these resources should capture the state of the resource, i.e. metrics for creation, deletion, etc. (emco_controller_creation_timestamp, emco_controller_deletion_timestamp, etc.) as described in the guidelines. This approach is suggested as it is unclear how to apply metrics capturing resource utilization to these resources.

The status of a deployment intent group deserves special consideration. The suggested approach is to support the labels necessary to execute equivalent queries as shown in EMCO Status Queries. This would enable alerting on the various states of the resources composing a deployment intent group.

MetricTypeDescriptionLabels
emco_deployment_intent_group_resourceGAUGE0 or 1project
app
composite_app_version
composite_profile
name
deployed_status
ready_status
app
cluster_provider
cluster
connectivity
resource_gvk
resource
resource_deployed_status
resource_ready_status

The deployment intent group shown in Example query - status=deployed would create the following metrics:

emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group",deployed_status="instantiated",ready_status="ready",app="firewall",cluster_provider="vfw-cluster-provider",cluster="edge01",connectivity="available",resource_gvk="ConfigMap.v1",resource="firewall-scripts-configmap",resource_deployed_status="applied",resource_ready_status="ready"}
emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group",deployed_status="instantiated",ready_status="ready",app="firewall",cluster_provider="vfw-cluster-provider",cluster="edge01",connectivity="available",resource_gvk="Deployment.v1.apps",resource="fw0-firewall",resource_deployed_status="applied",resource_ready_status="ready"}
emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group",deployed_status="instantiated",ready_status="ready",app="firewall",cluster_provider="vfw-cluster-provider",cluster="edge02",connectivity="available",resource_gvk="Config.v1",resource="firewall-scripts-configmap",resource_deployed_status="applied",resource_ready_status="ready"}
emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group",deployed_status="instantiated",ready_status="ready",app="firewall",cluster_provider="vfw-cluster-provider",cluster="edge02",connectivity="available",resource_gvk="Deployment.v1.apps",resource="fw0-firewall",resource_deployed_status="applied",resource_ready_status="ready"}
emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group",deployed_status="instantiated",ready_status="ready",app="packetgen",cluster_provider="vfw-cluster-provider",cluster="edge01",connectivity="available",resource_gvk="Deployment.v1.apps",resource="fw0-packetgen",resource_deployed_status="applied",resource_ready_status="ready"}
emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group",deployed_status="instantiated",ready_status="ready",app="packetgen",cluster_provider="vfw-cluster-provider",cluster="edge01",connectivity="available",resource_gvk="ConfigMap.v1.apps",resource="packetgen-scripts-configmap",resource_deployed_status="applied",resource_ready_status="ready"}
emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group",deployed_status="instantiated",ready_status="ready",app="packetgen",cluster_provider="vfw-cluster-provider",cluster="edge01",connectivity="available",resource_gvk="Service.v1.apps",resource="packetgen-service",resource_deployed_status="applied",resource_ready_status="ready"}
...

Some example queries:

DescriptionPromQL
deployedCountscount(emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group",resource_deployed_status="applied"})
readyCountscount(emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group",resource_ready_status="ready"})
count(emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group",resource_ready_status="notready"})
appscount(emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group"}) by (app)
clusters filtered by the sink and firewall appscount(emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group",app="sink"} or emco_deployment_intent_group_resource{project="testvfw",composite_app="compositevfw",composite_app_version="v1",composite_profile="vfw_composite-profile",name="vfw_deployment_intent_group",app="firewall"}) by (cluster_provider,cluster)

Tracing

Istio provides a starting point for tracing by creating a trace for each request in the sidecars.  But this is insufficient as it does not include the outgoing requests made during an inbound request.  What we'd like to see is a complete trace of, for example, an instantiate request to the orchestrator that includes the requests made to any controllers, etc.

In order to do this it is necessary to pass the tracing headers from the inbound request through to any outbound requests.  This will be done with the https://opentelemetry.io/ golang libraries.

Logging

Each log message must contain the timestamp and identifying information describing the resource, such as project, composite application, etc. in case of orchestration.

The priority is placed on error logs; logging other significant actions is secondary.

  • No labels

7 Comments

  1. Srinivasa Addepalli

    One general comment:

    • Every API Call (External API call or internal API calls among modules) shall have correlator for Kiali and other observability tools to figure out the related calls.  
    • Every API call shall have user ID in case of external API call.  In case of internal API call, it shall have internal certificate subject name of the caller.

    I am hoping that both of above are supported by ISTIO.  Since we have ISTIO running as sidecar proxies as well as ingress proxies, we would have comprehensive tracing.  I think that there is something that needs to be taken care by EMCO to pass the correlator across the API calls.  That enhancement is required in my view. That is, any EMCO module that generates API call to some other module may need to copy something from the incoming API call.  

    With respect to EMCO specific metrics:

    • Different Error counters;  Any error shall have corresponding error counter.  If there are X number of places, error being returned, then one shall have X number of counters.  Since different modules can generate errors, it is always good to have error base for each module and have error numbers from this base.
    • Non-Error metrics :  For every significant 
    • API Metrics:  Not sure whether ISTIO already supports it. If not, for every API type, one shall have one counter.
    • API Errors:  covered as part of "different error counters'.
    • For every resource, it is good to maintain three counters - Added records, deleted records and modified records.
    • For every action : Good to have a counter for every resource action (Instantiate, Apply etc.. for different resources)
    • Counters for every resource & action (as described above two bullets) shall be maintained on per project basis for resources that are part of the project.
    • Counters for every resource & action can be maintained on per composite application basis.
    • Counters for every & action can be maintained on per deployment intent group basis.
    • In case of RSYNC, it is good to have counters (Cumulative, Project, Composite App,  Deployment intent group basis)
      • Number of apps
      • Number of clusters
      • Successfully synchronized
      • Unsuccessful synchronization

    With respect to logs:

    • Every log shall have 'partner', 'customer', 'project', 'composite application', 'DIG name" in case of orchestration. Similar identification information from CLM, NSM, DCM etc..
    • Every log shall have time & date.
    • All error events must be logged.
    • Some significant actions can be logged.


    Some quick thoughts (smile).

    1. Todd Malsbary

      Thanks Srini, I incorporated your comments into the page.  One question: on API Metrics what is meant by "for every API type"?

  2. Nadathur Sundar

    Good to see that you are actively engaged, Srini! Thanks for the feedback.

    Since we already have another thread on tracing, I'll keep it brief here. It is up to each application (EMCO) to implement tracing within its components; ingress gateways like Istio can only help – by generating a trace id, root span id, passing some additional HTTP headers, etc. This is true of other ingress gateways like nginx too.

    Re. metrics, Srini has given a good account from a bottom-up perspective. I'd like to complement that with a top-down view. 

    A system has different layers, and the metrics to be monitored for each layer are different. At the most basic level, for a K8s system, we are interested in pods and services. For pods, the main focus would be resource consumption:  CPU/memory etc. The main set of metrics here are the USE set -- Utilization, Saturation and Errors -- for each container/pod. The cAdvisor metrics give us most of this info. For example, container_cpu_usage_seconds_total is a measure of CPU utilization for that container, whereas container_cpu_load_average_10s is a measure of CPU saturation, i.e., how many processes/tasks were queued up waiting to be run. All these can be gathered through Prometheus. So, we are good.

    One catch is that the cAdvisor metrics don't give us relative utilization, like container A is using 80% of a NIC's bandwidth or B is close to filling its file system. So, Prometheus has to be configured with alerts based on the known resource capacities.

    For services, the key metrics are summarized by Google's Golden Signals.

    • Latency: time it takes to service a request. We usually want the average and the maximum time. We also want to measure this for different API types (URLs) for each microservice. For example, for the orchestrator, we'd want the time for DIG instantiation, queries, etc.  For the clm, we'd want cluster queries/updates, etc. We need to check if Istio can provide both average and maximum for these. (Some people recommend using a histogram of latencies to identify 99th percentile etc. Istio's Request Duration may be available as a histogram. But we can probably keep that for the future if needed.)
    • Traffic: A measure of the load on the system. In our case, it would be HTTP requests/second. Here too, we'd want it per-uService and per-URL.
    • Errors: These are mostly requests that timed out or got a HTTP 400s/500s response. Need to check if Istio can provide timeouts and error responses separately.
    • Saturation: For a service, this is a measure of the "queue length", how many requests are waiting to be serviced. Not sure how we can get this.

    But the list of documented Istio metrics is rather small. It doesn't look like many of the above are supported today, but maybe we need to investigate more. There are ways to customize Istio metrics, but I haven't played with them yet. We can start with what we have and document the gaps.

    For services, some folks have used the RED metrics (Rate, Errors and Duration), but that seems to be a subset of the Golden Signals. The saturation metrics are not included - but we may not be able to implement them anyway in the first release. So, we may effectively be implementing just the RED metrics to start with.

    1. Todd Malsbary

      The prometheus client lib for golang provides some default collectors that give us cpu usage, etc. as you describe.  I can't find it nicely documented anywhere so I'll update this page with the list shortly.

      The builtin Istio metrics give us a good start for latency, traffic, and errors:

      • istio_request_duration_milliseconds can be used to measure latency
      • istio_requests_total can be used to measure requests/second
      • istio_requests_total also includes the response code and can be used to count 4xxs/5xxs, etc.

      Saturation is a good question, and also something I don't have an answer for yet.

  3. Grzegorz Panek

    Hi,

    few cents from my side.

    I can see that currently the objective of a "Observability" in EMCO is to utilize as much Prometheus to gather more info about EMCO resources itself.

    Just wanted to know what you think about making EMCO aware about real-time state of clusters that are managed by EMCO?

    I can see opportunity to integrate Cortex (https://cortexmetrics.io/) as an aggregator of metrics from different Prometheus instances ( responsible for single cluster) in order to have a global view on what's happening on clusters.

    It could allow for EMCO to have global view on e.g., cpu/ram/memory utilization at each cluster and finally to build smart-placement algorithms. That's an opportunity that no other tools provide for today (neither onap/nephio)


    1. Nadathur Sundar

      Good idea, Greg. We could have a multi-cluster aggregation of metrics which EMCO can query (or be notified by), and that should feed into smart placement. 

      Cortex is a good choice for multi-cluster aggregation. There is another CNCF sandbox project named Thanos, which also aims at high-availability and long-term storage for metrics. It is not immediately obvious how they compare: in fact, the two projects apparently collaborate to leverage from each other. Thanos has a couple of K8s operators in open source: Banzai and OCM (latter seems specific to OpenShift). Thanos uses the hub-cluster and managed-clusters model of OCM (and Nephio), where EMCO could be running in the hub cluster. That said, I am open to both.

  4. Srinivasa Addepalli

    Todd Malsbary,


    with respect to logs, I also suggest to have 'user id', 'organization of user' also part of every log message, if the message is related user actions.