Barometer uses leverages collectd, an opensource project to collect the system metrics ,and works closely with the collectd community. Collectd has various plugins that receive upstream contributions from the OPNFV community, more specifically the Barometer project. Increasing the automated test coverage benefits collectd as well as team as well. The aim of this project is to Increase and automate the test coverage for collectd to ensure it's integrity for use in Barometer .

Collectd currently has some unit tests in place but no functional tests as of now. The aim of the project is to develop functional test cases for a bunch of collectd plugins that receive attention from the barometer team. Finally, we are looking to We will also set up a Jenkins build pipeline which reports to post the status results of these tests back to to  the Collectd Github repository . This way, we contribute to the testing of collectd plugins that, as well as to ensure that the project's integrity is maintained.

Goals

(for every open PR).

Goals

1. Figure out Decide on how to run tests locally. Currently, functest is being used. We might need to investigate the feasibility and if required, set up new testing tool/architecture.
2. The existing tests defined for functest/xtesting are old and cover only a subset of plugins that existed 2 years ago. 
1. So tests will be needed for most plugins. There are a bunch of plugins that barometer uses:
   1. Log-parser -> Easy plugin to understand and get started
   2. Network Capabilities Plugin -> For influx  DB
      
   3. SNMP Write Plugin
   4. Memory RAS
   Plugins from the Collectd PR's(Plus sign indicate a high frequency of PR's):
   1. ethstat plugin
   2. SNMP +
   3. Processes plugin +
      
   4. SQL plugin +
   5. Exec Plugin
   6. Threshold plugin +
   7. Memory plugin +
   8. Bind Plugin
   9. Network Plugin + 
3. A review will be needed for the tests in baro_tests/ and determine whether these are suitable for the collectd community
4. Set up triggering a Jenkins build when a PR is submitted to the collectd repository, run the job on node 5 of intel pod13 on a nightly basis ( can be triggered upon a github PR at a later stage)
   
5. Contribute to making the barometer docs better, wherever there is confusion and things aren't mentioned clearly to make things easy for future contributors.

Tasks

Implementation Details

We begin with a review of the tests in baro_tests/  ensure that they work well and determine whether these are suitable for the collectd community and for Barometer. The testing tool is not yet finalised finalized and will be done based on further discussion with the Barometer community, keeping all the use cases in mind.

Fig.1.1 High-level architecture of the Jenkins CI setup

We will then proceed with by writing test cases for a few simple plug-ins (like the log-parser and Network-Plugin among other plugins, details for which can be found in the section above). The test case will be hosted in the barometer repository for the time being (and the framework that we will use will most probably be x-test or functest). Subsequently, we move towards creating , the deliverables section also mentions a bunch of plugins with some manual tests which will be automated as part of this project as well. To be inclusive of all plugins under development, there are a few plugins that we have decided to try and add testing for as they are under active development.

The next step will be to create a CI pipeline using the Jenkins GitHub plugin, which can should trigger builds/running of the tests when a PR is submitted . to the collectd repository (Fig 1.1 shows the possible high-level design of the testing architecture. The possible steps ).

The tentative steps to make this happen are:

1. Setup and install the Github GitHub PR builder plugin in Jenkins (ticket raised).
2. Get the Intel-POD13 connected to Jenkins as a slave.
3. Configuring the Jenkins slave in POD13 POD13 .
4. Support will be needed from the collectd Collectd community to configure the Github GitHub repository and web-hooks

Once the architecture and pipelines are set up, testing the overall functionality will be our focus. We will move ahead with adding tests for more plugins. Deliverables mention a bunch of plugins which are enabled in Barometer and contains a list of manual tests. We wish to automate existing tests for these plugins. Furthermore, Some more plugins have also been selected based on the active PR's in the collectd Github repository. Since these are the plugins under active development, having functional test cases for them makes more sense.

However, the list of plugins is tentative and subject to change upon further discussions with Barometer and collectd community.

Note: The Jenkins Job set up for running the tests cannot be triggered upon PR submission to Collectd github repository as the tool decided to be used was not quite the right fit. It can be taken up in the future once more tests are added to the barometer project and we figure out a way to set up the triggers.

Milestones

Left Deliverables

• Configuring the job to run upon PR submission to Github-Collectd
• Missing functional tests for plugins
• Plugins left for test coverage (hardware limitations, difficult to automate, special requirement)

Results

• Jenkins Job Design

Image Added

• Code Samples:
  • Tests for Capabilities Plugins
  • Docker Containers for Running Tests
  • Jenkins Job

Learnings & Insights

• Monitoring as a service
• Familiarity with open source development lifecycle
• Jenkins and CI pipeline set-up
• Best practices related to test case creation.

Project Report