Project Plan - Barometer CI setup and test case development

Barometer uses collectd to collect the system metrics and works closely with the collectd community. Collectd has various plugins that receive upstream contributions from the OPNFV community, more specifically the Barometer project. Increasing the automated test coverage benefits collectd as well as Barometer.

Collectd currently has some unit tests in place but no functional tests as of now. The aim of the project is to develop functional test cases for a bunch of collectd plugins that receive attention from the barometer team. Finally, we are looking to set up a Jenkins build pipeline which reports the status of tests back to Collectd Github repository. This way, we contribute to the testing of collectd plugins that, as well as to ensure that the project's integrity is maintained.

Goals

1. Decide on how to run tests locally. Currently, functest is being used. We might need to investigate the feasibility and if required, set up new testing tool/architecture.
2. The existing tests defined for functest/xtesting are old and cover only a subset of plugins that existed 2 years ago. 
1. So tests will be needed for most plugins. There are a bunch of plugins that barometer uses:
   1. Log-parser -> Easy plugin to understand and get started
   2. Network Plugin -> For influx  db 
   3. SNMP Write Plugin
   4. Memory RAS
2. Plugins from the Collectd PR's(Plus sign indicate a high frequency of PR's):
   1. ethstat plugin
   2. SNMP +
   3. Processes plugin +
   4. SQL plugin +
   5. Exec Plugin
   6. Threshold plugin +
   7. Memory plugin +
   8. Bind Plugin
   9. Network Plugin + 
3. A review will be needed for the tests in baro_tests/ and determine whether these are suitable for the collectd community
4. Set up triggering a Jenkins build when a PR is submitted to the collectd repository, run the job on node 5 of intel pod13 
5. Contribute to making the barometer docs better, wherever there is confusion and things aren't mentioned clearly to make things easy for future contributors.

Tasks

Implementation Details

We begin with a review of the tests in baro_tests/  ensure that they work well and determine whether these are suitable for the collectd community and for Barometer. The testing tool is not yet finalised and will be done based on further discussion with the Barometer community keeping all the use cases in mind.

We then proceed with writing test cases for a few simple plug-ins like the log-parser and Network-Plugin. The test case will be hosted in the barometer repository for the time being (and the framework that we will use will most probably be x-test or functest). Subsequently, we move towards creating a CI pipeline using Jenkins which can trigger builds when a PR is submitted. Fig 1.1 shows the possible high-level design of the testing architecture. The possible steps are:

1. Setup and install the Github PR builder plugin in Jenkins (ticket raised)
2. Get the Intel-POD13 connected to Jenkins as a slave
3. Configuring the Jenkins slave in POD13 
4. Support will be needed from the collectd community to configure the Github repository and web-hooks

Once the architecture and pipelines are set up, testing the overall functionality will be our focus. 

We move ahead with adding tests for more plugins. Deliverables mention a bunch of plugins which are enabled in Barometer and contains a list of manual tests. We wish to automate existing tests for these plugins. Furthermore, Some more plugins have also been selected based on the active PR's in the collectd Github repository. Since these are the plugins under active development, having functional test cases for them makes more sense.

However, the list of plugins is tentative and subject to change upon further discussions with Barometer and collectd community.

Milestones