Barometer leverages collectd, an opensource project to collect system metrics ,and works closely with the collectd community. Collectd has various plugins that receive upstream contributions from the Barometer team as well. The aim of this project is to Increase and automate the test coverage for collectd to ensure it's integrity for use in Barometer .

Collectd currently has some unit tests in place but no functional tests as of now. The aim of the project is to develop functional test cases for collectd plugins that receive attention from the barometer team. We will also set up a Jenkins build pipeline to post the results of these tests back to  the Collectd Github repository (for every open PR).

Goals

  1. Figure out how to run tests locally. Currently, functest is being used. We might need to investigate the feasibility and if required, set up new testing tool/architecture.
  2. The existing tests defined for functest/xtesting are old and cover only a subset of plugins that existed 2 years ago. 
    1. So tests will be needed for most plugins. There are a bunch of plugins that barometer uses:
      1. Log-parser -> Easy plugin to understand and get started
      2. Capabilities Plugin
      3. SNMP Write Plugin
      4. Memory RAS
  3. A review will be needed for the tests in baro_tests/ and determine whether these are suitable for the collectd community
  4. Set up triggering a Jenkins build on a nightly basis ( can be triggered upon a github PR at a later stage)
  5. Contribute to making the barometer docs better, wherever there is confusion and things aren't mentioned clearly to make things easy for future contributors.

Tasks

Week

Activity

Week 1 - Week 3


  • Explore Github Status API & Check how results could be posted back to Github with Jenkins
  • Get familiar with opnfv Jenkins CI
  • Selection of plugins for writing tests based on activity in collectd
  • Update barometer set-up documentation in the repository to align with the docs 
  • A review of the tests in baro_tests/ ,  fix, update and get them working
  • Decide upon the testing framework and architecture for writing tests.

Week 4 - Week 6


  • Test case development for log parser plugin
  •  Documenting the test cases and expected output and overall Jenkins CI architecture.
  • Create a small demo to illustrate the working of the final project, like a POC

Week 7 - Week 9

  • Test case creation for capabilities plugin
  • Set up docker containers for running tests
  • Optimizing build time for images for running tests

Week 10 - Week 12

  • Set up the jenkins job and test on OPNFV Jenkins sandbox

  • Finishing up tasks, cleanup, possible demo, user guide etc. More focus on documenting existing work. Can possibly take up more plugins if we have cycles.

Implementation Details

We begin with a review of the tests in baro_tests/  ensure that they work well and determine whether these are suitable for the collectd community and for Barometer. The testing tool is not yet finalized and will be done based on further discussion with the Barometer community, keeping all the use cases in mind.

Fig.1.1 High-level architecture of the Jenkins CI setup


We will then proceed by writing test cases for a few simple plug-ins (like the log-parser and Network-Plugin among other plugins, details for which can be found in the section above). The test case will be hosted in the barometer repository for the time being (and the framework that we will use will most probably be x-test or functest), the deliverables section also mentions a bunch of plugins with some manual tests which will be automated as part of this project as well. To be inclusive of all plugins under development, there are a few plugins that we have decided to try and add testing for as they are under active development.


The next step will be to create a CI pipeline using the Jenkins GitHub plugin, which should trigger builds/running of the tests when a PR is submitted to the collectd repository (Fig 1.1).

The tentative steps to make this happen are:

  1. Setup and install the GitHub PR builder plugin in Jenkins (ticket raised).
  2. Get the Intel-POD13 connected to Jenkins as a slave.
  3. Configuring the Jenkins slave in POD13 .
  4. Support will be needed from the Collectd community to configure the GitHub repository and web-hooks

Once the architecture and pipelines are set up, testing the overall functionality will be our focus. We will move ahead with adding tests for more plugins.

However, the list of plugins is tentative and subject to change upon further discussions with Barometer and collectd community.

Note: The Jenkins Job set up for running the tests cannot be triggered upon PR submission to Collectd github repository as the tool decided to be used was not quite the right fit. It can be taken up in the future once more tests are added to the barometer project and we figure out a way to set up the triggers.

Milestones

Evaluation Date

Evaluation Criteria

Q1 June 19

  • Reviewed the tests in baro_tests/
  • Test framework and tools have been decided

Q2 July 10

  • Looked at existing test cases
  • Added test cases for (2) new plugins
  • Setup a working example of one of the plugins in the sandbox/pod 13 Jenkins (depending on availability)

Q3 July 31

  • Added tests for more plugins (3), mentioned in the timeline
  • Setup CI pipeline
  • Ensure tests are automated

Q4 August 21

  • Ensure CI runs smoothly
  • Ensure proper documentation 

Left Deliverables

  • Configuring the job to run upon PR submission to Github-Collectd
  • Missing functional tests for plugins
  • Plugins left for test coverage (hardware limitations, difficult to automate, special requirement)

Results

  • Jenkins Job Design

Learnings & Insights

  • Monitoring as a service
  • Familiarity with open source development lifecycle
  • Jenkins and CI pipeline set-up
  • Best practices related to test case creation.

Project Report


  • No labels