Gold standard test jobs

The next step after the first example job, is to move on to gold standard test jobs. These include a test job for QEMU and test jobs for readily available ARM devices like cubietruck and beaglebone-black.

See also

Creating a gold standard image

Test writers are strongly recommended to follow these examples to have a known working setup for later test jobs. These standard test jobs are part of the regular functional testing performed by the LAVA software team prior to each release. If you have problems debugging a test job failure on any instance of LAVA, your recommended first action is to submit the closest standard test job and compare the results against other instances like staging.validation.linaro.org. To make this easier, all test writers are recommended to use version control for all test jobs and to keep a local copy of the closest standard test job for quick reference.

Note

A gold standard deployment for one device type is not necessarily supported for a second device type. Some devices will never be able to support all deployment methods due to hardware constraints or the lack of kernel support. This is not a bug in LAVA. If a particular deployment is supported but not stable on a device type, there will not be a gold standard image for that deployment. Any issues in the images using such deployments on that type are entirely down to the test writer to fix.

Note

Device type templates and other instance-specific configuration could affect the standard test jobs on specific instances. If a standard test job fails, check with the admins for that particular instance. The standard tests described here use support in the default device-type templates shipped with LAVA and are routinely tested on staging.validation.linaro.org.

Metadata

Standard test jobs include metadata which links the test job back to the build information, logs and scripts.

Metadata consists of a series of key value pairs where the key must be a single word (no whitespace, underscores and hyphens are allowed). Keys must be unique within the metadata.Keys used in metadata should be consistent across particular groups of test jobs with data specific to that test job in the value.

Ensure that you change the metadata to point at your local repository so that you can easily distinguish between the results with and without your modifications:

metadata:
  source: https://gitlab.com/lava/lava
  path: doc/v2/examples/test-jobs/qemu-amd64-standard-stretch.yaml

Automated test jobs should always use the metadata to specify:

  • the source repository of the job definition

  • the path to the job definition within the source repository

  • the details of the build

  • links to the build logs

  • links to the build scripts.

These details allow other users to adapt and modify the build artifacts for further tests.

The set of gold standard jobs has been defined in association with the Linaro QA team. These jobs will provide a known baseline for test definition writers, in a similar manner to the way the existing QA test definitions provide a base for more elaborate testing.

Note

Metadata also exists in the Lava Test Shell Definition but this is only used when writing the test shell definition, it is not picked up by LAVA. It remains useful to record the maintainer of a test shell definition, the scope and the kinds of devices for which the definition is expected to be useful. see also Test Definitions To explicitly add limited metadata to the job from the Lava Test Shell Definition, see Recording test case references. The job will automatically contain metadata relating to the URL, commit hash and path of the test shell definition(s) used in the test job.