Deploying test images using LXC

Containers are lightweight virtualization technology. LXC is a userspace interface for the Linux kernel containment features. Through a powerful API and simple tools, it lets Linux users easily create and manage system or application containers. The container provides a lightweight method to allow custom software to be used on the dispatcher. The container is used to provide transparent access.

LAVA supports LXC containers both as a standalone device type and as dynamic transparent environments in order to interact with external devices. In either case the LXC protocol is used.

Using LXC as Device Type

LXC is a device type of its own and devices could be added to dispatchers under this device type. A device of LXC device type is created within the dispatcher in which the device is configured, as illustrated in the following figure:

LXC standalone

The LXC device type uses the LXC protocol in order to share data elements across different actions within the job.

Protocol elements

protocols:
  lava-lxc:
    name: pipeline-lxc-test
    distribution: debian
    release: sid
    arch: amd64

Sample Job Definition

device_type: lxc
job_name: lxc-debian
timeouts:
  job:
    minutes: 30
  action:
    minutes: 5
priority: medium
visibility: public
metadata:
  source: https://git.linaro.org/lava-team/refactoring.git
  path: lxc-debian.yaml
protocols:
  lava-lxc:
    name: pipeline-lxc-test
    distribution: debian
    release: sid
    arch: amd64
actions:
- deploy:
    timeout:
      minutes: 30
    to: lxc
- boot:
    prompts:
    - '[root@(.*) /]#'
    timeout:
      minutes: 5
    method: lxc
- test:
    timeout:
      minutes: 5
    definitions:
    - repository: http://git.linaro.org/lava-team/lava-functional-tests.git
      from: git
      path: lava-test-shell/smoke-tests-basic.yaml
      name: smoke-tests

Namespaces

Namespaces were introduced to handle use-cases specific to LXC, but the principle can be expanded to other use-cases as and when required. In a job definition where multiple deploy, boot and test actions are specified, there must be a mechanism to describe how the actions are connected. This is the primary purpose of a namespace; it is the way to tie related actions together. The namespace itself is simply a label, test writers are advised to make the label chosen for each namespace meaningful for the purposes of the test job.

In the example below, there are two namespaces - one for the deploy, boot and test actions to perform inside the LXC and one for the deploy, boot and test actions to be performed on the DUT. To support this particular device, the test job needs to:

  1. deploy the container, including:

    • install software inside the container to control the device. In this case android-tools-fastboot.

  2. boot the container

  3. deploy files to the device

    • In this case, this connects to and turns on power to the device then uses software in the container to push files to the device using the bootloader.

  4. boot the device

  5. run a test shell on the device

  6. run a test shell in the container.

Note how the deploy, boot and test actions are interleaved. The use of namespaces is essential for the test shell in the container to be able to find and execute commands in the container. In this example, the software running in the container and the software running on the device need to be handled quite differently in each test shell. For example, when installing dependencies inside the container running Debian, the apt package manager is available. When installing dependencies in the test shell on the device, running OpenEmbedded, there might not be any package manager support. The namespace data is used to let each test shell identify the default shell and other data about the environment in each namespace.

 1actions:
 2- deploy:
 3    namespace: tlxc
 4    timeout:
 5      minutes: 15
 6    to: lxc
 7    packages:
 8    - android-tools-fastboot
 9
10- boot:
11    namespace: tlxc
12    prompts:
13    - 'root@(.*):/#'
14    - 'hikey:/'
15    timeout:
16      minutes: 5
17    method: lxc
18
19- deploy:
20    timeout:
21      minutes: 30
22    to: fastboot
23    namespace: hikey-oe
24    connection: lxc
25    images:
26      ptable:
27        url: http://images.validation.linaro.org/snapshots.linaro.org/openembedded/lkft/morty/hikey/rpb/4.9/83/bootloader/ptable-linux-8g.img
28        reboot: hard-reset
29      boot:
30        url: http://images.validation.linaro.org/snapshots.linaro.org/openembedded/lkft/morty/hikey/rpb/4.9/83/boot-0.0+AUTOINC+06e4def583-fb1158a365-r0-hikey-20170713193031-83.uefi.img
31        reboot: hard-reset
32      system:
33        url: http://images.validation.linaro.org/snapshots.linaro.org/openembedded/lkft/morty/hikey/rpb/4.9/83/rpb-console-image-hikey-20170808001820-83.rootfs.img.gz
34        compression: gz
35        apply-overlay: true
36        # ensure that this job raises a network interface with DHCP before relying on internet access
37    protocols:
38        lava-lxc:
39        - action: fastboot-deploy
40          request: pre-power-command
41          timeout:
42              minutes: 2
43
44- boot:
45    namespace: hikey-oe
46    auto_login:
47      login_prompt: 'login:'
48      username: root
49    prompts:
50    - 'root@hikey:~#'
51    timeout:
52      minutes: 5
53    method: uefi-menu
54    commands: fastboot
55    protocols:
56      lava-lxc:
57      # other action could be boot-fastboot
58      - action: uefi-commands
59        request: pre-os-command
60        timeout:
61          minutes: 2
62
63- test:
64    namespace: hikey-oe
65    timeout:
66      minutes: 5
67    definitions:
68    - repository: http://git.linaro.org/lava-team/lava-functional-tests.git
69      from: git
70      path: lava-test-shell/smoke-tests-basic.yaml
71      name: smoke-tests-basic-oe
72
73- test:
74    namespace: tlxc
75    timeout:
76      minutes: 5
77    definitions:
78    - repository: http://git.linaro.org/lava-team/lava-functional-tests.git
79      from: git
80      path: lava-test-shell/smoke-tests-basic.yaml
81      name: smoke-tests-basic-ubuntu

Note

The two test shells are almost identical but remember that all the results of this one test job will be reported together. The name of each test shell definition needs to be different for each test action. So the example uses name: smoke-tests-basic-oe for the hikey-oe namespace and name: smoke-tests-basic-ubuntu for the tlxc namespace.

See also

Namespace in the glossary, Connections and namespaces

Using the LXC protocol to support Android

LAVA Android Naming Conventions

  • production image - a build of Android which, when deployed to a device, means that the device is not visible to adb. This is typically how a device is configured when first sold to the consumer.

  • developer image - a build of Android which, when deployed to a device, means that the device is visible to adb. Devices configured this way will be able to have the image replaced using any machine, just by connecting a suitable cable, so these images are not typically deployed onto hardware which will be sold to the customer without having this image replaced with a production image.

Introduction

Installing tools like adb and fastboot on the dispatcher can be problematic. Some of these issues arise from the need to put many different types of devices onto a single dispatcher, other issues arise from needing to use different versions of the build on the devices. Testing an old system may require downgrading support like openjdk, new devices or new builds may require upgrading the same support. Containers isolate this variation so that each testjob can have a suitable container instead of needing to deal with changes on the dispatcher:

  1. Shared lock issues - Tools can require use of flock and similar methods to distinguish a connection to one device from another.

  2. Version disparities - different device versions, different OS versions, may require different support in debug tools like adb and fastboot.

  3. hardware issues - USB hub variability.

See also

Deploying LXC devices for more information on the administration of LXC for LAVA.

LXC in LAVA

Using the lava-lxc protocol, a Lava Test Shell is provided inside the LXC to support installing and configuring whatever tools, packages and files which the testjob will need to use. Installing adb in this test shell removes the need to have a POSIX type shell on the device. Files can be pushed and pulled from the device and executed using the Android support in the image.

Requirements and Limitations

  1. The image deployed to the device must enable the Android Debug Bridge, i.e. a developer image. This means enabling developer access over USB or TCP. This rules out the use of production images.

  2. A list of packages to install into the bare container to provide the necessary tools to communicate with the device.

  3. The LXC depends on underlying kernel architecture. For armel, armhf, etc. dispatcher should run on these architectures.

  4. Each distribution has its own template and the templates do not have common options. It can be difficult to have generic support for all distributions.

  5. namespaces to relate different job actions to run in the LXC and for the device.

Protocol elements

protocols:
  lava-lxc:
    name: lxc-hikey-test
    template: debian
    distribution: debian
    release: jessie
    mirror: http://mirror.bytemark.co.uk/debian
    verbose: true

actions:
# DEPLOY_LXC_BLOCK
- deploy:
    namespace: tlxc
    timeout:
      minutes: 5
    to: lxc
    packages:
    - android-tools-adb
    - android-tools-fastboot

# BOOT_LXC_BLOCK

Feedback from the device

Actions within the LXC can cause the device to emit messages on the serial console. Some devices can have problems maintaining the serial connection if this data is not flushed and the data itself can be useful to test writers to debug issues and failures.

LAVA automatically reads from all other namespaces whilst processing the test shell in another namespace and outputs this as feedback data. When viewing a test job log file, feedback can be turned on or off using the buttons at the top of the log file.

To support feedback, the lava-test-shell Individual connection overrides is set to 10 seconds by default. (There are no suitable prompts to match, so reading feedback continues until the connection timeout is reached, without failing the test shell itself.)

See also

Timeouts

Differences between LXC releases

The release specified in the lava-lxc protocol will determine some of the packages which will need to be installed in the container. In particular, any container based on a Debian later than jessie will need two packages to be added to the setup of the container before the container can be used: systemd and systemd-sysv. These two packages must be specified in the deployment list.

In addition, some packages will have been renamed between releases. For example, android-tools-adb exists in Debian unstable but it is an old build and will at some point be replaced by adb which is also available in unstable but not in jessie.

Caution

Always check the availability of the packages needed for particular releases by using a local chroot or VM. Only packages which are included in the specified release can be installed using the deployment list. Packages from other repositories will have to be installed using the test definition.

Sample Job Definition

device_type: hi6220-hikey
job_name: lxc-hi6220-hikey
timeouts:
  job:
    minutes: 60
  action:
    minutes: 15
  connection:
    minutes: 2
priority: medium
visibility: public

metadata:
  source: https://git.linaro.org/lava-team/refactoring.git
  path: hi6220-hikey.yaml

protocols:
  lava-lxc:
    name: lxc-hikey-test
    template: debian
    distribution: debian
    release: jessie
    mirror: http://mirror.bytemark.co.uk/debian
    verbose: true

actions:
# DEPLOY_LXC_BLOCK
- deploy:
    namespace: tlxc
    timeout:
      minutes: 5
    to: lxc
    packages:
    - android-tools-adb
    - android-tools-fastboot

# BOOT_LXC_BLOCK
- boot:
    namespace: tlxc
    prompts:
    - 'root@(.*):'
    - 'hikey: '
    timeout:
      minutes: 5
    method: lxc

# DEPLOY_TARGET_BLOCK
- deploy:
    timeout:
      minutes: 15
    namespace: droid
    to: fastboot
    images:
      ptable:
        url: http://example.com/hikey/ptable-aosp-8g.img
        reboot: hard-reset
      boot:
        url: http://example.com/hikey/boot.img.xz
        compression: xz
        reboot: hard-reset
      cache:
        url: http://example.com/hikey/cache.img.xz
        compression: xz
      userdata:
        url: http://example.com/hikey/userdata.img.xz
        compression: xz
      system:
        url: http://example.com/hikey/system.img.xz
        sha256sum: e0e82b5adfae84ff97f4f6488e5b4c64b0dfc7ad8a37b4bcbb887d9f85a6be0a
        compression: xz
    protocols:
      lava-lxc:
      - action: fastboot-deploy
        request: pre-power-command
        timeout:
          minutes: 2

# BOOT_TARGET_BLOCK
- boot:
    namespace: droid
    prompts:
    - 'root@(.*):/#'
    - 'hikey:/'
    timeout:
      minutes: 15
    method: fastboot

- test:
    namespace: tlxc
    timeout:
      minutes: 10
    definitions:
    - from: inline
      repository:
        metadata:
          format: Lava-Test Test Definition 1.0
          name: check-devices
          description: "LXC dev list"
        run:
          steps:
          - /sbin/ifconfig
      name: lxc-dev-inline
      path: inline/lxc-dev.yaml
    - repository: https://git.linaro.org/lava-team/refactoring.git/
      from: git
      path: android/lava-android-basic-lxc.yaml
      name: v2-make-adb-connection