LAVA Installation

LAVA Installation

The default install provides an Apache2 config suitable for a LAVA server at http://localhost/ once enabled.

See Packaging lava-server for distributions for more information or for debugging.

Requirements to Consider Before Installing LAVA

Software Requirements

See Debian-based distributions for instructions.

We currently recommend installing LAVA on Debian unstable, stretch or jessie. Installations using jessie (the current Debian stable release) should use updates available in jessie-backports.

Contributions to support other distributions are welcome.

If you’d like to help us with other distributions feel free to contact us using the lava-devel mailing list.

Note

See TFTP support requirement for changes in the tftp software requirements after the 2015.8 release.

Hardware Requirements

A small LAVA instance can be deployed on any modest hardware. We recommend at least one 1GB of RAM for runtime activity (this is shared, on a single host, among the database server, the application server and the web server). For storage please reserve about 20GB for application data, especially if you wish to mirror current public LAVA instance used by Linaro. LAVA uses append-only models so the storage requirements will grow at about several GB a year.

A small LAVA instance can be deployed on reasonably modest hardware. [2] We recommend:

  • At least 1GB of RAM for runtime activity (this is shared, on a single host, among the database server, the application server and the web server)
  • At least 20GB of storage for application data, job log files etc. in addition to the space taken up by the operating system.

Footnotes

[1]See the section Setting Up Serial Connections to LAVA Devices for details of configuring serial connections to devices
[2]If you are deploying many devices and expect to be running large numbers of jobs, you will obviously need more RAM and disk space

Device requirements

Devices you wish to deploy in LAVA need to be:
  • Physically connected to the server via usb, usb-serial, or serial or
  • connected over the network via a serial console server or
  • a fastboot capable device accessible from the server or
  • an emulated virtual machines and/or simulators that allow a serial connection

Multi-Node hardware requirements

If the instance is going to be sent any job submissions from third parties or if your own job submissions are going to use Multi-Node, there are additional considerations for hardware requirements.

Multi-Node is explicitly about synchronising test operations across multiple boards and running Multi-Node jobs on a particular instance will have implications for the workload of that instance. This can become a particular problem if the instance is running on virtualised hardware with shared I/O, a limited amount of RAM or a limited number of available cores.

Note

Downloading, preparing and deploying test images can result in a lot of synchronous I/O and if this instance is running the server and the dispatcher, running synchronised Multi-Node jobs can cause the load on that machine to rise significantly, possibly causing the server to become unresponsive.

It is strongly recommended that Multi-Node instances use a separate dispatcher running on non-virtualised hardware so that the (possibly virtualised) server can continue to operate.

Also, consider the number of boards connected to any one dispatcher. MultiNode jobs will commonly compress and decompress several test image files of several hundred megabytes at precisely the same time. Even with a powerful multi-core machine, this has been shown to cause appreciable load. It is worth considering matching the number of boards to the number of cores for parallel decompression and matching the amount of available RAM to the number and size of test images which are likely to be in use.

Which release to install

LAVA makes regular monthly releases called production releases which match the packages installed onto http://validation.linaro.org/. These releases are also uploaded to Debian unstable and backports (see Debian-based distributions). Packages uploaded to Debian typically migrate automatically into the current Ubuntu development release - at time of writing that is Ubuntu Utopic Unicorn, scheduled for release as 14.10. production releases are tracked in the release branch of the upstream git repositories.

Interim releases are made available from the the staging-repo.

During periods when the internal transitions within Debian require that lava-server is unable to migrate into the testing suite, users running Debian Jessie (testing) can obtain the same release using the people.linaro.org repository to provide packages which are not present in Debian Jessie.

The lava-dev package includes scripts to assist in local developer builds directly from local git working copies which allows for builds using unreleased code, development code and patches under review.

If in doubt, install the production release of lava-server from official distribution mirrors. (Backports are included on Debian mirrors.)

Installation Types

Single Master Instance installation

A single instance runs the web frontend, the database, the scheduler and the dispatcher on a single machine. If this machine is also running tests, the device (or devices) under test (DUT) will also need to be connected to this machine, possibly over the network, using USB or using serial cables.

To install a single master instance and create a superuser, refer to Debian-based distributions installation.

The old distributed_instance installation method has been deprecated as the refactoring introduces a much improved architecture for remote workers using ZMQ.

What is the Pipeline?

Note

Linaro production systems in the Cambridge lab began to migrate to the pipeline with the 2016.2 production release whilst retaining support for the deprecated model until the migration is complete. Deprecated support is due to be removed in 2017.

In parallel with the deprecated Single Master Instance installation and distributed_instance models, the dispatcher refactoring introduces changes and new elements which should not be confused with the previous production models. It is supported to install LAVA using solely the new design but there are some considerations regarding your current device usage. Submission requirements and device support can change before and during a migration to the new design.

This documentation includes notes on the new design, so to make things clearer, the following terms refer exclusively to the new design and have no bearing on Single Master Instance installation or Distributed Instance installation installation methods which are being used for current production instances in the Cambridge lab.

  1. pipeline
  2. refactoring
  3. device dictionary
  4. ZMQ

The pipeline model also changes the way that results are gathered, exported and queried, replacing the bundle stream, result bundle and filter dashboard objects. This new results functionality only operates on pipeline test jobs and is ongoing development, so some features are incomplete and likely to change in future releases. Admins can choose to not show the new results app, for example until pipeline devices are supported on that instance, by setting the PIPELINE to false in /etc/lava-server/settings.conf - make sure the file validates as JSON before restarting apache:

"PIPELINE": false

If the value is not set or set to true, the Results app will be displayed.

A note on wsgi buffers

When submitting a large amount of data to the django application, it is possible to get an HTTP 500 internal server error. This problem can be fixed by appending buffer-size = 65535 to /etc/lava-server/uwsgi.ini

Automated installation

Using debconf pre-seeding

debconf can be easily automated with a text file which contains the answers for debconf questions - just keep the file up to date if the questions change. For example, to preseed a worker install:

# cat preseed.txt
lava-server   lava-worker/db-port string 5432
lava-server   lava-worker/db-user string lava-server
lava-server   lava-server/master boolean false
lava-server   lava-worker/master-instance-name string default
lava-server   lava-worker/db-server string snagglepuss.codehelp
lava-server   lava-worker/db-pass string werewolves
lava-server   lava-worker/db-name string lava-server

Insert the seeds into the debconf database:

debconf-set-selections < preseed.txt
# debconf-show lava-server
* lava-worker/master-instance-name: default
* lava-server/master: false
* lava-worker/db-pass: werewolves
* lava-worker/db-port: 5432
* lava-worker/db-name: lava-server
* lava-worker/db-server: snagglepuss.codehelp
* lava-worker/db-user: lava-server

The strings available for seeding are in the Debian packaging for the relevant package, in the debian/<PACKAGE>.templates file.

User authentication

LAVA frontend is developed using Django web application framework and user authentication and authorization is based on standard Django auth subsystems. This means that it is fairly easy to integrate authentication against any source for which Django backend exists. Discussed below are tested and supported authentication methods for LAVA.

Note

The previous OpenID support is not compatible with newer versions of django (versions 1.9 or later) and as been disabled.

Local Django user accounts are supported. When using local Django user accounts, new user accounts need to be created by Django admin prior to use.

Support for OAuth2 is under investigation in LAVA.

Using Lightweight Directory Access Protocol (LDAP)

LAVA server could be configured to authenticate via Lightweight Directory Access Protocol ie., LDAP. LAVA uses django_auth_ldap backend for LDAP authentication.

Your chosen LDAP server is configured using the following parameters in /etc/lava-server/settings.conf (JSON syntax):

"AUTH_LDAP_SERVER_URI": "ldap://ldap.example.com",
"AUTH_LDAP_BIND_DN": "",
"AUTH_LDAP_BIND_PASSWORD": "",
"AUTH_LDAP_USER_DN_TEMPLATE": "uid=%(user)s,ou=users,dc=example,dc=com",
"AUTH_LDAP_USER_ATTR_MAP": {
  "first_name": "givenName",
  "email": "mail"
},

Use the following parameter to set a custom LDAP login page message:

"LOGIN_MESSAGE_LDAP": "If your Linaro email is first.second@linaro.org then use first.second as your username"

Other supported parameters are:

"AUTH_LDAP_GROUP_SEARCH": "ou=groups,dc=example,dc=com",
"AUTH_LDAP_USER_FLAGS_BY_GROUP": {
  "is_active": "cn=active,ou=django,ou=groups,dc=example,dc=com",
  "is_staff": "cn=staff,ou=django,ou=groups,dc=example,dc=com",
  "is_superuser": "cn=superuser,ou=django,ou=groups,dc=example,dc=com"
}

Note

Apart from the above supported parameters, in order to do more advanced configuration, make changes to /usr/lib/python2.7/dist-packages/lava_server/settings/common.py

Restart lava-server and apache2 services if this is changed.

LAVA server branding support

The icon, link, alt text, bug URL and source code URL of the LAVA link on each page can be changed in the settings /etc/lava-server/settings.conf (JSON syntax):

"BRANDING_URL": "http://www.example.org",
"BRANDING_ALT": "Example site",
"BRANDING_ICON": "https://www.example.org/logo/logo.png",
"BRANDING_HEIGHT": 26,
"BRANDING_WIDTH": 32,
"BRANDING_BUG_URL": "http://bugs.example.org/lava",
"BRANDING_SOURCE_URL": "https://github.com/example/lava-server",

If the icon is available under the django static files location, this location can be specified instead of a URL:

"BRANDING_ICON": "path/to/image.png",

There are limits to the size of the image, approximately 32x32 pixels, to avoid overlap.

The favicon is configurable via the Apache configuration:

Alias /favicon.ico /usr/share/lava-server/static/lava-server/images/logo.png

LAVA Dispatcher network configuration

/etc/lava-dispatcher/lava-dispatcher.conf supports overriding the LAVA_SERVER_IP with the currently active IP address using a list of network interfaces specified in the LAVA_NETWORK_IFACE instead of a fixed IP address, e.g. for LAVA installations on laptops and other devices which change network configuration between jobs. The interfaces in the list should include the interface which a remote worker can use to serve files to all devices connected to this worker.

Setting Up Serial Connections to LAVA Devices

Ser2net daemon

ser2net provides a way for a user to connect from a network connection to a serial port, usually over telnet.

http://ser2net.sourceforge.net/

ser2net is a dependency of lava-dispatcher, so will be installed automatically.

Example config (in /etc/ser2net.conf):

#port:connectiontype:idle_timeout:serial_device:baudrate databit parity stopbit
7001:telnet:0:/dev/serial_port1:115200 8DATABITS NONE 1STOPBIT

Note

In the above example we have the idle_timeout as 0 which specifies a infinite idle_timeout value. 0 is the recommended value. If the user prefers to give a positive finite idle_timeout value, then there is a possibility that long running jobs may terminate due to inactivity on the serial connection.

StarTech rackmount usb

W.I.P

  • udev rules:

    SUBSYSTEM=="tty", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6001", ATTRS{serial}=="ST167570", SYMLINK+="rack-usb02"
    SUBSYSTEM=="tty", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6001", ATTRS{serial}=="ST167569", SYMLINK+="rack-usb01"
    SUBSYSTEM=="tty", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6001", ATTRS{serial}=="ST167572", SYMLINK+="rack-usb04"
    SUBSYSTEM=="tty", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6001", ATTRS{serial}=="ST167571", SYMLINK+="rack-usb03"
    

This will create a symlink in /dev called rack-usb01 etc. which can then be addressed in the Ser2net daemon config file.

Contact and bug reports

Please report bugs using bugzilla: https://bugs.linaro.org/enter_bug.cgi?product=LAVA%20Framework

You can also report bugs using reportbug and the Debian Bug Tracking System: https://bugs.debian.org/cgi-bin/pkgreport.cgi?pkg=lava-server

Feel free to contact us at validation (at) linaro (dot) org and on the #linaro-lava channel on OFTC.