Replacing VMGroups using Secondary Connections¶
One of the primary use cases for secondary connections is the ability to watch a job from multiple perspectives. This includes being able to start a task in one job and interact with that task from another job. For example, if one job runs QEMU to start a VM on the host machine, another connection can test that the VM responds on the relevant ports, depending on what is installed inside the image downloaded onto the host machine. More complex jobs could start a daemon in a debugger and connect to the output of the daemon itself in one job and to the output of the debugger in another job. This does not necessarily require the watching job to do anything, the job could just record the output.
The role of a Virtual Machine Group is to arrange a test job such that a host device, which must have virtualization support, boots into a base image and installs a daemon to allow other test jobs to connect - typically this will be Secure Shell connections (ssh).
Once the secondary connections are logged in, any program accessible in the
base image can be used by the secondary connection. As highlighted in the link above, tasks involving package installation or other
system-wide operations are best done in the test job managing the host device
and preferably before the secondary connection jobs start. So the
install: deps:
of all jobs in the group should all be collated into the
test definition(s) of the host role.
The secondary connection support performs all the synchronization steps that are required prior to the test shell starting in each of the secondary test jobs. The MultiNode API remains available for any synchronization requirements during the test shell operation. Importantly the test shell running on the host device should wait for a lava-sync before completing the test shell definition. Otherwise the secondary connection jobs may find that the underlying system suddenly goes away unexpectedly.
With all of that in place, the actual operation of a test involving multiple
virtual machines is little different to a test involving a daemon or long
running task. Something in the secondary job test shell needs to start the
process and something needs to end the process so that the final sync
can
occur. If the task is not capable of terminating itself, it will be necessary
for one test shell (often the host role on the host device) to be able to
terminate it or the test shell will not complete. The choice of how to do this
is left to the test writer.
The command line to use to start the VM(s) on the host device is controlled entirely by the test specification. This is different to the old deprecated vmgroups support. Image files and other components can be present in the image deployed to the host device or downloaded by the host device or after the secondary connection is logged in. The delayed start requirements are already covered by the secondary connections, so the test shell can start the VM immediately, if desired, or do any preliminary tests first.
Structure of an example job for a mustang¶
This job uses an nfsrootfs for the host device on a mustang (using the U-Boot
pipeline support for the mustang) as the host role. The NFS is a base Debian
Jessie arm64, so the initial lava test shell operation on the host is to
install openssh-server
and then use the lava-echo-ipv4
helper to
declare the IPv4 address of the host machine.
The secondary connection support picks up the IP address in the pipeline actions, so the id of the message needs to be declared to the relevant action. The test shell sends:
- ifconfig
Then it tells the secondary connections to start:
- lava-send ipv4 ipaddr=$(lava-echo-ipv4 eth0)
This tells the dispatcher for the guest
role to start the deployment. The
test definition files for secondary connection jobs are copied onto the host
device immediately before the connection is made, then the contents are
unpacked to be able to run the test shell. The tarball will exist on the host
device as /<job_id>-overlay-<level>.tar.gz
. For job ID 74585
and level
1.1.5
, this would result in /74585-overlay-1.1.5.tar.gz
.
In this example job, the host test shell does nothing except wait for the clients to complete their own tests before proceeding to run a final test shell of smoke tests.
The receiving action is declared as:
protocols:
lava-multinode:
- action: prepare-scp-overlay
request: lava-wait
# messageID matches hostID
messageID: ipv4
message:
# the key of the message matches value of the host_key
# the value of the message gets substituted
ipaddr: $ipaddr
Note
the messageID specified to lava-send
(ipv4), is also the
messageID specified to the prepare-scp-overlay
action within the
pipeline. In addition, the content of the sent message is declared.
lava-send
uses the syntax key=value
, the YAML uses the equivalent
syntax of key: value
. As the value will be substituted with the real IP
address, the value in the YAML is marked as replaceable using the $
prefix.
The message parameters are passed to the boot
action of the guest
role
so that the details can be retrieved:
Finally, the jobs with the guest
role are booted - this establishes the
connection between the dispatcher and the host device using ssh. Once logged
in, each job completes the boot stage and starts the test shell for that job.
- boot:
role:
- guest
method: ssh
connection: ssh
prompts:
- 'root@linaro-developer:'
parameters:
hostID: ipv4 # messageID
Notes¶
Starting the VM(s) is for the test writer to implement, depending on the support required and the objectives of the test. In the example below, the host device simply runs the smoke tests definition in the position where images could be downloaded and QEMU started.
Use inlines - this example keeps all of the MultiNode API calls to the inline definitions. This is a recommended practice and future developments will make it easier to match up the synchronization calls from inline definitions. So, to adapt this job to do other tasks while the secondary connections jobs are running those test shells, move the final
lava-sync clients
to another inline definition and do the other calls in between.Completion - It is useful for the host device test shell to do something after completing the final
lava-sync
or the host device may complete the test shell before the secondary connections can logout correctly, resulting in the secondary connection jobs being incomplete. A final test definition of smoke tests or other quick checks could be useful.
View or Download mustang-ssh-guest.yaml
1job_name: ssh guest with standard images
2
3timeouts:
4 job:
5 minutes: 60
6 action:
7 minutes: 20
8 connection:
9 minutes: 2
10
11priority: medium
12visibility: public
13
14# PROTOCOLS_BLOCK
15protocols:
16 lava-multinode:
17 roles:
18 guest:
19 connection: ssh
20 count: 3
21 expect_role: host
22 host_role: host
23 request: lava-start
24 timeout:
25 minutes: 15
26 host:
27 count: 1
28 device_type: mustang
29 timeout:
30 minutes: 10
31
32metadata:
33 docs-source: actions-deploy-to-ssh
34 docs-filename: examples/test-jobs/mustang-ssh-guest.yaml
35 build-readme: http://images.validation.linaro.org/snapshots.linaro.org/components/lava/standard/debian/stretch/arm64/3/debian-arm64-readme.html
36 build-console: https://ci.linaro.org/view/lava-ci/job/lava-debian-stretch-arm64/3/console
37 build-script: http://images.validation.linaro.org/snapshots.linaro.org/components/lava/standard/debian/stretch/arm64/3/build-foreign-nfs.sh
38
39actions:
40- deploy:
41 role:
42 - host
43 authorize: ssh
44 kernel:
45 url: http://images.validation.linaro.org/snapshots.linaro.org/components/lava/standard/debian/stretch/arm64/3/vmlinuz-4.9.0-2-arm64
46 type: zimage
47 modules:
48 url: http://images.validation.linaro.org/snapshots.linaro.org/components/lava/standard/debian/stretch/arm64/3/modules.tar.gz
49 compression: gz
50 nfsrootfs:
51 url: http://images.validation.linaro.org/snapshots.linaro.org/components/lava/standard/debian/stretch/arm64/3/stretch-arm64-nfs.tar.gz
52 compression: gz
53 ramdisk:
54 url: http://images.validation.linaro.org/snapshots.linaro.org/components/lava/standard/debian/stretch/arm64/3/initrd.img-4.9.0-2-arm64
55 compression: gz
56 timeout:
57 minutes: 12
58 to: tftp
59
60# DEPLOY_SSH_BLOCK
61- deploy:
62 role:
63 - guest
64 connection: ssh
65 protocols:
66 lava-multinode:
67 - action: prepare-scp-overlay
68 request: lava-wait
69 # messageID matches hostID
70 messageID: ipv4
71 message:
72 # the key of the message matches value of the host_key
73 # the value of the message gets substituted
74 ipaddr: $ipaddr
75 timeout: # delay_start timeout
76 minutes: 21
77 timeout:
78 minutes: 22
79 to: ssh
80
81# BOOT_HOST_BLOCK
82- boot:
83 role:
84 - host
85 auto_login:
86 login_prompt: 'login:'
87 username: root
88 commands: nfs
89 method: grub-efi
90 prompts:
91 - 'root@stretch:'
92 timeout:
93 minutes: 5
94
95- boot:
96 role:
97 - guest
98 method: ssh
99 connection: ssh
100 prompts:
101 - 'root@linaro-developer:'
102 parameters:
103 hostID: ipv4 # messageID
104 host_key: ipaddr # message key
105 timeout:
106 minutes: 23
107
108- test:
109 role:
110 - host
111 definitions:
112 - from: inline
113 name: ssh-inline
114 path: inline/ssh-install.yaml
115 repository:
116 install:
117 deps:
118 - openssh-server
119 metadata:
120 description: install step
121 format: Lava-Test Test Definition 1.0
122 name: install-ssh
123 os:
124 - debian
125 scope:
126 - functional
127 install:
128 deps:
129 - openssh-server
130 - net-tools
131 - iproute2
132 run:
133 steps:
134 - ls -al /root/.ssh/
135 - ifconfig
136 - lava-send ipv4 ipaddr=$(lava-echo-ipv4 eth0)
137 - lava-send lava_start
138 - lava-sync clients
139 - repository: http://git.linaro.org/lava-team/lava-functional-tests.git
140 from: git
141 path: lava-test-shell/smoke-tests-basic.yaml
142 name: smoke-tests
143 - from: git
144 repository: http://git.linaro.org/lava-team/lava-functional-tests.git
145 path: lava-test-shell/params/nfs.yaml
146 name: nfs-dns
147 - repository: http://git.linaro.org/lava-team/lava-functional-tests.git
148 from: git
149 path: lava-test-shell/single-node/singlenode03.yaml
150 name: singlenode-advanced
151 timeout:
152 minutes: 30
153
154- test:
155 role:
156 - guest
157 definitions:
158 - repository: http://git.linaro.org/lava-team/lava-functional-tests.git
159 from: git
160 path: lava-test-shell/smoke-tests-basic.yaml
161 name: pre-smoke-tests
162 - from: inline
163 name: ssh-client
164 path: inline/ssh-client.yaml
165 repository:
166 metadata:
167 description: client starts
168 format: Lava-Test Test Definition 1.0
169 name: client-ssh
170 os:
171 - debian
172 scope:
173 - functional
174 run:
175 steps:
176 - lava-sync clients
177 - tar -tzf /$(pwd|awk -F'-' '{print $2}'|awk -F'/' '{print $1}')-overlay*.tar.gz
178 - repository: http://git.linaro.org/lava-team/lava-functional-tests.git
179 from: git
180 path: lava-test-shell/smoke-tests-basic.yaml
181 name: post-smoke-tests
182 timeout:
183 minutes: 5
Running operations inside the guest VM¶
A guest VM started by running QEMU on the command line is not a LAVA environment (unless the test writer deliberately copies files into it from another job), so it will not run a lava test shell by default. Tasks can be executed within the VM from any of the other jobs running on the host device, dependent on support provided by the test writer.
Remember, although LAVA tries to stay out of the way of how the test runs once the secondary connection has logged in, there are some things test writers need to consider to be able to automate tests like these.
If you start QEMU with the
-nographics
option rather than as a daemon, the secondary connection gets connected to the console of that VM at the point within the test shell where the call to QEMU is made.Make sure you know if the image being used has a serial console configured.
If the image being launched stops at a
login:
prompt, the test definition will need to handle that prompt or log in to the VM in some other way. e.g. by having one of the other secondary connections set up a configuration to usessh
to log in to the VM - the keys needed for this login will need to be handled by the test writer.The test shell will pause, waiting for QEMU to return, unless QEMU is configured to do otherwise or a wrapper like
pexpect
is used. (The LAVA QEMU devices run a QEMU command usingpexpect.spawn
but this is not necessarily suitable for test jobs.)If the VM is started as a daemon, then the test shell will need to have a way of monitoring when the VM is ready and then connect to the VM, as appropriate.
Note
The lava-start API call only acts once - i.e. the host role starts,
then the other jobs wait until lava-start
is sent - at which point these
jobs will download any test shell definitions and try to connect to the IP
address declared. It is better to have a synchronization which the test
writer controls, after all the jobs have connected to the host device.