ci: Add scripts for controlling bare-metal chezas.
This will let us: - deploy kernels for testing code depending on new kernel featuers - Ensure a pristine state in the HW before starting our tests - Avoid disk rot on the chezas taking them out (we'd lost 3/9 in a few months). Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5247>
This commit is contained in:
Eric Anholt
@@ -3,40 +3,76 @@
|
||||
Testing Mesa with gitlab-runner running on the devices being tested
|
||||
(DUTs) proved to be too unstable, so this set of scripts is for
|
||||
running Mesa testing on bare-metal boards connected to a separate
|
||||
system using gitlab-runner. Currently only "fastboot" devices are
|
||||
supported.
|
||||
system using gitlab-runner. Currently only "fastboot" and "ChromeOS
|
||||
Servo" devices are supported.
|
||||
|
||||
In comparison with LAVA, this doesn't involve maintaining a separate
|
||||
webservice with its own job scheduler and replicating jobs between the
|
||||
two. It also places more of the board support in git, instead of
|
||||
webservice configuration. Most importantly, it doesn't download the
|
||||
rootfs as artifacts on each job, so we can avoid traffic to
|
||||
freedesktop.org. On the other hand, the serial interactions and
|
||||
bootloader support are more primitive.
|
||||
webservice configuration. On the other hand, the serial interactions
|
||||
and bootloader support are more primitive.
|
||||
|
||||
## Requirements
|
||||
## Requirements (fastboot)
|
||||
|
||||
This testing requires power control of the DUTs by the gitlab-runner
|
||||
machine, since this is what we use to reset the system and get back to
|
||||
a pristine state at the start of testing.
|
||||
|
||||
We require access to the console output from the gitlb-runner system,
|
||||
since that is how we get the final results back from te tests. You
|
||||
We require access to the console output from the gitlab-runner system,
|
||||
since that is how we get the final results back from the tests. You
|
||||
should probably have the console on a serial connection, so that you
|
||||
can see bootloader progress.
|
||||
|
||||
The boards need to be able to have a kernel/initramfs supplied by the
|
||||
gitlab-runner system, since the initramfs is what contains the Mesa
|
||||
testing payload. Currently only "fastboot" devices are supported.
|
||||
testing payload.
|
||||
|
||||
The boards should have networking, so that (in a future iteration of
|
||||
this code) we can extract the dEQP .xml results to artifacts on
|
||||
gitlab.
|
||||
|
||||
## Requirements (servo)
|
||||
|
||||
For servo-connected boards, we can use the EC connection for power
|
||||
control to reboot the board. However, loading a kernel is not as easy
|
||||
as fastboot, so we assume your bootloader can do TFTP, and that your
|
||||
gitlab-runner mounts the runner's tftp directory specific to the board
|
||||
at /tftp in the container.
|
||||
|
||||
Since we're going the TFTP route, we also use NFS root. This avoids
|
||||
packing the rootfs and sending it to the board as a ramdisk, which
|
||||
means we can support larger rootfses (for piglit or tracie testing),
|
||||
at the cost of needing more storage on the runner.
|
||||
|
||||
Telling the board about where its TFTP and NFS should come from is
|
||||
done using dnsmasq on the runner host. For example, this snippet in
|
||||
the dnsmasq.conf.d in the google farm, with the gitlab-runner host we
|
||||
call "servo".
|
||||
|
||||
```
|
||||
dhcp-host=1c:69:7a:0d:a3:d3,10.42.0.10,set:servo
|
||||
|
||||
# Fixed dhcp addresses for my sanity, and setting a tag for
|
||||
# specializing other DHCP options
|
||||
dhcp-host=a0:ce:c8:c8:d9:5d,10.42.0.11,set:cheza1
|
||||
dhcp-host=a0:ce:c8:c8:d8:81,10.42.0.12,set:cheza2
|
||||
|
||||
# Specify the next server, watch out for the double ',,'. The
|
||||
# filename didn't seem to get picked up by the bootloader, so we use
|
||||
# tftp-unique-root and mount directories like
|
||||
# /srv/tftp/10.42.0.11/jwerner/cheza as /tftp in the job containers.
|
||||
tftp-unique-root
|
||||
dhcp-boot=tag:cheza1,cheza1/vmlinuz,,10.42.0.10
|
||||
dhcp-boot=tag:cheza2,cheza2/vmlinuz,,10.42.0.10
|
||||
|
||||
dhcp-option=tag:cheza1,option:root-path,/srv/nfs/cheza1
|
||||
dhcp-option=tag:cheza2,option:root-path,/srv/nfs/cheza2
|
||||
```
|
||||
|
||||
## Setup
|
||||
|
||||
Each board will be registered in fd.o gitlab. You'll want something
|
||||
like this to register:
|
||||
like this to register a fastboot board:
|
||||
|
||||
```
|
||||
sudo gitlab-runner register \
|
||||
@@ -52,27 +88,35 @@ sudo gitlab-runner register \
|
||||
--non-interactive
|
||||
```
|
||||
|
||||
For a servo board, you'll need to also volume mount the board's NFS
|
||||
root dir at /nfs and TFTP kernel directory at /tftp.
|
||||
|
||||
The registration token has to come from a fd.o gitlab admin going to
|
||||
https://gitlab.freedesktop.org/admin/runners
|
||||
|
||||
The name scheme for Google's lab is google-freedreno-boardname-nn, and
|
||||
our tag is google-freedreno-db410c. The tag is what identifies a
|
||||
board type so that board-specific jobs can be dispatched into that
|
||||
pool.
|
||||
The name scheme for Google's lab is google-freedreno-boardname-n, and
|
||||
our tag is something like google-freedreno-db410c. The tag is what
|
||||
identifies a board type so that board-specific jobs can be dispatched
|
||||
into that pool.
|
||||
|
||||
We need privileged mode and the /dev bind mount in order to get at the
|
||||
serial console and fastboot USB devices (--device arguments don't
|
||||
apply to devices that show up after container start, which is the case
|
||||
with fastboot). We use host network mode so that we can (in the
|
||||
future) spin up a server to collect XML results.
|
||||
with fastboot, and the servo serial devices are acctually links to
|
||||
/dev/pts). We use host network mode so that we can (in the future)
|
||||
spin up a server to collect XML results for fastboot.
|
||||
|
||||
Once you've added your boards, you're going to need to specify the
|
||||
board-specific env vars, adding something like this `environment` line
|
||||
to each runner in `/etc/gitlab-runner/config.toml`
|
||||
Once you've added your boards, you're going to need to add a little
|
||||
more customization in `/etc/gitlab-runner/config.toml`. First, add
|
||||
`concurrent = <number of boards>` at the top ("we should have up to
|
||||
this many jobs running managed by this gitlab-runner"). Then for each
|
||||
board's runner, set `limit = 1` ("only 1 job served by this board at a
|
||||
time"). Finally, add the board-specific environment variables
|
||||
required by your bare-metal script, something like:
|
||||
|
||||
```
|
||||
[[runners]]
|
||||
name = "google-freedreno-db410c-01"
|
||||
name = "google-freedreno-db410c-1"
|
||||
environment = ["BM_SERIAL=/dev/ttyDB410c8", "BM_POWERUP=google-power-up.sh 8", "BM_FASTBOOT_SERIAL=15e9e390"]
|
||||
```
|
||||
|
||||
|
||||
97
.gitlab-ci/bare-metal/cros-servo.sh
Executable file
97
.gitlab-ci/bare-metal/cros-servo.sh
Executable file
@@ -0,0 +1,97 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Boot script for Chrome OS devices attached to a servo debug connector, using
|
||||
# NFS and TFTP to boot.
|
||||
|
||||
# We're run from the root of the repo, make a helper var for our paths
|
||||
BM=$CI_PROJECT_DIR/.gitlab-ci/bare-metal
|
||||
|
||||
# Runner config checks
|
||||
if [ -z "$BM_SERIAL" ]; then
|
||||
echo "Must set BM_SERIAL in your gitlab-runner config.toml [[runners]] environment"
|
||||
echo "This is the CPU serial device."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ -z "$BM_SERIAL_EC" ]; then
|
||||
echo "Must set BM_SERIAL in your gitlab-runner config.toml [[runners]] environment"
|
||||
echo "This is the EC serial device for controlling board power"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -d /nfs ]; then
|
||||
echo "NFS rootfs directory needs to be mounted at /nfs by the gitlab runner"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -d /tftp ]; then
|
||||
echo "TFTP directory for this board needs to be mounted at /tftp by the gitlab runner"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# job config checks
|
||||
if [ -z "$BM_KERNEL" ]; then
|
||||
echo "Must set BM_KERNEL to your board's kernel FIT image"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ -z "$BM_ROOTFS" ]; then
|
||||
echo "Must set BM_ROOTFS to your board's rootfs directory in the job's variables"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ -z "$BM_CMDLINE" ]; then
|
||||
echo "Must set BM_CMDLINE to your board's kernel command line arguments"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
set -ex
|
||||
|
||||
# Create the rootfs in the NFS directory. rm to make sure it's in a pristine
|
||||
# state, since it's volume-mounted on the host.
|
||||
rm -rf /nfs/*
|
||||
mkdir -p /nfs/results
|
||||
. $BM/rootfs-setup.sh /nfs
|
||||
|
||||
# Set up the TFTP kernel/cmdline. When we support more than one board with
|
||||
# this method, we'll need to do some check on the runner name or something.
|
||||
rm -rf /tftp/*
|
||||
cp $BM_KERNEL /tftp/vmlinuz
|
||||
echo "$BM_CMDLINE" > /tftp/cmdline
|
||||
|
||||
# Start watching serials, and power up the device.
|
||||
$BM/serial-buffer.py $BM_SERIAL_EC | tee serial-ec-output.txt | sed -u 's|^|SERIAL-EC> |g' &
|
||||
$BM/serial-buffer.py $BM_SERIAL | tee serial-output.txt | sed -u 's|^|SERIAL-CPU> |g' &
|
||||
while [ ! -e serial-output.txt ]; do
|
||||
sleep 1
|
||||
done
|
||||
# Flush any partial commands in the EC's prompt, then ask for a reboot.
|
||||
$BM/write-serial.py $BM_SERIAL_EC ""
|
||||
$BM/write-serial.py $BM_SERIAL_EC reboot
|
||||
|
||||
# This is emitted right when the bootloader pauses to check for input. Emit a
|
||||
# ^N character to request network boot, because we don't have a
|
||||
# direct-to-netboot firmware on cheza.
|
||||
$BM/expect-output.sh serial-output.txt "load_archive: loading locale_en.bin"
|
||||
$BM/write-serial.py $BM_SERIAL `printf '\016'`
|
||||
|
||||
# Wait for the device to complete the deqp run
|
||||
$BM/expect-output.sh serial-output.txt "DEQP RESULT"
|
||||
|
||||
# power down the CPU on the device
|
||||
$BM/write-serial.py $BM_SERIAL_EC 'power off'
|
||||
|
||||
set -ex
|
||||
|
||||
# Bring artifacts back from the NFS dir to the build dir where gitlab-runner
|
||||
# will look for them. Note that results/ may already exist, so be careful
|
||||
# with cp.
|
||||
mkdir -p results
|
||||
cp -Rp /nfs/results/. results/
|
||||
|
||||
set +e
|
||||
if grep -q "DEQP RESULT: pass" serial-output.txt; then
|
||||
exit 0
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
@@ -44,57 +44,9 @@ fi
|
||||
|
||||
set -ex
|
||||
|
||||
# Copy the rootfs to a temporary for our setup, as I believe changes to the
|
||||
# container can end up impacting future runs.
|
||||
cp -Rp $BM_ROOTFS rootfs
|
||||
|
||||
# Set up the init script that brings up the system.
|
||||
cp $BM/init.sh rootfs/init
|
||||
|
||||
set +x
|
||||
# Pass through relevant env vars from the gitlab job to the baremetal init script
|
||||
touch rootfs/set-job-env-vars.sh
|
||||
chmod +x rootfs/set-job-env-vars.sh
|
||||
for var in \
|
||||
CI_COMMIT_BRANCH \
|
||||
CI_COMMIT_TITLE \
|
||||
CI_JOB_ID \
|
||||
CI_JOB_URL \
|
||||
CI_MERGE_REQUEST_SOURCE_BRANCH_NAME \
|
||||
CI_MERGE_REQUEST_TITLE \
|
||||
CI_NODE_INDEX \
|
||||
CI_NODE_TOTAL \
|
||||
CI_PIPELINE_ID \
|
||||
CI_RUNNER_DESCRIPTION \
|
||||
DEQP_CASELIST_FILTER \
|
||||
DEQP_EXPECTED_RENDERER \
|
||||
DEQP_NO_SAVE_RESULTS \
|
||||
DEQP_PARALLEL \
|
||||
DEQP_RUN_SUFFIX \
|
||||
DEQP_VER \
|
||||
FD_MESA_DEBUG \
|
||||
FLAKES_CHANNEL \
|
||||
IR3_SHADER_DEBUG \
|
||||
NIR_VALIDATE \
|
||||
; do
|
||||
val=`echo ${!var} | sed 's|"||g'`
|
||||
echo "export $var=\"${val}\"" >> rootfs/set-job-env-vars.sh
|
||||
done
|
||||
echo "Variables passed through:"
|
||||
cat rootfs/set-job-env-vars.sh
|
||||
set -x
|
||||
|
||||
# Add the Mesa drivers we built, and make a consistent symlink to them.
|
||||
mkdir -p rootfs/$CI_PROJECT_DIR
|
||||
tar -C rootfs/$CI_PROJECT_DIR/ -xf $CI_PROJECT_DIR/artifacts/install.tar
|
||||
ln -sf $CI_PROJECT_DIR/install rootfs/install
|
||||
|
||||
# Copy the deqp runner script and metadata.
|
||||
cp .gitlab-ci/deqp-runner.sh rootfs/deqp/.
|
||||
cp .gitlab-ci/$DEQP_SKIPS rootfs/$CI_PROJECT_DIR/install/deqp-skips.txt
|
||||
if [ -n "$DEQP_EXPECTED_FAILS" ]; then
|
||||
cp .gitlab-ci/$DEQP_EXPECTED_FAILS rootfs/$CI_PROJECT_DIR/install/deqp-expected-fails.txt
|
||||
fi
|
||||
# Create the rootfs in a temp dir
|
||||
mkdir rootfs
|
||||
. .gitlab-ci/bare-metal/rootfs-setup.sh rootfs
|
||||
|
||||
# Finally, pack it up into a cpio rootfs.
|
||||
pushd rootfs
|
||||
|
||||
55
.gitlab-ci/bare-metal/rootfs-setup.sh
Normal file
55
.gitlab-ci/bare-metal/rootfs-setup.sh
Normal file
@@ -0,0 +1,55 @@
|
||||
#!/bin/bash
|
||||
|
||||
rootfs_dst=$1
|
||||
|
||||
# Copy the rootfs to a temporary for our setup, as I believe changes to the
|
||||
# container can end up impacting future runs.
|
||||
cp -Rp $BM_ROOTFS/. $rootfs_dst
|
||||
|
||||
# Set up the init script that brings up the system.
|
||||
cp $BM/init.sh $rootfs_dst/init
|
||||
|
||||
set +x
|
||||
# Pass through relevant env vars from the gitlab job to the baremetal init script
|
||||
touch $rootfs_dst/set-job-env-vars.sh
|
||||
chmod +x $rootfs_dst/set-job-env-vars.sh
|
||||
for var in \
|
||||
CI_COMMIT_BRANCH \
|
||||
CI_COMMIT_TITLE \
|
||||
CI_JOB_ID \
|
||||
CI_JOB_URL \
|
||||
CI_MERGE_REQUEST_SOURCE_BRANCH_NAME \
|
||||
CI_MERGE_REQUEST_TITLE \
|
||||
CI_NODE_INDEX \
|
||||
CI_NODE_TOTAL \
|
||||
CI_PIPELINE_ID \
|
||||
CI_RUNNER_DESCRIPTION \
|
||||
DEQP_CASELIST_FILTER \
|
||||
DEQP_EXPECTED_RENDERER \
|
||||
DEQP_NO_SAVE_RESULTS \
|
||||
DEQP_PARALLEL \
|
||||
DEQP_RUN_SUFFIX \
|
||||
DEQP_VER \
|
||||
FD_MESA_DEBUG \
|
||||
FLAKES_CHANNEL \
|
||||
IR3_SHADER_DEBUG \
|
||||
NIR_VALIDATE \
|
||||
; do
|
||||
val=`echo ${!var} | sed 's|"||g'`
|
||||
echo "export $var=\"${val}\"" >> $rootfs_dst/set-job-env-vars.sh
|
||||
done
|
||||
echo "Variables passed through:"
|
||||
cat $rootfs_dst/set-job-env-vars.sh
|
||||
set -x
|
||||
|
||||
# Add the Mesa drivers we built, and make a consistent symlink to them.
|
||||
mkdir -p $rootfs_dst/$CI_PROJECT_DIR
|
||||
tar -C $rootfs_dst/$CI_PROJECT_DIR/ -xf $CI_PROJECT_DIR/artifacts/install.tar
|
||||
ln -sf $CI_PROJECT_DIR/install $rootfs_dst/install
|
||||
|
||||
# Copy the deqp runner script and metadata.
|
||||
cp .gitlab-ci/deqp-runner.sh $rootfs_dst/deqp/
|
||||
cp .gitlab-ci/$DEQP_SKIPS $rootfs_dst/$CI_PROJECT_DIR/install/deqp-skips.txt
|
||||
if [ -n "$DEQP_EXPECTED_FAILS" ]; then
|
||||
cp .gitlab-ci/$DEQP_EXPECTED_FAILS $rootfs_dst/$CI_PROJECT_DIR/install/deqp-expected-fails.txt
|
||||
fi
|
||||
11
.gitlab-ci/bare-metal/write-serial.py
Executable file
11
.gitlab-ci/bare-metal/write-serial.py
Executable file
@@ -0,0 +1,11 @@
|
||||
#!/usr/bin/python3
|
||||
|
||||
import sys
|
||||
import serial
|
||||
|
||||
dev = sys.argv[1]
|
||||
command = sys.argv[2] + '\n'
|
||||
|
||||
ser = serial.Serial(dev, 115200, timeout=5)
|
||||
ser.write(command.encode())
|
||||
ser.close()
|
||||
Reference in New Issue
Block a user