We should be explicit that we are cancelling jobs once the script finds
some log messages that are linked with known issues. That means the
script preemptively retried the job without giving chances to recover.
Adds magenta color to cancelled jobs.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17389>
Implement a log-based retry hint for R8152 issue described in #6681,
which is based on detecting these two consecutive lines:
```
r8152 <USB> eth0: Tx status -71
nfs: server <IP> not responding, still trying
```
Where <IP> and <USB> could be any IP and USB addresses, respectfully.
This commit is a temporary fix since it requires a section-aware log
follower, implemented in !16323. When the cited MR is merged, one will
make a proper fix on top of that.
Closes: #6681
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17389>
In some jobs, such as
https://gitlab.freedesktop.org/gallo/mesa/-/jobs/24904100, the kmsg is
interleaved with stderr/stdout in serial console, making it difficult to
confidently find the log messages to detect when the DUT is booting,
when the DUT is running etc.
Luckily, LAVA sends redundant messages about their signals. We can use
them to mitigate the chance of missing an interleaved message by being
more open to different messages, using the regex on both `debug` and
`target` LAVA log levels.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
There are some leftovers in the jobs logs after the result log line.
Only print until the init-stage2.sh output, to raise the chance to check
for the test script results at the first glance in the Gitlab logs.
Extra changes:
- Add `hung` status for jobs considered hanging in the Gitlab
- print them after the retry loop
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
test_full_yaml_log is a test that will look for a LAVA log YAML file at
`/tmp/log.yaml` and consume it as it was a realtime CI job.
It is useful for debugging issues related with LAVA.
Let's keep it skipped by default, to avoid introducing entire logs into
the codebase.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
Currently, the submitter consider that every new log that comes from the
DUT console is a signal that the device is healthy, but maybe that is
not the case, since in some kernel hangs/failures, no output is
presented except from some kernel messages.
This commit bypass the heartbeat when the LogFollower detect a kernel
message. Any log line that does follow the kmsg pattern will make the
job labeled as healthy again.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
- Create LogFollower to capture LAVA log and process it adding some
- GitlabSection and color treatment to it
- Break logs further, make new gitlab sections between testcases
- Implement LogFollower as ContextManager to deal with incomplete LAVA
jobs.
- Use template method to simplify gitlab log sections management
- Fix sections timestamps
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
New tests, dEQP line rasterization test fix that lets Intel pass.
Clears out bogus xfails from 1.3.2.0 uprev on a630, which I suspect were
"we lost the device twice on a full run once, and those fails got pasted
in without checking if it happened a full run again" (since we haven't
seen them in other full run attempts).
Also clears out the a630 vk asan xfails (essentially all tests run) by
turning off leak detection which was just catching leaks in vkcts.
Acked-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17304>
The LLVM backend is not officially supported by the RADV developers,
but it has been useful early during bring-up, or later when users are
experiencing what looks like a compiler bug. It is thus beneficial to
keep it working.
However, maintaining the vkcts expectations for every platform requires
more work and machine time than what we would like to commit to. This
is why we agreed that we would only keep LLVM tested on the latest
family of Radeon GPUs.
Signed-off-by: Martin Roukala (né Peres) <martin.roukala@mupuf.org>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Acked-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16841>
Most CI tests are currently running on LLVM 11 (released over 2 years
ago), which predates some of the GPUs we have in CI and prevents
testing RADV's LLVM backend.
LLVM 13 is known to work for RADV, released almost 8 months ago, and
is already available in most distributions. Fedora 36 is even already
on LLVM 14.
So this commit updates x86 testing on llvm 13.
v2:
- store the llvm apt repo key locally (Michel Dänzer)
Signed-off-by: Martin Roukala (né Peres) <martin.roukala@mupuf.org>
Acked-by: Timur Kristóf <timur.kristof@gmail.com>
Acked-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17188>
This drops the mesa/gallium lists from some build rules, since zink common
rules brings them in already. If we do more driver common rules, we might
end up with those core lists appearing in the yaml multiple times, but
that seems like a small price to pay for not being able to forget some.
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Acked-by: Eric Engestrom <eric@engestrom.ch>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17287>
LAVA job logs have an ongoing problem of message interleaving with kmsg.
So any kernel dumps and LAVA signals (which are being printed in kmsg)
will have a chance to clutter the pattern matching for `hwci: mesa:
(pass|fail)` line.
v2:
- Add an 1 second sleep before exiting the test script, to give enough
time to print the result message without conflicting with LAVA ENDTC
signal from kmsg
Closes: #6714
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17175>
Additionally improve the style of the build script.
- Instead of using platform build tools, use the CMake wrappers
- Instead of build all targets, then manually stripping, use the
CMake helpers.
Contributed by Andres Gomez.
Reviewed-by: Martin Roukala <martin.roukala@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17184>
