This avoids the surprising behaviour where `--target jobname` works for
some jobs but not others, because gitlab adds `X/N` at the end of these
job names.
If the user does specify something like `jobname 1/.*` to only run the
first, the extra `\d+/\d+` is ignored, just like if the job isn't
`parallel:` and therefore doesn't end with `X/N`.
If the user really wants to fail to match parallel jobs (previous
behaviour), they can simply add a `$` at the end of the job name/regex
(but also, I don't see why someone would want that behaviour).
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27530>
This validation code has 2 bugs, the main one being that it is wrong and
is refusing perfectly valid codes. Let's remove this until we come up
with a valid check.
This reverts commit cd8b546205.
Fixes: cd8b546205 ("bin/ci: Add GitLab basic token validation")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27312>
Fix an issue in `ci_run_n_monitor.py` where the token was not
being correctly propagated to the GitlabGQL abstraction. This addresses
misbehavior in scenarios like running pipelines in a private fork,
ensuring proper functionality.
Also document `find_dependencies` function.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27206>
When generating the list of targets from a script, being able to just
pass `--target "${list[@]}"` is very convenient.
The list of targets is simply converted to an "or" regex, matching any
of the `--target`s given.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27252>
Use the CustomLogger class and CLI tool to create strutured logs
for poe scripts which are used by broadcom and nouveau jobs.
Renamed stage lint to code-validation and added python-test job
which runs the tests for structured and customer logger to ci.
Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25179>
This commit introduces the CustomLogger class, which provides methods
for updating, creating, and managing dut jobs and phases in a structured
log in below json format.
{
"_timestamp": "2023-10-05T06:16:42.603921",
"dut_job_type": "rpi3",
"farm": "igalia",
"dut_jobs": [
{
"status": "pass",
"submitter_start_time": "2023-10-05T06:16:42.745862",
"dut_start_time": "2023-10-05T06:16:42.819964",
"dut_submit_time": "2023-10-05T06:16:45.866096",
"dut_end_time": "2023-10-05T06:24:13.533394",
"dut_name": "igalia-ci01-rpi3-1gb",
"dut_state": "finished",
"dut_job_phases": [
{
"name": "boot",
"start_time": "2023-10-05T06:16:45.865863",
"end_time": "2023-10-05T06:17:14.801002"
},
{
"name": "test",
"start_time": "2023-10-05T06:17:14.801009",
"end_time": "2023-10-05T06:24:13.610296"
}
],
"submitter_end_time": "2023-10-05T06:24:13.680729"
}
],
"job_combined_status": "pass",
"dut_attempt_counter": 1
}
This class uses the existing StructuredLogger module,
which provides a robust and flexible logging utility supporting multiple
formats. It also includes a command-line tool for updating, creating,
and modifying dut jobs and phases through command-line arguments.
This can be used in ci job scripts to update information in structured logs.
Unit tests also have been added for the new class.
Currently, only LAVA jobs create structured log files, and this will be
extended for other jobs using this tool.
Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25179>
Reading the local root config file and then asking gitlab to evaluate it
in the context of some other version will cause issues if they are not
identical.
Instead, the local document should be a simple include of whatever is
the root config file at that commit.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26074>
We already print all the detected target jobs from regex and its
dependencies. But for more complex regexes the list can be cumbersome,
and an aggregate list of dependencies and targets can be more value, so
add these prints as well.
This is what looks like:
```
Running 10 dependency jobs:
alpine/x86_64_lava_ssh_client, clang-format, debian-arm64,
debian-testing, debian/arm64_build, debian/x86_64_build,
debian/x86_64_build-base, kernel+rootfs_arm64, kernel+rootfs_x86_64,
rustfmt
Running 15 target jobs:
a618_gl 1/4, a660_gl 1/2, intel-tgl-skqp, iris-amly-egl, iris-apl-deqp
1/3, iris-cml-deqp 1/4, iris-glk-deqp 1/2, iris-kbl-deqp 1/3,
lima-mali450-deqp:arm64, lima-mali450-piglit:arm64 1/2,
panfrost-g52-gl:arm64 1/3, panfrost-g72-gl:arm64 1/3,
panfrost-t720-gles2:arm64, panfrost-t860-egl:arm64, zink-anv-tgl
```
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25940>
Modify the GraphQL query used to fetch all jobs within a pipeline,
transitioning from fetching data via stage nodes to a direct job
retrieval approach.
The prior method was not paginated, potentially overloading the server
and complicating result parsing due to the structure of stage nodes. The
new approach simplifies data interpretation and handles job lists
exceeding 100 elements by implementing pagination with helper functions
to concatenate paginated results.
- Transitioned from extracting jobs from stage nodes to a direct query
for all jobs in the pipeline, improving data readability and server
performance.
- With the enhanced data clarity from the updated query, removed the
Dag+JobMetadata tuple as it's now redundant. The refined query
provides a more comprehensive job data including job name, stage, and
dependencies.
- The previous graph query relied on a graph node that will (or should)
be paginated anyway.
Closes: #10050
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25940>
Make query support pagination by supplying the paginated key.
In the following toy example, the paginated key is:
["levels", "cars"]
```graphql
query vehicle_store($location: ID!) {
levels {
cars {
pageInfo {
hasNextPage
endCursor
}
...
}
}
}
```
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25940>
For some reason AIOHTTPTransport started to use MultiDict after doing
some adjustments in the GraphQL query, which made `filecache` fail
because MultiDict object are not pickle-able.
Changing the transport strategy from AIOHTTPTransport to
RequestsHTTPTransport, which dropped one requirement. We aren't doing
async anyway, all the calls were sync before.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25940>
