Commit Graph

63 Commits

Author SHA1 Message Date
Eric Engestrom
032d4a20f9 bin/ci_run_n_monitor: explain that the 'Universal Recycling symbol' ♲ emoji means these jobs were cancelled
And not "restarted" for instance, which would also be a reasonable
interpretation.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30223>
2024-07-18 19:21:45 +00:00
Sergi Blanch Torne
e6de8e2533 ci: fix run_n_monitor single execution
When there is a single job as a target and, when it is triggered, it takes
more than one loop time in pending status, it confuses the script to fall in
a wrong return of the pipeline monitoring.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/11517

Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30219>
2024-07-17 11:25:50 +00:00
Sergi Blanch Torne
5873b3ac14 ci: run_n_monitor, sort by name when listing jobs
The job loops, w/o and explicit sort, are sorted by the job id. As they produce
logs, humans could feel an improvement by sorting by name.

Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29444>
2024-07-11 08:28:31 +00:00
Sergi Blanch Torne
12c1bdc31d ci: run_n_monitor, listing job names with a padding
Formatting the output lines with a padding in the job name fields may help
the readability of the information when one tries to follow the progress of
a specific job.

Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29444>
2024-07-11 08:28:31 +00:00
Sergi Blanch Torne
45f19b3631 ci: run_n_monitor, arguments review and unicode
Coding style homogenous for all the methods in the file. Document
unicode symbols used.

Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29444>
2024-07-11 08:28:31 +00:00
Eric Engestrom
0b09cf63a8 ci_run_n_monitor: add RUNNING_STATUSES and use it where appropriate
A couple of these were trivially missing the `created` status which
usually doesn't stay for long enough for this very slow script to notice.

The `cancel_job()` function will no longer cancel manual jobs waiting to
be started.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29917>
2024-07-04 09:14:36 +00:00
Eric Engestrom
a578101d5b ci_run_n_monitor: use COMPLETED_STATUSES in more places
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29917>
2024-07-04 09:14:36 +00:00
Eric Engestrom
fb2adbeeb7 ci_run_n_monitor: be coherent about using sets for element in group checks
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29917>
2024-07-04 09:14:35 +00:00
Eric Engestrom
fa3d529f55 ci_run_n_monitor: add support for new canceling job status
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29917>
2024-07-04 09:14:35 +00:00
Sergi Blanch Torne
0c092dc5c4 ci: run_n_monitor, collect and summarize
The job duration is printed during the test progress. It can be collected and
summarized at the end of the process. It is interesting when doing a stress
test as one will have this information together at the end.

Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29419>
2024-06-14 08:04:56 +00:00
Sergi Blanch Torne
2d6e72c2cb ci: continue stress run'n'monitor
When the tool is used to stress test a pipeline, if there are jobs already ran,
use their information like it does when the stress flag is not set.

This provides consistency between the behavior when stress argument is not set,
to when it is set. When it is not set, it uses the information about jobs that
are already done. When it is set, it has to use the information about the
already ran jobs. Also, it saves resources by triggering the minimum required.

Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29432>
2024-06-11 11:22:29 +02:00
Sergi Blanch Torne
d963fd596e ci: fix stress counter in run'n'monitor
The stress counter after enable_job(retry), often stores the new status
instead of the job complete. So, the summary printed later doesn't show the
real evolution of the test.

Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29416>
2024-05-27 20:30:36 +00:00
Eric Engestrom
189b4193ee ci_run_n_monitor: explain how to pass multiple targets without having to use regexes
Fixes: 6825c67c99 ("ci_run_n_monitor: allow passing multiple targets")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28461>
2024-03-29 19:23:24 +00:00
Eric Engestrom
bc38b68ab8 ci_run_n_monitor: read job logs as utf-8
Fixes regular crashes like this one:

    Traceback (most recent call last):
      File "bin/ci/ci_run_n_monitor.py", line 478, in <module>
        print_log(cur_project, target_job_id)
      File "bin/ci/ci_run_n_monitor.py", line 270, in print_log
        lines = job.trace().decode("raw_unicode_escape").splitlines()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    UnicodeDecodeError: 'rawunicodeescape' codec can't decode bytes in position 6090-6091: truncated \uXXXX escape
    decoding with 'raw_unicode_escape' codec failed

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27653>
2024-02-29 14:17:02 +00:00
Eric Engestrom
3acb00290d ci_run_n_monitor: allow detached heads as well
When running on a detached head (eg. checkout of a tag or a specific
commit), there is no active branch, so we can't perform this check; just
skip it and assume the user knows what they're doing.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27578>
2024-02-16 11:39:50 +00:00
Eric Engestrom
ef744fa589 ci_run_n_monitor: explain why/when there might be no tracked remote
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27578>
2024-02-16 11:39:50 +00:00
Corentin Noël
b0af2b30ab ci_run_n_monitor: Allow the upstream format to not exist
When pushing directly to a fork, the upstream branch does not exist and thus
returns an empty string.

Signed-off-by: Corentin Noël <corentin.noel@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27567>
2024-02-12 10:52:53 +01:00
Eric Engestrom
9beb9081fc ci_run_n_monitor: print the target regex before adding the X/N bit
To avoid showing something too complex to the user, especially since this isn't something they added themselves or can remove.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27530>
2024-02-09 15:58:10 +00:00
Eric Engestrom
7154c1eb77 ci_run_n_monitor: implicitly include parallel: jobs
This avoids the surprising behaviour where `--target jobname` works for
some jobs but not others, because gitlab adds `X/N` at the end of these
job names.

If the user does specify something like `jobname 1/.*` to only run the
first, the extra `\d+/\d+` is ignored, just like if the job isn't
`parallel:` and therefore doesn't end with `X/N`.

If the user really wants to fail to match parallel jobs (previous
behaviour), they can simply add a `$` at the end of the job name/regex
(but also, I don't see why someone would want that behaviour).

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27530>
2024-02-09 15:58:10 +00:00
Eric Engestrom
45df652d0b ci_run_n_monitor: refresh job state when starting it
Avoids job staying in 'manual' state and making the script think there's
nothing left running.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27499>
2024-02-08 22:22:54 +00:00
Eric Engestrom
8a05cb8da7 ci_run_n_monitor: track new job when retrying a job
When retrying a job, we are creating a new job, so we need to keep track
of this new job now.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27499>
2024-02-08 22:22:54 +00:00
Eric Engestrom
5bc1a62fbb ci_run_n_monitor: add method to get a pipeline job by its id
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27499>
2024-02-08 22:22:54 +00:00
Eric Engestrom
bce1230587 ci_run_n_monitor: update job when it goes through enable_job()
`enable_job()` modifies the job, so we need to make sure we get the
updated job back out of it.

The next two commits take care of the two specific code paths.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27499>
2024-02-08 22:22:54 +00:00
Eric Engestrom
5758a5d660 ci_run_n_monitor: add some types for gitlab objects
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27499>
2024-02-08 22:22:54 +00:00
Eric Engestrom
044c51b6bf ci_run_n_monitor: warn user if they forgot to push the branch
Only perform this check if they set `--rev HEAD` (or don't specify it);
let's assume if they select another commit, they know what they're
doing.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26969>
2024-02-08 21:38:21 +00:00
Eric Engestrom
5d293f01cc ci_run_n_monitor: avoid spamming a ton of "new status: created" for all the jobs at the beginning
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27419>
2024-02-02 15:25:22 +00:00
Guilherme Gallo
8eabdb3909 bin/ci: Refactor read_token function
Make `read_token` utilize the `get_gitlab_pipeline_from_url` to reuse
code from `gitlab_gql.py`.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27206>
2024-01-26 00:37:05 -03:00
Guilherme Gallo
50fcea9c34 bin/ci: Propagate the token to GitlabGQL
Fix an issue in `ci_run_n_monitor.py` where the token was not
being correctly propagated to the GitlabGQL abstraction. This addresses
misbehavior in scenarios like running pipelines in a private fork,
ensuring proper functionality.

Also document `find_dependencies` function.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27206>
2024-01-26 00:37:03 -03:00
Eric Engestrom
6825c67c99 ci_run_n_monitor: allow passing multiple targets
When generating the list of targets from a script, being able to just
pass `--target "${list[@]}"` is very convenient.

The list of targets is simply converted to an "or" regex, matching any
of the `--target`s given.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27252>
2024-01-25 21:16:13 +00:00
Eric Engestrom
eeba409c54 ci_run_n_monitor: drop always-true condition
`--target` is mandatory.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27252>
2024-01-25 21:16:13 +00:00
Helen Koike
9e8cbc8e91 ci/ci_run_n_monitor: move get_gitlab_pipeline_from_url() to gitlab_common
Move this code to gitlab_common since it can be re-used by other
scripts.

Signed-off-by: Helen Koike <helen.koike@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25793>
2024-01-17 13:10:07 +00:00
Eric Engestrom
76be39ca6d ci_run_n_monitor: allow picking a pipeline by its MR
It picks the latest pipeline from the MR; if you want another pipeline,
use `--pipeline-url`.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26381>
2023-12-03 11:28:49 +00:00
Helen Koike
2999091287 ci/ci_run_n_monitor: abort when target gets skipped
when a target receives skipped state it is because some of the
dependencies failed, abort the script and print the reason

Signed-off-by: Helen Koike <helen.koike@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26155>
2023-11-20 13:03:09 +00:00
Eric Engestrom
a5e20a5c31 ci_run_n_monitor: require user to add an explicit .* at the end if jobs like *-full are wanted
Most of the time, these jobs are not wanted, so let's make this a full
match instead of prefix match so that users only get what they ask for.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26066>
2023-11-17 08:12:31 +00:00
Guilherme Gallo
f8b8352022 ci/bin: Print a summary list of dependency and target jobs
We already print all the detected target jobs from regex and its
dependencies. But for more complex regexes the list can be cumbersome,
and an aggregate list of dependencies and targets can be more value, so
add these prints as well.

This is what looks like:

```
Running 10 dependency jobs:
 alpine/x86_64_lava_ssh_client, clang-format, debian-arm64,
 debian-testing, debian/arm64_build, debian/x86_64_build,
 debian/x86_64_build-base, kernel+rootfs_arm64, kernel+rootfs_x86_64,
 rustfmt

Running 15 target jobs:
 a618_gl 1/4, a660_gl 1/2, intel-tgl-skqp, iris-amly-egl, iris-apl-deqp
 1/3, iris-cml-deqp 1/4, iris-glk-deqp 1/2, iris-kbl-deqp 1/3,
 lima-mali450-deqp:arm64, lima-mali450-piglit:arm64 1/2,
 panfrost-g52-gl:arm64 1/3, panfrost-g72-gl:arm64 1/3,
 panfrost-t720-gles2:arm64, panfrost-t860-egl:arm64, zink-anv-tgl
```

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25940>
2023-11-08 02:18:17 +00:00
Guilherme Gallo
278fc1c22a ci/bin: gql: Improve queries for jobs/stages retrieval
Modify the GraphQL query used to fetch all jobs within a pipeline,
transitioning from fetching data via stage nodes to a direct job
retrieval approach.

The prior method was not paginated, potentially overloading the server
and complicating result parsing due to the structure of stage nodes. The
new approach simplifies data interpretation and handles job lists
exceeding 100 elements by implementing pagination with helper functions
to concatenate paginated results.

- Transitioned from extracting jobs from stage nodes to a direct query
  for all jobs in the pipeline, improving data readability and server
  performance.
- With the enhanced data clarity from the updated query, removed the
  Dag+JobMetadata tuple as it's now redundant. The refined query
  provides a more comprehensive job data including job name, stage, and
  dependencies.
- The previous graph query relied on a graph node that will (or should)
  be paginated anyway.

Closes: #10050
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25940>
2023-11-08 02:18:17 +00:00
Guilherme Gallo
46b033a9bf ci/bin: Fix find_dependency function calls
`find_dependency` signature changed, so we need to update their callings
correctly.

Fixes: 94655ff811

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25940>
2023-11-08 02:18:17 +00:00
Eric Engestrom
94655ff811 ci_run_n_monitor: compile target_jobs_regex only once
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26070>
2023-11-07 06:08:44 +00:00
Guilherme Gallo
c7b67d8619 ci/bin: Use iid instead of SHA in gitlab_gql
We were using sha to fetch the pipeline from GraphQL, but that leads to
wrong results when MR and branch pipelines exist. For example,
using pipeline-url as the MR pipeline:
- https://gitlab.freedesktop.org/gallo/mesa/-/pipelines/1017182
This would lead into the branch pipeline:
- https://gitlab.freedesktop.org/gallo/mesa/-/pipelines/1013189

Also simplify the GQL query, it had lots of unused data.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25858>
2023-10-27 00:36:48 +00:00
Eric Anholt
74315051a9 ci_run_n_monitor: Always resolve --rev arguments for looking up pipelines.
Otherwise you'd loop forever never finding it with --rev <paste from
gitlab UI> or --rev <branch name>

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25824>
2023-10-25 01:48:18 +00:00
Helen Koike
663ad191ca ci/ci_run_n_monitor: print job duration time
If the job is running, it prints the elapsed time so far, if it finished
already, it prints the time it took.

Signed-off-by: Helen Koike <helen.koike@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25822>
2023-10-24 12:57:14 +00:00
Helen Koike
9f4f52f4f3 ci/ci_run_n_monitor: keep monitoring if a job is still running
When running multiple targets, the script was returning if any of the
targets failed.
But sometimes some of the targets are still running and we want to
monitor it until it finishes, even if some of the targets failed.

So check if any targeted jobs are in pending or running before deciding
to finish.

Signed-off-by: Helen Koike <helen.koike@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25826>
2023-10-22 09:30:56 +00:00
Eric Anholt
fb95f1d55c ci_run_n_monitor: Poll mesa/mesa and user/mesa for pipelines at the same time.
Now you don't fail if you're trying to test a mesa/mesa MR pipeline and
gitlab takes more than 10s to create it.  And you don't have to wait 10
seconds to get things started (aka see if your regex was right) if you're
testing a user/mesa fork pipeline.

Fixes: 941d92408e ("bin/ci_run_n_monitor: automatically pick MR pipelines when they exist")
Closes: #9894
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25810>
2023-10-20 08:23:24 +00:00
Eric Engestrom
d3e96aa02e ci_run_n_monitor: dependency jobs must always be started
Fixes: 6b49b477ac ("ci/ci_run_n_monitor: simplify enable/cancel logic in monitor_pipeline()")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25715>
2023-10-13 18:37:56 +00:00
Helen Koike
51521e1f9d ci/ci_run_n_monitor: limit repetitions on --stress
--stress options now receives the number of repetitions or -1 to behave
as before.

Signed-off-by: Helen Koike <helen.koike@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25473>
2023-10-11 21:50:58 +00:00
Helen Koike
57fa35f19c ci/ci_run_n_monitor: allow <user>/<project> in --project
Allow to monitor pipelines in any <user>/<project> on gitlab.fdo.

If developers want to monitor MR pipelines, they can run with
--project mesa/mesa to force it to be picked from mesa username.

Signed-off-by: Helen Koike <helen.koike@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25473>
2023-10-11 21:50:58 +00:00
Helen Koike
6b49b477ac ci/ci_run_n_monitor: simplify enable/cancel logic in monitor_pipeline()
leave to enable_job() and cancel_job() to do the right thing according
to the job status.

let target canceled jobs to be re-runned.

Signed-off-by: Helen Koike <helen.koike@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25473>
2023-10-11 21:50:58 +00:00
Helen Koike
17843ad7c6 ci/ci_run_n_monitor: merge enable_job with retry_job
Signed-off-by: Helen Koike <helen.koike@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25473>
2023-10-11 21:50:58 +00:00
Helen Koike
90da4a6845 ci/ci_run_n_monitor: make --target mandatory
There is no point running without a target, just make it mandatory and
simplify the script.

Use ".*" to run all targets.

Signed-off-by: Helen Koike <helen.koike@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25473>
2023-10-11 21:50:58 +00:00
Helen Koike
633c08688b ci/ci_run_n_monitor: merge print_job_status_change with print_job_status
Signed-off-by: Helen Koike <helen.koike@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25473>
2023-10-11 21:50:57 +00:00