Commit Graph

121 Commits

Author SHA1 Message Date
Eric Engestrom
2c6e8b2dd5 Revert "bin/ci_run_n_monitor: explain that the 'Universal Recycling symbol' ♲ emoji means these jobs were cancelled"
This reverts commit 032d4a20f9.

The `if not to_cancel: return` was a red herring as what actually matters
is the job status, which is checked in each cancel_job() call, so we
can't know in advance whether anything will be cancelled, so let's just
drop this text explanation.

In the meantime we've also improved the emoji next to cancelled jobs, so
let's hope there is no longer any need to explain what this long list of
job names means.

Fixes: 032d4a20f9 ("bin/ci_run_n_monitor: explain that the 'Universal Recycling symbol' ♲ emoji means these jobs were cancelled")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30265>
2024-07-19 16:03:24 +00:00
Eric Engestrom
330006375e bin/ci_run_n_monitor: replace ♲ with 🗙 to represent cancelled jobs
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30223>
2024-07-18 19:21:45 +00:00
Eric Engestrom
964ecac0a4 bin/ci_run_n_monitor: add text labels next to the emojis
IMO emojis are nice to add next to the information to recognize things
easily in cases where they are visible, but they should not *replace*
the information.

This adds a readable text next to all emojis (some already had one,
like the "job duration" ones).

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30223>
2024-07-18 19:21:45 +00:00
Eric Engestrom
032d4a20f9 bin/ci_run_n_monitor: explain that the 'Universal Recycling symbol' ♲ emoji means these jobs were cancelled
And not "restarted" for instance, which would also be a reasonable
interpretation.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30223>
2024-07-18 19:21:45 +00:00
Sergi Blanch Torne
e6de8e2533 ci: fix run_n_monitor single execution
When there is a single job as a target and, when it is triggered, it takes
more than one loop time in pending status, it confuses the script to fall in
a wrong return of the pipeline monitoring.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/11517

Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30219>
2024-07-17 11:25:50 +00:00
Sergi Blanch Torne
5873b3ac14 ci: run_n_monitor, sort by name when listing jobs
The job loops, w/o and explicit sort, are sorted by the job id. As they produce
logs, humans could feel an improvement by sorting by name.

Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29444>
2024-07-11 08:28:31 +00:00
Sergi Blanch Torne
12c1bdc31d ci: run_n_monitor, listing job names with a padding
Formatting the output lines with a padding in the job name fields may help
the readability of the information when one tries to follow the progress of
a specific job.

Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29444>
2024-07-11 08:28:31 +00:00
Sergi Blanch Torne
d80d35ceff ci: run_n_monitor, pretty duration with padding
When printing time durations, the smaller units can always have the same
number of digits. Helps to have aligned fields when printing.

Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29444>
2024-07-11 08:28:31 +00:00
Sergi Blanch Torne
45f19b3631 ci: run_n_monitor, arguments review and unicode
Coding style homogenous for all the methods in the file. Document
unicode symbols used.

Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29444>
2024-07-11 08:28:31 +00:00
Eric Engestrom
9a3172e489 bin/ci: allow bugfixes in requirements.txt
The one exception is filecache which is just a 0.x which means any other
0.x might break compatibility, so we can't just set it to `0.*`.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30016>
2024-07-04 20:15:28 +00:00
Eric Engestrom
0b09cf63a8 ci_run_n_monitor: add RUNNING_STATUSES and use it where appropriate
A couple of these were trivially missing the `created` status which
usually doesn't stay for long enough for this very slow script to notice.

The `cancel_job()` function will no longer cancel manual jobs waiting to
be started.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29917>
2024-07-04 09:14:36 +00:00
Eric Engestrom
a578101d5b ci_run_n_monitor: use COMPLETED_STATUSES in more places
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29917>
2024-07-04 09:14:36 +00:00
Eric Engestrom
fb2adbeeb7 ci_run_n_monitor: be coherent about using sets for element in group checks
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29917>
2024-07-04 09:14:35 +00:00
Eric Engestrom
fa3d529f55 ci_run_n_monitor: add support for new canceling job status
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29917>
2024-07-04 09:14:35 +00:00
Eric Engestrom
44e6850016 bin/ci: escape literal url in regex
Fixes: b24dd1fa1c ("ci: Fix parse GitLab pipeline url")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29718>
2024-06-14 11:46:52 +00:00
Sergi Blanch Torne
0c092dc5c4 ci: run_n_monitor, collect and summarize
The job duration is printed during the test progress. It can be collected and
summarized at the end of the process. It is interesting when doing a stress
test as one will have this information together at the end.

Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29419>
2024-06-14 08:04:56 +00:00
Sergi Blanch Torne
b24dd1fa1c ci: Fix parse GitLab pipeline url
When the namespace have a dash, this method cannot recogniza properly
the fields in a url. Better to use a regular expression quickly defining
the fields. The exception raised, when the pattern is not recognized
would help more the handler.

Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29683>
2024-06-13 14:10:57 +00:00
David Heidelberg
27cc8e375b ci: introduce tool for comparing nightly runs
nightly_compare.
Compare the two latest scheduled pipelines and provide information
about the jobs you're interested in.

The job part reports:
 - RED previously passing jobs
 - YELLOW jobs which failed before, but continue to fail
   here is also available link to the previous failed run
 - If no job failing, program exits.

The test part reports:
 - everything in lovely table

If any failing job is found, after this phase, commit list between these
two scheduled run is printed (you can also use the WebUI link).

Example: I care about all Adreno jobs passing and one Radeon (r300)
with gallium-nine tests.

```
./bin/ci/nightly_compare.py --target "a[3-7][0-9][0-9].*|r300-rv530-nine"
```

Co-authored-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Signed-off-by: David Heidelberg <david@ixit.cz>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29392>
2024-06-12 09:26:07 +00:00
Sergi Blanch Torne
2d6e72c2cb ci: continue stress run'n'monitor
When the tool is used to stress test a pipeline, if there are jobs already ran,
use their information like it does when the stress flag is not set.

This provides consistency between the behavior when stress argument is not set,
to when it is set. When it is not set, it uses the information about jobs that
are already done. When it is set, it has to use the information about the
already ran jobs. Also, it saves resources by triggering the minimum required.

Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29432>
2024-06-11 11:22:29 +02:00
Sergi Blanch Torne
d963fd596e ci: fix stress counter in run'n'monitor
The stress counter after enable_job(retry), often stores the new status
instead of the job complete. So, the summary printed later doesn't show the
real evolution of the test.

Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29416>
2024-05-27 20:30:36 +00:00
Eric Engestrom
189b4193ee ci_run_n_monitor: explain how to pass multiple targets without having to use regexes
Fixes: 6825c67c99 ("ci_run_n_monitor: allow passing multiple targets")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28461>
2024-03-29 19:23:24 +00:00
Eric Engestrom
bc38b68ab8 ci_run_n_monitor: read job logs as utf-8
Fixes regular crashes like this one:

    Traceback (most recent call last):
      File "bin/ci/ci_run_n_monitor.py", line 478, in <module>
        print_log(cur_project, target_job_id)
      File "bin/ci/ci_run_n_monitor.py", line 270, in print_log
        lines = job.trace().decode("raw_unicode_escape").splitlines()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    UnicodeDecodeError: 'rawunicodeescape' codec can't decode bytes in position 6090-6091: truncated \uXXXX escape
    decoding with 'raw_unicode_escape' codec failed

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27653>
2024-02-29 14:17:02 +00:00
Eric Engestrom
656e329fa2 ci_run_n_monitor: fix handling of optional jobs again
Looks like a recent rework broke this again, so let's add the existence check back.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27874>
2024-02-29 14:11:32 +00:00
Eric Engestrom
3acb00290d ci_run_n_monitor: allow detached heads as well
When running on a detached head (eg. checkout of a tag or a specific
commit), there is no active branch, so we can't perform this check; just
skip it and assume the user knows what they're doing.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27578>
2024-02-16 11:39:50 +00:00
Eric Engestrom
ef744fa589 ci_run_n_monitor: explain why/when there might be no tracked remote
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27578>
2024-02-16 11:39:50 +00:00
Corentin Noël
b0af2b30ab ci_run_n_monitor: Allow the upstream format to not exist
When pushing directly to a fork, the upstream branch does not exist and thus
returns an empty string.

Signed-off-by: Corentin Noël <corentin.noel@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27567>
2024-02-12 10:52:53 +01:00
Eric Engestrom
9beb9081fc ci_run_n_monitor: print the target regex before adding the X/N bit
To avoid showing something too complex to the user, especially since this isn't something they added themselves or can remove.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27530>
2024-02-09 15:58:10 +00:00
Eric Engestrom
7154c1eb77 ci_run_n_monitor: implicitly include parallel: jobs
This avoids the surprising behaviour where `--target jobname` works for
some jobs but not others, because gitlab adds `X/N` at the end of these
job names.

If the user does specify something like `jobname 1/.*` to only run the
first, the extra `\d+/\d+` is ignored, just like if the job isn't
`parallel:` and therefore doesn't end with `X/N`.

If the user really wants to fail to match parallel jobs (previous
behaviour), they can simply add a `$` at the end of the job name/regex
(but also, I don't see why someone would want that behaviour).

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27530>
2024-02-09 15:58:10 +00:00
Eric Engestrom
716ed7d62a gitlab_gql: print error returned by server in --print-merged-yaml
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27535>
2024-02-09 08:01:20 +00:00
Eric Engestrom
45df652d0b ci_run_n_monitor: refresh job state when starting it
Avoids job staying in 'manual' state and making the script think there's
nothing left running.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27499>
2024-02-08 22:22:54 +00:00
Eric Engestrom
8a05cb8da7 ci_run_n_monitor: track new job when retrying a job
When retrying a job, we are creating a new job, so we need to keep track
of this new job now.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27499>
2024-02-08 22:22:54 +00:00
Eric Engestrom
5bc1a62fbb ci_run_n_monitor: add method to get a pipeline job by its id
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27499>
2024-02-08 22:22:54 +00:00
Eric Engestrom
bce1230587 ci_run_n_monitor: update job when it goes through enable_job()
`enable_job()` modifies the job, so we need to make sure we get the
updated job back out of it.

The next two commits take care of the two specific code paths.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27499>
2024-02-08 22:22:54 +00:00
Eric Engestrom
5758a5d660 ci_run_n_monitor: add some types for gitlab objects
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27499>
2024-02-08 22:22:54 +00:00
Eric Engestrom
044c51b6bf ci_run_n_monitor: warn user if they forgot to push the branch
Only perform this check if they set `--rev HEAD` (or don't specify it);
let's assume if they select another commit, they know what they're
doing.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26969>
2024-02-08 21:38:21 +00:00
Eric Engestrom
5d293f01cc ci_run_n_monitor: avoid spamming a ton of "new status: created" for all the jobs at the beginning
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27419>
2024-02-02 15:25:22 +00:00
Eric Engestrom
2d290eda36 Reapply "bin/ci: Add GitLab basic token validation"
This reverts commit e39fed5737.

One change from the original commit (cd8b546205):
- in `validate_gitlab_token()`, `raise ValueError` was replaced with
  `return False`.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27331>
2024-01-29 18:21:29 +00:00
Eric Engestrom
e39fed5737 Revert "bin/ci: Add GitLab basic token validation"
This validation code has 2 bugs, the main one being that it is wrong and
is refusing perfectly valid codes. Let's remove this until we come up
with a valid check.

This reverts commit cd8b546205.

Fixes: cd8b546205 ("bin/ci: Add GitLab basic token validation")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27312>
2024-01-29 01:44:43 +00:00
Guilherme Gallo
cd8b546205 bin/ci: Add GitLab basic token validation
Validate the token to be able to warn the user about wrong
authentication settings.
Treat the default token argument case, when the token file is not found.
Add some loggings to guide the user.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27206>
2024-01-26 00:37:05 -03:00
Guilherme Gallo
8eabdb3909 bin/ci: Refactor read_token function
Make `read_token` utilize the `get_gitlab_pipeline_from_url` to reuse
code from `gitlab_gql.py`.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27206>
2024-01-26 00:37:05 -03:00
Guilherme Gallo
708a26c607 bin/ci: Move get_token_from_default_dir to common
Moved the `get_token_from_default_dir` method to a common module for
shared use between `gitlab_gql.py` and `ci_run_n_monitor.py`. This
migration facilitates better code organization and potential future
reuse.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27206>
2024-01-26 00:37:05 -03:00
Guilherme Gallo
50fcea9c34 bin/ci: Propagate the token to GitlabGQL
Fix an issue in `ci_run_n_monitor.py` where the token was not
being correctly propagated to the GitlabGQL abstraction. This addresses
misbehavior in scenarios like running pipelines in a private fork,
ensuring proper functionality.

Also document `find_dependencies` function.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27206>
2024-01-26 00:37:03 -03:00
Eric Engestrom
6825c67c99 ci_run_n_monitor: allow passing multiple targets
When generating the list of targets from a script, being able to just
pass `--target "${list[@]}"` is very convenient.

The list of targets is simply converted to an "or" regex, matching any
of the `--target`s given.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27252>
2024-01-25 21:16:13 +00:00
Eric Engestrom
eeba409c54 ci_run_n_monitor: drop always-true condition
`--target` is mandatory.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27252>
2024-01-25 21:16:13 +00:00
Helen Koike
bf461d856f ci/ci_post_gantt: add script that post gantt to Marge's messages
Generate Gantt chart and post an in thread reply to Marge's messages
with it in html format.

html format is used for being interactive.

Signed-off-by: Helen Koike <helen.koike@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26796>
2024-01-20 00:02:56 +00:00
Helen Koike
b289028d0a ci/ci_gantt_chart: show duration on hover
Show the duration of the given phase on hover.

Signed-off-by: Helen Koike <helen.koike@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25793>
2024-01-17 13:10:07 +00:00
Helen Koike
ffaa247b4f ci/ci_gantt_chart: add option to save output to a file
Allow saving in html and in image formats.

Signed-off-by: Helen Koike <helen.koike@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25793>
2024-01-17 13:10:07 +00:00
Helen Koike
cf86e97ace ci/ci_gantt_chart: add timeout vertical line
Add pipeline total duration and add a timeout vertical line to indicate
the 1h Timeout mark.

Signed-off-by: Helen Koike <helen.koike@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25793>
2024-01-17 13:10:07 +00:00
Helen Koike
61ab9ae2af ci/ci_gantt_chart: add tool to analyse pipeline execution time
ci_gantt_chart.py generates a gantt chart from a given pipeline url

Signed-off-by: Helen Koike <helen.koike@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25793>
2024-01-17 13:10:07 +00:00
Helen Koike
9e8cbc8e91 ci/ci_run_n_monitor: move get_gitlab_pipeline_from_url() to gitlab_common
Move this code to gitlab_common since it can be re-used by other
scripts.

Signed-off-by: Helen Koike <helen.koike@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25793>
2024-01-17 13:10:07 +00:00