ci/lava: Detect R8152 issues preemptively and retry

Implement a log-based retry hint for R8152 issue described in #6681,
which is based on detecting these two consecutive lines:

```
r8152 <USB> eth0: Tx status -71
nfs: server <IP> not responding, still trying
```

Where <IP> and <USB> could be any IP and USB addresses, respectfully.

This commit is a temporary fix since it requires a section-aware log
follower, implemented in !16323. When the cited MR is merged, one will
make a proper fix on top of that.

Closes: #6681

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17389>
This commit is contained in:
Guilherme Gallo
2022-07-06 22:59:11 -03:00
committed by Marge Bot
parent 45a4b01427
commit 2c51b7a9c9
4 changed files with 110 additions and 1 deletions

View File

@@ -0,0 +1,40 @@
from __future__ import annotations
import re
from dataclasses import dataclass, field
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from lava.utils import LogFollower
from lava.exceptions import MesaCIKnownIssueException
from lava.utils.log_section import LogSectionType
@dataclass
class LAVALogHints:
log_follower: LogFollower
has_r8152_issue_history: bool = field(default=False, init=False)
def detect_failure(self, new_lines: list[dict[str, Any]]):
for line in new_lines:
self.detect_r8152_issue(line)
def detect_r8152_issue(self, line):
if (
self.log_follower.phase == LogSectionType.TEST_CASE
and line["lvl"] == "target"
):
if re.search(r"r8152 \S+ eth0: Tx status -71", line["msg"]):
self.has_r8152_issue_history = True
return
if self.has_r8152_issue_history and re.search(
r"nfs: server \d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3} not responding, still trying",
line["msg"],
):
raise MesaCIKnownIssueException(
"Probable network issue failure encountered, retrying the job"
)
self.has_r8152_issue_history = False