User guide for the annotation interface

The main purpose of the website is collecting log annotations for the training dataset. On the homepage, or under Review Logs tab, we provide a GUI for annotating failed build logs. This section is meant for people (package developers, maintainers, etc.) who want to help improve Log Detective. They can do that by submitting log annotations here, thus growing our training dataset.

How to upload a log
Log Detective can download the build directly from Koji, Copr or Packit. If you have the log somewhere else (different Koji instance, local Mock build), you can point Log Detective to URL where it can be downloaded. The URL can be shortlived, e.g., Pastebin, etc. If you have more logs for one build, you can concat them. E.g., cat build.log root.log | fpaste. Open the paste in a web browser to find its RAW URL, and provide it to Log Detective.
How to annotate a log
  • After uploading a log, find its relevant part, highlight it with mouse, and click on "Add" button in the right column.
  • This will create new Form Field "Snippet 1" in the right column. State there why this part of log is interresting. E.g., "Here the build fails, because the /usr/bin/make command is not available." You can add more these snipets.
  • Then navigate to "Why did the build fail?" and describe the reason, e.g., "The build failed because `make` command was not available in the buildroot and the %install sections requires it."
  • And finaly go to "How to fix the issue?" and provide a hint how to fix it. E.g., "You have to tell Mock to install the command `make` in buildroot. You can do that by putting `BuildRequires: make`. If you are not sure what package provides this command you can check it using `dnf whatprovides /usr/bin/make` command."
  • And then Submit your annotation. And remember - the more details you provide, the more details will the AI provide to you later.
A few log snippet examples with explanations for reference
Example 1
Log Snippet
Problem: package python3-pydantic-2.5.3-1.fc40.noarch requires python(abi) = 3.12, but none of the providers can be installed
  - package python3-devel-3.13.0~a3-1.fc40.x86_64 conflicts with python3 < 3.13.0~a3-1.fc40 provided by python3-3.12.1-2.fc40.i686
  - package python3-devel-3.13.0~a3-1.fc40.x86_64 conflicts with python3 < 3.13.0~a3-1.fc40 provided by python3-3.12.1-2.fc40.x86_64
  - cannot install the best candidate for the job
Explanation
DNF5 couldn't install `python3-pydantic-2.5.3-1.fc40.noarch` as the best find the best candidate for dependency `python(abi) = 3.12` had conflicts with `python3 < 3.13.0~a3-1.fc40`.
Example 2
Log Snippet
Problem 1: conflicting requests
  - nothing provides python3.10dist(decorator) needed by python3-autodoc-0.5.0-1.fc36.noarch from copr_base
  - nothing provides python3.10dist(webtest) needed by python3-autodoc-0.5.0-1.fc36.noarch from copr_base
  - nothing provides python(abi) = 3.10 needed by python3-autodoc-0.5.0-1.fc36.noarch from copr_base
 Problem 2: cannot install the best candidate for the job
  - nothing provides python(abi) = 3.12 needed by python3-sphinx-autodoc-typehints-1.25.2-1.fc40.noarch from copr_base
  - nothing provides python3.12dist(sphinx) >= 7.1.2 needed by python3-sphinx-autodoc-typehints-1.25.2-1.fc40.noarch from copr_base
 Problem 3: cannot install the best candidate for the job
  - nothing provides python(abi) = 3.12 needed by python3-xarray-2023.11.0-1.fc40.noarch from copr_base
  - nothing provides python3.12dist(numpy) >= 1.22 needed by python3-xarray-2023.11.0-1.fc40.noarch from copr_base
  - nothing provides python3.12dist(pandas) >= 1.4 needed by python3-xarray-2023.11.0-1.fc40.noarch from copr_base
  - nothing provides python3.12dist(packaging) >= 21.3 needed by python3-xarray-2023.11.0-1.fc40.noarch from copr_base
Explanation
There are conflicting packages. It seems that some packages requires python 3.10 while other requires python3.12.
Example 3
Log Snippet
[error: Bad exit status from /var/tmp/rpm-tmp.yAXiQF (%build)
Bad exit status from /var/tmp/rpm-tmp.yAXiQF (%build)
RPM build errors:
Child return code was: 1]: [The build failed in `%build` phase.]
=============
[CMake Generate step failed.  Build files cannot be regenerated correctly.]: [CMake cannot prepare the build environment.]
=============
[CMake Error in CMakeLists.txt:
Imported target "Qt6::QmlPrivate" includes non-existent path
"/usr/include/qt6/QtCore/6.8.0"
in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:
* The path was deleted, renamed, or moved to another location.
* An install or uninstall procedure did not complete successfully.
* The installation package was faulty and references files it does not
provide.
CMake Error in CMakeLists.txt:
Imported target "Qt6::QmlPrivate" includes non-existent path
"/usr/include/qt6/QtCore/6.8.0"
in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:
* The path was deleted, renamed, or moved to another location.
* An install or uninstall procedure did not complete successfully.
* The installation package was faulty and references files it does not
provide.]: [You are missing devel files for qt6-qtbase-private.]
Explanation
Issue: The build failed in `%build` phase becase CMake was missing devel files for qt6-qtbase-private.
Resolution: You should add build dependency to your spec file:

BuildRequires: qt6-qtbase-private-devel
Example 4
Log Snippet
=========================== short test summary info ============================
ERROR tests/test_packager_dashboard_bodhi.py - ValueError: The 'OIDC_REQUIRE_...
ERROR tests/test_packager_dashboard_bugzilla.py - ValueError: The 'OIDC_REQUI...
ERROR tests/test_packager_dashboard_health_check.py - ValueError: The 'OIDC_R...
ERROR tests/test_packager_dashboard_helpers.py - ValueError: The 'OIDC_REQUIR...
ERROR tests/test_packager_dashboard_koschei.py - ValueError: The 'OIDC_REQUIR...
ERROR tests/test_packager_dashboard_orphans.py - ValueError: The 'OIDC_REQUIR...
ERROR tests/test_packager_dashboard_pagure.py - ValueError: The 'OIDC_REQUIRE...
!!!!!!!!!!!!!!!!!!! Interrupted: 7 errors during collection !!!!!!!!!!!!!!!!!!!!
======================== 2 warnings, 7 errors in 1.97s =========================
py312: exit 2 (2.40 seconds) /builddir/build/BUILD/oraculum-0.2.4> pytest pid=749
  py312: FAIL code 2 (2.40 seconds)
  evaluation failed :( (2.46 seconds
Explanation
Pytest test cases that failed.
Example 5
Log Snippet
rm: cannot remove '/builddir/build/BUILD/mariadb11.4-11.4.5-build/BUILDROOT/usr/share/man/man1/myrocks_hotbackup.1*': No such file or directory
Explanation
'rm' command failed. File does not exist. This can happen e.g. after package rebase, when the content of the sources changes. Usually happens in %prep or %install sections.
Demo with comments
Demo video

Documentation of the Log Detective tool

Here, we describe Log Detective itself, its main features, and plans for the future. You can also use this website to try out Log Detective for yourself (under Explain logs). For further testing and experimentation, we encourage you to visit our GitHub repository and learn more there.

Why? (The motivation)
The build logs can have thousands of lines. When the build fails, it is very hard to spot the problem, even for seasoned engineers. The string “ERROR” is not always an error. The error is not always at the end of a log. And you have to be fluent in packaging to recognize how to fix the problem. We gathered data from Copr, and it takes developers days and even weeks to fix the failed build.
How? (The overview)
We decided to build an AI tool that will read the log and use LLM to give you a human-like answer. We know that the goal is achievable, but we do not have enough data. We have millions of failed jobs from Koji, and Copr. But it is not paired with a description of what needs to be fixed and how. Therefore, we started with Phase 1 and we have created this website where you can upload the failed build log and annotate it. During our experiments, we learned that we will require around one thousand annotated logs to be able to build the AI tool.
What? (The goal)

The final tool will be an AI model and our artifact will be a binary blob of trained data. This model together with the command line tool will be able to read the build log and give you guidance on why the build failed. Log Detective is already successfully integrated with Copr, and we plan to also incorporate this tool to other build systems (Packit, Koji, ...) to give you a nice UI.

We also use Log Detective in merge requests on Gitlab as a bot. The bot analyses the failed CI builds and posts its full analysis as a comment inside the merge request. See the GitLab webhook source code.
The limitations

The tool is only limited to describing/explaining the failures. The explanations may be incorrect or "hallucinated" (as is the case with all state-of-the-art LLMs). Log Detective cannot perform actions (like fixing the build). Additionally, to save time and computational costs, it analyzes log snippets, and not the entire log (since most of the log is usually irrelevant to the build failure). The snippet extraction is an additional preprocessing step which adds a layer of potential inaccuracy, since the extraction may remove possibly relevant parts of the log (even if on the first glance they seem unimportant).

Disclaimer: When using the tool, you should always review the AI-generated explanations and possible fixes for accuracy before using them. The response is merely a suggestion, and by no means is it a replacement for a review by an experienced developer.
We need your help
Please upload a build log from your recent failed build - we can import it from Copr, Packit, Koji and even from arbitrary URL. Provided that it points to a parsable text file. Mark which lines of the log are relevant to the failure. And describe how it can be fixed. The more elaborate you will be, the better will be the final tool.
Future Plans
We plan to add a review tool to Log Detective. Similarly what Common Voice has. This will help us to identify bad samples and remove them from the data set.

FAQ

What is our first goal?
Our first step is to collect 1000 annotated logs. This is the estimated amount we need to start producing useful results when training AI. After we reach this goal, we will begin working on a CLI tool that will use the trained data. We will then continue to collect data as 2-3 thousands samples should be needed to get satisfactionary results.
What AI model are you using?
We do not know yet. Tomáš experimented with several models and literally every week he came with a new model that just became available and that is better than our previous choice. We assume that when we start training the AI model, we will be using a model that does not exist yet.
What about privacy?

We do not track any sensitive data. When you upload a build log, make sure you have the rights to do that. All data is available under CDLA-Permissive-2.0 license. The tool is designed to only analyze build logs.

Data retention notice: Log Detective does not store any data sent for analysis, nor does it store resulting explanations and solutions. The logs and responses are available to the user only for the duration of the interaction with Log Detective (on the client side).
Did you consider using AI to generate the data set?
Yes, we considered it. However, we cannot use ChatGPT and similar tools because the problematic license can taint the whole project. Additionally, using AI-generated data to teach AI is prone to hallucinations. We are considering using data from Bugzilla and other tools (if Legals allows that), but we want to have a strong core dataset with a clear license.
I have lots of failed logs, can I bulk upload them?
Uploading the log is 5 seconds work. But you have to annotate it. Describe what part of log is relevant and why. Why the build failed and how can be fixed. This takes several minutes. Therefore bulk uploading will not help us.
I did not see failed log for ages - do you have some?
Pick anything from Koji's Failed Builds

Contact us / Provide feedback

We encourage you to reach out to us in case you have any questions or suggestions. You can also help us by providing feedback on any incorrect, unhelpful, or problematic explanations. Use the following feedback methods: