Commit Graph

367 Commits

Author SHA1 Message Date
Richard Purdie
a263c0b0d3 srcipts/generate-chartdata: Drop dateutil dependency
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-30 16:33:54 +01:00
Richard Purdie
400e1e1b2b scripts/utils: Allow tags to be updated from upstream (meta-agl has force pushed them)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-29 16:44:03 +01:00
Richard Purdie
54630b14ce scripts/run-patchmetrics: Ensure cve data for last year is copied
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-29 07:45:46 +01:00
Richard Purdie
01f4920d78 scripts/generate-charts: Add json data for last year as well as multiyear
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-29 00:04:23 +01:00
Richard Purdie
70cdf365ef run-docs-build: Switch to newer docs tarball
This updates docs generate to a newer version of sphinx.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-23 13:31:41 +01:00
Michael Opdenacker
a1575b063b run-docs-build: allow build warnings again
This allows to switch to a more recent of Sphinx
which will generate warnings with old versions of docs.

This way, it's not immediately necessary to patch
all such versions.

This commit reverts
https://git.yoctoproject.org/yocto-autobuilder-helper/commit/?id=8273124feb9da2ffe93fcee7c4529d5597e1684a
which originally reverted
https://git.yoctoproject.org/yocto-autobuilder-helper/commit/?id=931d409b255a85f2217ca093d8391a678ce00ddb

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-23 13:30:26 +01:00
Richard Purdie
1baf966db6 run-docs-build: Ensure origin is specified for master
We may not have fetched the master revision if a different one was being
built so using origin/master works consistently in all cases.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-23 13:29:56 +01:00
Richard Purdie
3921d4f3ec scripts: Ensure directories exist
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-14 22:09:16 +01:00
Richard Purdie
9d7820da27 scripts/docs_add_banner: Fix several issues
* Fix a syntax error causing the script to fail
* Skip symlinks files (symlink directories were already skipped)
* Drop unneeded whitespace
* Abstract path join to a variable

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-03 10:51:38 +01:00
Abongwa Amahnui Bonalais
5dd90ea43b scripts/docs_add_banner: Add a banner on the old documentation docs.
Add a script which adds banners to the old docs and outdated dunfell docs
so that the user knows they're using old documentation.

Signed-off-by: Abongwa Bonalais Amahnui <abongwabonalais@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-03 10:51:08 +01:00
Quentin Schulz
9ebeb61276 scripts: run-docs-build: do not extract eclipse directories from old docs
For some reason, the old docs tarball includes many eclipse
subdirectories which are just cluttering the docs website up.

Therefore, let's just not extract eclipse directories from the tarball.

Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-01 09:14:58 +01:00
Richard Purdie
a772bf318d scripts/cve-generate-chartdata: No need to exlude db
These are already handled in all branches we care about so drop the exclusion.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-25 17:55:12 +01:00
Richard Purdie
60902768cc scripts/run-patchmetrics: Drop honister, remove oldder json files
Keeping all the previous json files in tree has too much overhead
so drop the old ones. Honister doesn't support cve-check with json.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-25 17:52:39 +01:00
Richard Purdie
47d8883222 scripts/run-patchmetrics: Continue CVE checks for honister for now
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-25 09:46:15 +01:00
Richard Purdie
10ecb12155 scripts: Add script to update CVE count data in json file
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-25 08:32:19 +01:00
Richard Purdie
6655466ea6 scripts/run-patchmetrics: Seperate out cve-check json files into direcotries
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-24 15:21:23 +01:00
Richard Purdie
573aee4421 scripts/patchmetrics-generate-chartdata: Remove unused data
When generating the chart specific data, remove elements which aren't
used to reuse the filesize which improves browser loading speed.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-24 11:05:17 +01:00
Richard Purdie
7c7b96cf19 scripts/patchmetrics-update: Fix typo
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-23 23:18:37 +01:00
Richard Purdie
803bdecd62 scripts/run-patchmetrics: Ensure tmpdir is clean between build runs
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-23 22:56:23 +01:00
Richard Purdie
125f1dbea4 scripts: Add patch to filter the chart data to make it usable
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-23 22:54:54 +01:00
Richard Purdie
c491e01653 scripts/run-patchmetrics: Ensure environment works for dunfell
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-23 22:11:19 +01:00
Richard Purdie
e3beeb9328 scripts: Extend CVE testing to kirkstone and dunfell branches
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-23 16:22:35 +01:00
Richard Purdie
7eca1ae4d7 scripts: Add ability to add/process missing patch stats data
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-23 16:05:23 +01:00
Richard Purdie
ca67ed751e run-patchmetrics: Add very basic cve-check functionality
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-17 17:14:13 +01:00
Richard Purdie
8702fdbb72 Clarify license and copyright information
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-17 10:29:45 +01:00
Richard Purdie
348467c2c8 run-patchmetrics: Use -m for realpath for results dir as it doesn't exist
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-13 17:21:45 +01:00
Richard Purdie
5b73a9b6eb run-patchmetrics: Use full variable names
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-13 17:19:17 +01:00
Richard Purdie
fd1aa61d87 scripts/run-patchmetrics: Ensure paths are sanitized
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-13 16:39:59 +01:00
Richard Purdie
1887e59889 config/scripts: Allow publishing of patch metrics
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-13 16:37:10 +01:00
Richard Purdie
2765a6a314 config/scripts: Add initial patch metric tracking code
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-13 16:21:57 +01:00
Richard Purdie
47ef936ea5 config/run-config: Allow other OE-Core machines to be tested manually
These are not part of our standard testing but allow them to
be triggered manually to assess their status.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-13 15:17:54 +01:00
Richard Purdie
91f49091cc scripts/run-docs-build: Use set-versions.py to get the latest release branch name
The intent here is to use the latest release branch name instead of a tagged version.
For the 4.0 case, this means the full migration guide will be shown instead of
the reduced information in the 4.0 tag.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-26 11:35:28 +01:00
Quentin Schulz
f418022d8f scripts: run-docs-build: reuse logic to link to latest tag
Since the latest tag already has a specific handling in the forloop,
let's just move the symlink creation inside the forloop.

Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-20 13:13:15 +01:00
Quentin Schulz
1870fb3a6c scripts: run-docs-build: make latest documentation tag the default version
Currently, the master branch is the default branch being displayed by
default when reaching the Yocto docs website. When big changes are
implemented for the next release, these are shown immediately to the
user, even though there is currently no release available for those
changes. This is an issue when e.g. behaviors are changed, new features
are added, variables get renamed or some syntaxes change because the
user might try to use things that aren't available to them yet.

I believe more people are using released version of Yocto
Project/Bitbake than people working on latest master. So let's make the
default version of the docs the latest tag (in terms of version number,
not date of tagging) to avoid too much confusion.

The master branch of the docs is now available at /dev subpath.

Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-20 13:13:15 +01:00
Quentin Schulz
4cd985b23d scripts: run-docs-build: use set_versions.py and switchers.js.in from master
set_versions.py and switchers.js.in need to be up-to-date so that a
consistent behavior is kept between different branches and tags of the
documentation.

Right now, kirkstone branch is lagging behind master and therefore does
not have the latest changes from master (e.g. the new obsolete
algorithm, which obviously isn't an issue right now, but will be in two
years). Using master version for those scripts also lightens the
maintenance burden.

Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-19 22:07:09 +01:00
Quentin Schulz
2852909e55 scripts: run-docs-build: add option to skip upstream rsync
The rsync only makes sense for the Autobuilder as most people don't have
access to docs@docs.yoctoproject.org. Therefore, to allow for easier
contribution to this script, let's allow to skip the rsync so the script
can continue to run.

Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-19 22:07:09 +01:00
Quentin Schulz
2f972bb742 scripts: run-docs-build: allow to pass buildtools script and docbook tarball paths
Currently, the path to buildtools script and docbook tarball are
hardcoded to work on Yocto Project Autobuilder. However, this makes it
harder to contribute to this script because it is very unlikely those
paths exist on a developer PC.

Instead, let's allow to override variables by using the environment and
make the current hardcoded values the default ones.

Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-19 22:07:09 +01:00
Quentin Schulz
71527dd20d scripts: run-docs-build: transform argument paths into absolute paths
The script is written in such a way that absolute paths are expected.
Instead of failing weirdly at some point in the script, let's just make
all paths passed to this script absolute by calling realpath on them.

Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-19 22:07:09 +01:00
Richard Purdie
5946a6f028 scripts/run-docs-build: Support auto generation of switchers.js and poky.yaml
This uses the set_versions.py script from the docs master branch to handle
creation of the swictchers.js version information as well as the data in
poky.yaml.

This allows patches to those sections of the docs to be dropped.

It is assumed the patch to use set_versions is applied to the docs transitions
branch so that it's switchers.js files no longer need to be tweaked.

This does lead to user visible changes on the website:

- Older versioned released docs gain their current version and the
  latest version listed in the switcher rather than being unlisted
- The list of releases is normally filtered down to our active ones
  (hardknott, honister and dev right now)
- 3.3's doc references to gatesgath are corrected to hardknott
- Docs for unversioned release branches (not linked on the website) now
  use 3.1.999 versioning instead of confusing them with the last relased version
- 3.1.13 refers to 3.1.13 instead of 3.1.12

all of which seem to be reasonable improvements.

The big advantage of these changes is that with a single change to the script
in the master branch, the right thing should happen for all the current docs
and at release time, only a docs rebuild should need to be triggered after tags
are pushed.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-22 22:17:02 +00:00
Quentin Schulz
6fd933fe06 scripts: run-docs-build: factor out yocto-docs tags and branches building
Except patching which is specific to tags and yocto- tag prefix
stripping, the logic is identical, so let's merge both loops together.

Cc: Quentin Schuls <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-18 17:02:09 +00:00
Quentin Schulz
c2e3a4515b scripts: run-docs-build: simplify sphinx-buildable yocto-docs tag list fetching
The commit that introduced Sphinx support in yocto-docs is
01dd5af7954e24552aca022917669b27bb0541ed. Any tag containing this commit
is buildable by sphinx.

Dunfell tags don't all have Sphinx support. However, all tags containing
the introducing commit c25fe058b88b893b0d146f3ed27320b47cdec236 are
buildable by sphinx.

Therefore, let's just list all tags which contains either of those two
commits instead of the complex series of pipes and shell commands.

Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-18 17:02:09 +00:00
Quentin Schulz
d69a2adc43 scripts: run-docs-build: automatically build new yocto-docs branches
Since commit 01dd5af7954e24552aca022917669b27bb0541ed, all later
releases of yocto-docs can be built with Sphinx. Instead of manually
updating this list, let's have git return the list of remote branches
which contains the commit.

dunfell branch was initially released without Sphinx support but was
later patched, hence why it's explicitly listed.

Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-18 17:02:09 +00:00
Quentin Schulz
592863ebab scripts: run-docs-build: factor out all yocto-docs branches building
master, master-next and transition only differ from other branches by
their output directory name. Let's put everything in common and only
have a check on whether the branch is master, master-next or transition
and modify the output dir in those cases.

Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-18 17:02:09 +00:00
Quentin Schulz
72ca6fc8dc scripts: run-docs-build: automatically build new Bitbake branches
Since commit 84ccba0f4aff91528f764523fe1205a354c889ed, docs of all later
releases can be built with Sphinx. Instead of manually updating this
list, let's have git return the list of remote branches which contains
this commit.

1.46 branch was initially released without Sphinx support but was later
patched, hence why it's explicitly listed.

Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-18 17:02:09 +00:00
Quentin Schulz
7ed0c8a92f scripts: run-docs-build: factor out all bitbake branches building
master and master-next only differ from other branches by their output
directory name. Let's put everything in common and only have a check on
whether the branch is master or master-next and modify the output dir in
those cases.

Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-18 17:02:09 +00:00
Quentin Schulz
8273124feb Revert "scripts: run-docs-build: do not turn sphinx warnings into errors"
The patch was meant as a quick temporary work-around to have the docs
built and published.

Now that releases where -W flag is set (turning warnings into errors)
are appropriately patched to make those warnings disappear (on Sphinx
v3.2.1 which is the one used on the builder), this patch can be reverted
so that next time a warning appears the doc building will fail but will
not destroy the doc website (because of commit
6a4e6ef18d "scripts: run-docs-build: make
the script fail hard ASAP when there's an error", since rsync will not
be run if any error happens before).

This reverts commit 931d409b25.

Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-15 22:03:37 +00:00
Quentin Schulz
8b80b6f485 scripts: docs-build-patches: fix incorrect conf.py for yocto-3.4.1
The patch was missing from yocto-3.4.1 release and only made it to
yocto-3.4.2 unfortunately so let's backport it.

Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-15 22:03:37 +00:00
Quentin Schulz
4a97ab144d scripts: docs-build-patches: fix bitbake objects.inv being used
Each Yocto Project release is validated for a specific Bitbake release.

By mistake, most release docs are pointing at the development branch of
Bitbake documentation instead of the one that was validated for the
Yocto Project release.

This fixes the issue by patching the conf.py of impacted releases.

Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-15 22:03:37 +00:00
Quentin Schulz
62e38ffaab scripts: run-docs-build: simplify patching logic
There's no need to keep a list of tags requiring to be patched since the
tag is part of the path where patches are stored.

Therefore, let's only check if there's a patch directory for a given tag
and if so, apply all patches in there.

Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-15 22:03:37 +00:00
Richard Purdie
c0a7cac39a oe-build-perf-report-email: Drop phantomjs email generation support
This approach didn't work reliably so switch to text based emails
pointing at the website version with graphs and data.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-15 17:22:03 +00:00