yocto-autobuilder-helper/scripts/run-docs-build

#!/bin/bash

SPDX-License-Identifier: GPL-2.0-only

Called with $1 as the build directory

$2 as the path to yocto-docs

$3 as the path to bitbake

Environment variables:

- buildtools_url as the path to buildtools script for the docs.

Can be found here: https://autobuilder.yocto.io/pub/buildtools/

- docbookarchive_url as the path to old (pre 3.1.5 and Sphinx migration) docs tarball

Can be found here: https://downloads.yoctoproject.org/mirror/docbook-mirror/docbook-archives-20201105.tar.xz

- PUBLISH (0/1) for whether the files should be rsync'ed to docs.yoctoproject.org

set -e set -u set -o pipefail set -x

builddir=$(realpath "$1") ypdocs=$(realpath "$2/documentation/") bbdocs=$(realpath "$3/doc/") outputdir=$builddir/output scriptdir="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" PUBLISH=${PUBLISH:-1} sharedir=jq -r '.["BASE_SHAREDDIR"]' < $HOME/config-local.json

Modified 5.1 tools image to include rsvg, switch to next released version

buildtools_url=${buildtools_url:-https://downloads.yoctoproject.org/releases/yocto/yocto-4.1.2/buildtools/x86_64-buildtools-docs-nativesdk-standalone-5.1-modified.sh} buildtools_url_old=${buildtools_url_old:-https://downloads.yoctoproject.org/releases/yocto/yocto-4.1.2/buildtools/x86_64-buildtools-docs-nativesdk-standalone-4.1.2.sh} docbookarchive_url=${docbookarchive_url:-https://downloads.yoctoproject.org/mirror/docbook-mirror/docbook-archives-20201105.tar.xz}

buildtools_name=basename $buildtools_url buildtools_localpath=${sharedir}/cluster-downloads-cache/${buildtools_name} if [ ! -e ${buildtools_localpath} ]; then mkdir -p dirname ${buildtools_localpath} wget ${buildtools_url} -O ${buildtools_localpath} chmod a+x ${buildtools_localpath} fi cd $builddir mkdir buildtools ${buildtools_localpath} -y -d $builddir/buildtools

buildtools_name_old=basename $buildtools_url_old buildtools_localpath_old=${sharedir}/cluster-downloads-cache/${buildtools_name_old} if [ ! -e ${buildtools_localpath_old} ]; then mkdir -p dirname ${buildtools_localpath_old} wget ${buildtools_url_old} -O ${buildtools_localpath_old} chmod a+x ${buildtools_localpath_old} fi

cd $builddir mkdir buildtools-old ${buildtools_localpath_old} -y -d $builddir/buildtools-old

To build the latexpdf output we need Xetex (which can handle international characters)

Rather than require it on all autobuilder workers, along with dependencies, we install

a copy onto NFS which the workers can use. The full install is 6GB so just install

the pieces we've found we need via tlmgr on top of a small install. This script

creates the install if it isn't present.

For local docs builds, we assume the user has the tools available.

if [ -e /srv/autobuilder/valkyrie.yocto.io/ ]; then textooldir=/srv/autobuilder/valkyrie.yocto.io/docs-build-tex-tools if [ ! -e ${textooldir} ]; then mkdir -p /tmp/textemp pushd /tmp/textemp wget https://mirror.ctan.org/systems/texlive/tlnet/install-tl-unx.tar.gz zcat < install-tl-unx.tar.gz | tar xf - cd install-tl-*/ ./install-tl --scheme=small --texdir=${textooldir} --no-interaction PATH=$PATH:${textooldir}/bin/x86_64-linux tlmgr install titlesec varwidth tabulary needspace upquote framed capt-of wrapfig fncychap gnu-freefont ctex latexmk nimbus15 popd fi PATH=$PATH:${textooldir}/bin/x86_64-linux fi

Provide our own fonts.conf path to make fonts installed with tlmgr available

system-wide. It will pick up the fonts.conf file in this directory.

This is then used by rsvg-convert to pick up the Nimbus font when converting

files.

export FONTCONFIG_PATH="$scriptdir/docsfontconfig"

Getting the old docbook built docs from an archive. Not rebuilding them.

docbookarchive_localpath=${sharedir}/cluster-downloads-cache/basename ${docbookarchive_url} if [ ! -e ${docbookarchive_localpath} ]; then mkdir -p dirname ${docbookarchive_localpath} wget ${docbookarchive_url} -O ${docbookarchive_localpath} fi

mkdir $outputdir cd $outputdir echo Extracing old content from archive tar --exclude=eclipse -xJf ${docbookarchive_localpath}

$scriptdir/docs_add_banner.py

cd $bbdocs mkdir $outputdir/bitbake

A decision was made to keep updating all the Sphinx generated docs for the moment,

even the ones corresponding to no longer supported releases

https://lists.yoctoproject.org/g/docs/message/2193

We copy the releases.rst file from master so that all versions of the docs

see the latest releases.

first_sphinx_commit=84ccba0f4aff91528f764523fe1205a354c889ed

latest_branch=$(git branch --remote --contains "$first_sphinx_commit" --format '%(refname:lstrip=3)' --sort='-version:refname' | grep --max-count=1 "[0-9].[0-9]")

for branch in 1.46 $(git branch --remote --contains "$first_sphinx_commit" --format '%(refname:lstrip=3)'); do if [ "$branch" = "HEAD" ]; then continue fi

echo Building bitbake $branch branch
git checkout $branch
git reset --hard
git clean -ffdx
git checkout origin/master releases.rst

(
    . $builddir/buildtools/environment-setup*
    make clean
    SPHINXOPTS="-j auto" make publish
)

if [ "$branch" = "master-next" ]; then
    branch="next"
mkdir $outputdir/bitbake/$branch
elif [ "$branch" = "master" ]; then
    branch="dev"
    mkdir $outputdir/bitbake/$branch
elif [ "$branch" = "$latest_branch" ]; then
    branch=""
    mkdir $outputdir/bitbake/$latest_branch
    cp -r ./_build/final/* $outputdir/bitbake/$latest_branch
else
mkdir $outputdir/bitbake/$branch
fi

cp -r ./_build/final/* $outputdir/bitbake/$branch
git reset --hard
git clean -ffdx

done

if [ "$PUBLISH" -ne 0 ]; then # only sync bitbake folder for now. We need bitbake to be published first # since the bitbake intersphinx index will be downloaded to build yocto-docs cd $outputdir rsync -irlp --checksum --ignore-times --delete bitbake docs@docs.yoctoproject.org:docs/ fi

cd $ypdocs

transition must build after master for the switchers.js file

Again, keeping even the no longer supported releases (see above comment)

first_sphinx_commit=01dd5af7954e24552aca022917669b27bb0541ed first_dunfell_sphinx_commit=c25fe058b88b893b0d146f3ed27320b47cdec236

ypdocsbranch=$(git symbolic-ref -q --short HEAD || git rev-parse HEAD)

git checkout origin/master set_versions.py #latest_tag=$(git tag --contains "$first_sphinx_commit" --contains "$first_dunfell_sphinx_commit" --sort="version:refname" 'yocto-*' | tail -1 | sed 's/yocto-//') latest_tag=$(./set_versions.py getlatest) git reset --hard git clean -ffdx

for branch in "$ypdocsbranch" dunfell $(git branch --remote --contains "$first_sphinx_commit" --format '%(refname:lstrip=3)') $(git tag --contains "$first_sphinx_commit" --contains "$first_dunfell_sphinx_commit" 'yocto-*') transition; do if [ "$branch" = "HEAD" ]; then continue fi

# Do not build <release>-next branches as they are development branches only
# Do build master-next branch and request branch though!
if echo "$branch" | grep -v "master-next\|$ypdocsbranch" | grep -q -E "-next$"; then
    continue
fi
# Do not build contrib/ branches as they are development branches only
# Do build request branch though!
if echo "$branch" | grep -v "$ypdocsbranch" | grep -q -E "^contrib/"; then
    continue
fi

echo Building $branch
git checkout $branch
git reset --hard
git clean -ffdx

if [ -e "${scriptdir}/docs-build-patches/${branch}/" ]; then
    echo Adding patch for $branch
    git am "${scriptdir}/docs-build-patches/${branch}/"000*
fi

git checkout origin/master sphinx-static/switchers.js.in set_versions.py
if [ -e poky.yaml ]; then
    cp poky.yaml poky.yaml.in
    case $branch in
        yocto-*)
            ./set_versions.py $(echo "$branch" | sed 's/yocto-//')
            ;;
        *)
            ./set_versions.py
            ;;
    esac
fi

(
    # Anything older than scarthgap needs the older buildtools
    if git merge-base --is-ancestor 0cdc0afd3332459d30cfc8f4c2e62bdcc23f5ed5 HEAD; then
        . $builddir/buildtools/environment-setup*
    else
        . $builddir/buildtools-old/environment-setup*
    fi
    make clean
    SPHINXOPTS="-j auto" make publish
)

# Strip yocto- from tag names
branch=$(echo "$branch" | sed 's/yocto-//')

if [ "$branch" = "master-next" ]; then
    branch="next"
mkdir -p $outputdir/$branch
elif [ "$branch" = "master" ]; then
    branch="dev"
    mkdir -p $outputdir/$branch
elif [ "$branch" = "$latest_tag" ]; then
    branch=""
    mkdir -p $outputdir/$latest_tag
    cp -r ./_build/final/* $outputdir/$latest_tag
    echo Linking to $latest_tag as current
    ln -s $latest_tag $outputdir/current
elif [ "$branch" = "transition" ]; then
    branch=""
else
mkdir -p $outputdir/$branch
fi

cp -r ./_build/final/* $outputdir/$branch
git reset --hard
git clean -ffdx

done

Update bitbake switchers.js with the copy from master ypdocs

cd $outputdir/bitbake find . -name switchers.js -exec cp $outputdir/current/_static/switchers.js {} ;

if [ "$PUBLISH" -ne 0 ]; then cd $outputdir rsync -irlp --checksum --ignore-times --delete . docs@docs.yoctoproject.org:docs/ fi