MIRRORS/meta-virtualization
1
0
Fork 0
mirror of git://git.yoctoproject.org/meta-virtualization.git synced 2025-07-19 20:59:41 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity

oe-go-mod-autogen: add README and extended --help information

Browse Source 
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
This commit is contained in:
Bruce Ashfield 2023-09-08 19:58:53 +00:00
parent ad460da806
commit 5510a784ff
2 changed files with 178 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

81
scripts/README-oe-go-mod-autogen.txt Normal file
View File

@ -0,0 +1,81 @@
Description:
oe-go-mod-autogen.py is a helper script for go mod recipes. It is based
on intiial efforts to use only the git fetcher versus golangs module
infrastructure.
Example:
cmd: <path_to>/meta-virtualization/scripts/oe-go-mod-autogen.py \
--repo https://github.com/docker/compose --rev v2.20.3
output: src_uri.inc, relocation.inc, modules.txt
Copy these three generated files to replace the original ones,
then we only need update PV and SRCREV, and docker-compose is upgraded.
See --help for more explanations and examples.
Below are some technical details.
* get module's repo from module name
This script checks the following two URLs to determine the module's repo.
1. https://<module_name_tweaked>?=go-get=1
2. https://pkg.go.dev/<module_name_tweaked>
The module_name_tweaked is derived from module_name, with the last components
removed one by one. Let me use two examples to explain this.
For module_name sigs.k8s.io/json, the sigs.k8s.io/json is first used as
module_name_tweaked for searching. And we can correctly get the repo URL, so
the search stops.
For module_name github.com/k3s-io/etcd/api/v3, the following ones are used
as module_name_tweaked:
github.com/k3s-io/etcd/api/v3
github.com/k3s-io/etcd/api
github.com/k3s-io/etcd
And when searching 'github.com/k3s-io/etcd', we get the repo URL, so the search
stops.
* determine the srcdir:destdir mapping in 'vendor' creation
To correctly form the 'vendor' directory, the mapping is critical.
This script makes use of tag matching and path matching to determine
the subpath in the repo for the module.
* avoid subpath being overriden by parent path
We need to avoid subpath being overriden by parent path. This is needed
for both SRC_URI ordering in src_uri.inc and the sites mapping ordering
in relocation.inc. This script simply uses the length as the ordering key,
simply for the reason that if a path is a subpath of another path, it must
be longer.
* the .git suffix is removed to sync with each other
Unlike normal recipes, go mod recipe usually have many SRC_URIs. This script
remove the '.git' suffix from repo URL so that the repo URLs are in sync
with each.
* basic directory hierarchy and caching mechanism
<cwd>/repos: hold the repos downloaded and checked
<cwd>/wget-contents: hold the contents to determine the module's repo
<cwd>/wget-contents/<module_name>.repo_url.cache: the repo value cache
This is to avoid unnecessary URL fetching and repo cloning.
* the ERROR_OUT_ON_FETCH_AND_CHECKOUT_FAILURE switch in script
The script must get the correct repo_url, fullsrc_rev and subpath for
each required module in go.mod to correctly generate the src_uri.inc and
relocation.inc files. If this process fails for any required module, this
script stop immediately, as I deliberately set ERROR_OUT_ON_FETCH_AND_CHECKOUT_FAILURE
to True in this script. The purpose is to encourage people to report
problems to meta-virt so that we can improve this script according to
these feedbacks. But this variable can set to False, then the script
only records the failed modules in self.modules_unhandled with reasons
added, people can modify the generated src_uri.inc and relocation.inc
to manually handle these unhandled modules if they are urgent to
add/upgrade some go mod recipes.

100
scripts/oe-go-mod-autogen.py
View File

@ -644,17 +644,111 @@ def main():
epilog=textwrap.dedent('''\
Overview:
=========
go-mod-oe is a tool for processing go dependencies to generate
dependencies suitable for OE fetcher consumption.
In particular, it creates a build structure suitable for
'-mod="vendor"' go builds. All dependencies are in the vendor/
directory, so no golang specific fetching or network access happens
during the build.
'-mod="vendor"' go builds. Once complete all go mod dependencies
are in the vendor/ directory, so no golang specific fetching or
network access happens during the build.
The files src_uri.inc, relocation.inc and modules.txt are generated
and suitable for recipe inclusion.
A recipe build can then use these files to leverage the git fetcher
and related functionality (mirrors, sstate, etc).
Note 1: --rev does not have to be a tag, if you want to track the tip of
a branch specify the latest git has on that branch, and it will
be used.
Note 2: This script does not generate an entire recipe, the way the
the outputs are used can be modified as required.
Note 3: if a go.mod has a bad revision, or needs to be manually updated
to fetch fixes: go.mod in the main repository (see the repos/
directory). If go.mod is edited, modules.txt also has to be
updated to match the revision information.
How to use in a recipe:
=======================
There are examples in meta-virtualization of recipes that use this
script and stragegy for builds: docker-compose, nerdcli, k3s
1) The recipe should set the master repository SRCREV details, and then include
the src_uri.inc file:
SRCREV_nerdcli = "e084a2df4a8861eb5f0b0d32df0643ef24b81093"
SRC_URI = "git://github.com/containerd/nerdctl.git;name=nerdcli;branch=master;protocol=https"
include src_uri.inc
This results in the SRC_URI being fully populated with the main
repository and all dependencies.
2) The recipe should either copy, or include the relocation.inc file. It sets
a variable "sites" that is a list of source locations (where the src_uri.inc
fetches) and destination in a vendor directory, it also has a do_compile:prepend()
that contains a loop which relocates the fetches into a vendor.copy directory.
It is expected to be processed as follows, before compilation starts:
# sets the "sites" variable and copies files
include relocation.inc
The do_compile:prepend, contains the following loop:
cd ${S}/src/import
# this moves all the fetches into the proper vendor structure
# expected for build
for s in ${sites}; do
site_dest=$(echo $s | cut -d: -f1)
site_source=$(echo $s | cut -d: -f2)
force_flag=$(echo $s | cut -d: -f3)
mkdir -p vendor.copy/$site_dest
if [ -n "$force_flag" ]; then
echo "[INFO] $site_dest: force copying .go files"
rm -rf vendor.copy/$site_dest
rsync -a --exclude='vendor/' --exclude='.git/' vendor.fetch/$site_source/ vendor.copy/$site_dest
else
[ -n "$(ls -A vendor.copy/$site_dest/*.go 2> /dev/null)" ] && { echo "[INFO] vendor.fetch/$site_source -> $site_dest: go copy skipped (files present)" ; true ; } || { echo "[INFO] $site_dest: copying .go files" ; rsync -a --exclude='vendor/' --exclude='.git/' vendor.fetch/$site_source/ vendor.copy/$site_dest ; }
fi
done
The main compile() function, should set the appropriate GO variables,
copy modules.txt and build the appripriate target:
# our copied .go files are to be used for the build
ln -sf vendor.copy vendor
3) The modules.txt file should be copied into the recipe directory, included
on the SRC_URI and copied into place after the relocation has been
processed.
# patches and config
SRC_URI += "file://0001-Makefile-allow-external-specification-of-build-setti.patch \\
file://modules.txt \
"
.....
cp ${WORKDIR}/modules.txt vendor/
Example: Updating the K3S recipe
================================
% cd meta-virtualization/recipe-containers/k3s/
# produces src_uri.inc, relocation.inc and modules.txt in the current directory
% ../../scripts/oe-go-mod-autogen.py --repo https://github.com/rancher/k3s.git --rev v1.27.5+k3s1
% cp modules.txt k3s/
... add and commit files.
'''))
parser.add_argument("--repo", help = "Repo for the recipe.", required=True)
parser.add_argument("--rev", help = "Revision for the recipe.", required=True)