oe-go-mod-autogen: add README and extended --help information

Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>

scripts/README-oe-go-mod-autogen.txt

@@ -0,0 +1,81 @@
+Description:
+
+    oe-go-mod-autogen.py is a helper script for go mod recipes. It is based
+    on intiial efforts to use only the git fetcher versus golangs module
+    infrastructure.
+    
+    Example:
+
+      cmd: <path_to>/meta-virtualization/scripts/oe-go-mod-autogen.py \
+           --repo https://github.com/docker/compose --rev v2.20.3
+      output: src_uri.inc, relocation.inc, modules.txt
+    
+    Copy these three generated files to replace the original ones,
+    then we only need update PV and SRCREV, and docker-compose is upgraded.
+
+    See --help for more explanations and examples.
+    
+    Below are some technical details.
+    
+    * get module's repo from module name
+    
+      This script checks the following two URLs to determine the module's repo.
+      1. https://<module_name_tweaked>?=go-get=1
+      2. https://pkg.go.dev/<module_name_tweaked>
+    
+      The module_name_tweaked is derived from module_name, with the last components
+      removed one by one. Let me use two examples to explain this.
+    
+      For module_name sigs.k8s.io/json, the sigs.k8s.io/json is first used as
+      module_name_tweaked for searching. And we can correctly get the repo URL, so
+      the search stops.
+    
+      For module_name github.com/k3s-io/etcd/api/v3, the following ones are used
+      as module_name_tweaked:
+      github.com/k3s-io/etcd/api/v3
+      github.com/k3s-io/etcd/api
+      github.com/k3s-io/etcd
+      And when searching 'github.com/k3s-io/etcd', we get the repo URL, so the search
+      stops.
+    
+    * determine the srcdir:destdir mapping in 'vendor' creation
+    
+      To correctly form the 'vendor' directory, the mapping is critical.
+      This script makes use of tag matching and path matching to determine
+      the subpath in the repo for the module.
+    
+    * avoid subpath being overriden by parent path
+    
+      We need to avoid subpath being overriden by parent path. This is needed
+      for both SRC_URI ordering in src_uri.inc and the sites mapping ordering
+      in relocation.inc. This script simply uses the length as the ordering key,
+      simply for the reason that if a path is a subpath of another path, it must
+      be longer.
+    
+    * the .git suffix is removed to sync with each other
+    
+      Unlike normal recipes, go mod recipe usually have many SRC_URIs. This script
+      remove the '.git' suffix from repo URL so that the repo URLs are in sync
+      with each.
+    
+    * basic directory hierarchy and caching mechanism
+    
+      <cwd>/repos: hold the repos downloaded and checked
+      <cwd>/wget-contents: hold the contents to determine the module's repo
+      <cwd>/wget-contents/<module_name>.repo_url.cache: the repo value cache
+      This is to avoid unnecessary URL fetching and repo cloning.
+    
+    * the ERROR_OUT_ON_FETCH_AND_CHECKOUT_FAILURE switch in script
+    
+      The script must get the correct repo_url, fullsrc_rev and subpath for
+      each required module in go.mod to correctly generate the src_uri.inc and
+      relocation.inc files. If this process fails for any required module, this
+      script stop immediately, as I deliberately set ERROR_OUT_ON_FETCH_AND_CHECKOUT_FAILURE
+      to True in this script. The purpose is to encourage people to report
+      problems to meta-virt so that we can improve this script according to
+      these feedbacks. But this variable can set to False, then the script
+      only records the failed modules in self.modules_unhandled with reasons
+      added, people can modify the generated src_uri.inc and relocation.inc
+      to manually handle these unhandled modules if they are urgent to
+      add/upgrade some go mod recipes.
+    

scripts/oe-go-mod-autogen.py

@@ -644,17 +644,111 @@ def main():
         epilog=textwrap.dedent('''\
 
         Overview:
+        =========
 
           go-mod-oe is a tool for processing go dependencies to generate
           dependencies suitable for OE fetcher consumption.
 
           In particular, it creates a build structure suitable for
-          '-mod="vendor"' go builds. All dependencies are in the vendor/
+          '-mod="vendor"' go builds. Once complete all go mod dependencies
-          directory, so no golang specific fetching or network access happens
+          are in the vendor/ directory, so no golang specific fetching or
-          during the build.
+          network access happens during the build.
 
           The files src_uri.inc, relocation.inc and modules.txt are generated
           and suitable for recipe inclusion.
+
+          A recipe build can then use these files to leverage the git fetcher
+          and related functionality (mirrors, sstate, etc).
+
+          Note 1: --rev does not have to be a tag, if you want to track the tip of
+                  a branch specify the latest git has on that branch, and it will
+                  be used.
+
+          Note 2: This script does not generate an entire recipe, the way the
+                  the outputs are used can be modified as required.
+
+          Note 3: if a go.mod has a bad revision, or needs to be manually updated
+                  to fetch fixes: go.mod in the main repository (see the repos/
+                  directory). If go.mod is edited, modules.txt also has to be
+                  updated to match the revision information.
+
+        How to use in a recipe:
+        =======================
+
+        There are examples in meta-virtualization of recipes that use this
+        script and stragegy for builds: docker-compose, nerdcli, k3s
+
+          1) The recipe should set the master repository SRCREV details, and then include
+             the src_uri.inc file:
+
+               SRCREV_nerdcli = "e084a2df4a8861eb5f0b0d32df0643ef24b81093"
+               SRC_URI = "git://github.com/containerd/nerdctl.git;name=nerdcli;branch=master;protocol=https"
+
+               include src_uri.inc
+
+            This results in the SRC_URI being fully populated with the main
+            repository and all dependencies.
+
+          2) The recipe should either copy, or include the relocation.inc file. It sets
+             a variable "sites" that is a list of source locations (where the src_uri.inc
+             fetches) and destination in a vendor directory, it also has a do_compile:prepend()
+             that contains a loop which relocates the fetches into a vendor.copy directory.
+
+             It is expected to be processed as follows, before compilation starts:
+
+                # sets the "sites" variable and copies files
+                include relocation.inc
+
+             The do_compile:prepend, contains the following loop:
+
+               cd ${S}/src/import
+               # this moves all the fetches into the proper vendor structure
+               # expected for build
+               for s in ${sites}; do
+                   site_dest=$(echo $s | cut -d: -f1)
+                   site_source=$(echo $s | cut -d: -f2)
+                   force_flag=$(echo $s | cut -d: -f3)
+                   mkdir -p vendor.copy/$site_dest
+                   if [ -n "$force_flag" ]; then
+                       echo "[INFO] $site_dest: force copying .go files"
+                       rm -rf vendor.copy/$site_dest
+                       rsync -a --exclude='vendor/' --exclude='.git/' vendor.fetch/$site_source/ vendor.copy/$site_dest
+                   else
+                       [ -n "$(ls -A vendor.copy/$site_dest/*.go 2> /dev/null)" ] && { echo "[INFO] vendor.fetch/$site_source -> $site_dest: go copy skipped (files present)" ; true ; } || { echo "[INFO] $site_dest: copying .go files" ; rsync -a --exclude='vendor/' --exclude='.git/' vendor.fetch/$site_source/ vendor.copy/$site_dest ; }
+                   fi
+               done
+
+            The main compile() function, should set the appropriate GO variables,
+            copy modules.txt and build the appripriate target:
+
+               # our copied .go files are to be used for the build
+               ln -sf vendor.copy vendor
+
+           3) The modules.txt file should be copied into the recipe directory, included
+              on the SRC_URI and copied into place after the relocation has been
+              processed.
+
+               # patches and config
+               SRC_URI += "file://0001-Makefile-allow-external-specification-of-build-setti.patch \\
+                           file://modules.txt \
+                          "
+
+            .....
+
+               cp ${WORKDIR}/modules.txt vendor/
+
+        Example: Updating the K3S recipe
+        ================================
+
+          % cd meta-virtualization/recipe-containers/k3s/
+          # produces src_uri.inc, relocation.inc and modules.txt in the current directory
+          % ../../scripts/oe-go-mod-autogen.py --repo https://github.com/rancher/k3s.git --rev v1.27.5+k3s1
+
+          % cp modules.txt k3s/
+
+          ... add and commit files.
+
+
         '''))
     parser.add_argument("--repo", help = "Repo for the recipe.", required=True)
     parser.add_argument("--rev", help = "Revision for the recipe.", required=True)