meta-virtualization/scripts/QUICKSTART-oe-go-mod-vcs.md

Quickstart: Go Module VCS Build System for Yocto/BitBake

This guide covers how to create and maintain Go recipes using the go-mod-vcs system, which provides reproducible, offline Go builds by fetching dependencies directly from their git repositories.

Overview

The go-mod-vcs system replaces vendor directories with a build-time module cache constructed from git repositories. This provides:

• Reproducible builds - Every module comes from a verified git commit
• Offline builds - After do_fetch, no network access is required
• Auditable dependencies - Every dependency is traceable to a specific git commit
• Smaller recipes - No need to maintain large vendor directories in source trees

How It Works

┌─────────────────────────────────────────────────────────────────────┐
│                         Your Recipe                                 │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────────┐ │
│  │ myapp_git.bb    │  │ go-mod-git.inc  │  │ go-mod-cache.inc    │ │
│  │ (your code)     │  │ (git fetches)   │  │ (module metadata)   │ │
│  └─────────────────┘  └─────────────────┘  └─────────────────────┘ │
└───────────────────────────────┬─────────────────────────────────────┘
                                │
        ┌───────────────────────┼───────────────────────┐
        │                       │                       │
        ▼                       ▼                       ▼
   ┌─────────┐           ┌─────────────┐         ┌───────────────┐
   │do_fetch │           │do_create_   │         │  do_compile   │
   │         │──────────▶│module_cache │────────▶│               │
   │(git)    │           │(build cache)│         │(go build)     │
   └─────────┘           └─────────────┘         └───────────────┘

1. do_fetch - BitBake fetches all git repositories listed in go-mod-git.inc
2. do_create_module_cache - The go-mod-vcs class builds a Go module cache from git checkouts
3. do_compile - Go builds offline using the pre-populated module cache

---

Converting an Existing Recipe

Step 1: Add Discovery Configuration

Add these lines to your recipe (before inherit go):

# go-mod-discovery configuration
GO_MOD_DISCOVERY_BUILD_TARGET = "./cmd/..."      # Your build target
GO_MOD_DISCOVERY_GIT_REPO = "https://github.com/org/repo.git"
GO_MOD_DISCOVERY_GIT_REF = "${SRCREV}"

inherit go-mod-discovery

Step 2: Include the Generated Files

Add includes after your SRC_URI:

SRC_URI = "git://github.com/org/repo;branch=main;protocol=https;destsuffix=${GO_SRCURI_DESTSUFFIX}"

include go-mod-git.inc
include go-mod-cache.inc

Step 3: Create Placeholder Files

Create empty placeholder files (they'll be generated):

cd recipes-containers/myapp/
touch go-mod-git.inc go-mod-cache.inc

Step 4: Run Discovery

bitbake myapp -c discover_and_generate

This will:

1. Build your project with network access to discover dependencies
2. Extract module metadata from the Go module cache
3. Generate go-mod-git.inc and go-mod-cache.inc files

Step 5: Update do_compile()

Ensure your do_compile() doesn't set conflicting environment variables. The go-mod-vcs.bbclass automatically sets:

• GOMODCACHE - Points to the built module cache
• GOPROXY=off - Enforces offline build
• GOSUMDB=off - Disables checksum database
• GOTOOLCHAIN=local - Uses the native Go toolchain

A minimal do_compile():

do_compile() {
    cd ${S}/src/import

    export GOPATH="${S}/src/import/.gopath:${STAGING_DIR_TARGET}/${prefix}/local/go"
    export CGO_ENABLED="1"

    ${GO} build -trimpath ./cmd/...
}

Step 6: Build and Test

bitbake myapp

---

Creating a New Recipe

Minimal Recipe Template

SUMMARY = "My Go Application"
HOMEPAGE = "https://github.com/org/myapp"

SRCREV = "abc123def456..."
SRC_URI = "git://github.com/org/myapp;branch=main;protocol=https;destsuffix=${GO_SRCURI_DESTSUFFIX}"

include go-mod-git.inc
include go-mod-cache.inc

LICENSE = "Apache-2.0"
LIC_FILES_CHKSUM = "file://src/import/LICENSE;md5=..."

GO_IMPORT = "import"
PV = "v1.0.0+git"

# go-mod-discovery configuration
GO_MOD_DISCOVERY_BUILD_TARGET = "./cmd/..."
GO_MOD_DISCOVERY_GIT_REPO = "https://github.com/org/myapp.git"
GO_MOD_DISCOVERY_GIT_REF = "${SRCREV}"

inherit go goarch
inherit go-mod-discovery

do_compile() {
    cd ${S}/src/import

    export GOPATH="${S}/src/import/.gopath:${STAGING_DIR_TARGET}/${prefix}/local/go"
    export CGO_ENABLED="1"

    ${GO} build -trimpath -o ${B}/myapp ./cmd/myapp
}

do_install() {
    install -d ${D}${bindir}
    install -m 755 ${B}/myapp ${D}${bindir}/
}

Generate Dependencies

# Create placeholder files
touch go-mod-git.inc go-mod-cache.inc

# Run discovery
bitbake myapp -c discover_and_generate

# Build
bitbake myapp

---

Updating a Recipe to a New Version

Quick Update (Same Repository)

1. Update the SRCREV in your recipe:

   SRCREV = "new_commit_hash_here"
   

2. Re-run discovery:

   bitbake myapp -c discover_and_generate
   

3. Build:

   bitbake myapp
   

Full Workflow Example

# 1. Find the new commit/tag
git ls-remote https://github.com/org/myapp refs/tags/v2.0.0

# 2. Update SRCREV in recipe
# SRCREV = "abc123..."

# 3. Clean old discovery cache (optional, recommended for major updates)
bitbake myapp -c clean_discovery

# 4. Run discovery and generation
bitbake myapp -c discover_and_generate

# 5. Review generated files
git diff recipes-containers/myapp/go-mod-*.inc

# 6. Build and test
bitbake myapp

# 7. Commit changes
git add recipes-containers/myapp/
git commit -m "myapp: update to v2.0.0"

---

Discovery Tasks Reference

Task	Purpose	Network?
discover_modules	Build project and download modules from proxy	Yes
extract_modules	Extract metadata from cache to JSON	No
generate_modules	Generate .inc files from metadata	No
discover_and_generate	All three steps in sequence	Yes
show_upgrade_commands	Print command lines without running	No
clean_discovery	Remove the discovery cache	No

Step-by-Step vs All-in-One

All-in-one (recommended for most cases):

bitbake myapp -c discover_and_generate

Step-by-step (useful for debugging):

bitbake myapp -c discover_modules    # Download modules
bitbake myapp -c extract_modules     # Extract to JSON
bitbake myapp -c generate_modules    # Generate .inc files

---

Configuration Variables

Variable	Default	Description
GO_MOD_DISCOVERY_BUILD_TARGET	(required)	Go build target (e.g., ./cmd/...)
GO_MOD_DISCOVERY_GIT_REPO	(required for generate)	Git repository URL
GO_MOD_DISCOVERY_GIT_REF	${SRCREV}	Git commit/tag
GO_MOD_DISCOVERY_SRCDIR	${S}/src/import	Directory containing go.mod
GO_MOD_DISCOVERY_BUILD_TAGS	${TAGS}	Go build tags
GO_MOD_DISCOVERY_RECIPEDIR	${FILE_DIRNAME}	Output directory for .inc files

---

Troubleshooting

"module lookup disabled by GOPROXY=off"

This error during do_compile means a module is missing from the cache.

Fix:

# Re-run discovery to find missing modules
bitbake myapp -c discover_and_generate
bitbake myapp

Discovery Cache Location

The discovery cache persists in ${TOPDIR}/go-mod-discovery/${PN}/${PV}/ and survives bitbake -c cleanall. To fully reset:

bitbake myapp -c clean_discovery
bitbake myapp -c discover_and_generate

Viewing Generated Commands

To see what commands would be run without executing them:

bitbake myapp -c show_upgrade_commands

Build Fails After SRCREV Update

If changing SRCREV causes sstate errors:

# Clean sstate for the recipe
bitbake myapp -c cleansstate

# Re-run discovery
bitbake myapp -c discover_and_generate

# Build
bitbake myapp

Multiple Source Repositories

For recipes with multiple git sources, use named SRCREVs:

SRCREV_myapp = "abc123..."
SRCREV_plugins = "def456..."
SRCREV_FORMAT = "myapp_plugins"

SRC_URI = "\
    git://github.com/org/myapp;name=myapp;branch=main;protocol=https;destsuffix=${GO_SRCURI_DESTSUFFIX} \
    git://github.com/org/plugins;name=plugins;branch=main;protocol=https;destsuffix=${GO_SRCURI_DESTSUFFIX}/plugins \
"

GO_MOD_DISCOVERY_GIT_REF = "${SRCREV_myapp}"

---

Example Recipes

Simple Recipe (rootlesskit)

SRCREV_rootless = "8059d35092db167ec53cae95fb6aa37fc577060c"
SRCREV_FORMAT = "rootless"

SRC_URI = "git://github.com/rootless-containers/rootlesskit;name=rootless;branch=master;protocol=https;destsuffix=${GO_SRCURI_DESTSUFFIX}"

include go-mod-git.inc
include go-mod-cache.inc

GO_MOD_DISCOVERY_BUILD_TARGET = "./cmd/..."
GO_MOD_DISCOVERY_GIT_REPO = "https://github.com/rootless-containers/rootlesskit.git"
GO_MOD_DISCOVERY_GIT_REF = "${SRCREV_rootless}"

inherit go goarch go-mod-discovery

Complex Recipe (k3s with build tags)

TAGS = "static_build netcgo osusergo providerless"

GO_MOD_DISCOVERY_BUILD_TARGET = "./cmd/server/main.go"
GO_MOD_DISCOVERY_GIT_REPO = "https://github.com/rancher/k3s.git"
GO_MOD_DISCOVERY_GIT_REF = "${SRCREV_k3s}"

inherit go goarch go-mod-discovery

---

Files Generated

go-mod-git.inc

Contains SRC_URI entries for each module's git repository:

SRC_URI += "git://github.com/spf13/cobra;protocol=https;nobranch=1;rev=e94f6d0...;name=git_41456771_1;destsuffix=vcs_cache/2d91d6bc..."

go-mod-cache.inc

Contains module metadata and inherits the build class:

inherit go-mod-vcs

GO_MODULE_CACHE_DATA = '[{"module":"github.com/spf13/cobra","version":"v1.8.1","vcs_hash":"2d91d6bc...","timestamp":"2024-06-01T10:31:11Z","subdir":""},...]'

---

Hybrid Mode: Mixing gomod:// and git:// Fetchers

For large projects like k3s with hundreds of modules, the VCS-only approach (all git://) can be slow due to the large number of git clones. Hybrid mode provides a faster alternative by using:

• gomod:// - Fast proxy.golang.org downloads for most modules
• git:// - VCS provenance for selected important modules (e.g., containerd, k8s.io)

Benefits

Mode	Fetch Speed	VCS Provenance	Use Case
VCS (git:// only)	Slower	Full	Security-critical, audit requirements
Hybrid (gomod:// + git://)	Faster	Selective	Development, CI, most builds

Step 1: Build in VCS Mode First

Ensure your recipe works in VCS mode before converting:

bitbake myapp -c discover_and_generate
bitbake myapp

Step 2: Run Recommendations

After a successful VCS build, analyze which modules to keep as git:// vs convert to gomod://:

bitbake myapp -c go_mod_recommend

This outputs size-based recommendations and suggests prefixes to keep as git://.

Step 3: Generate Hybrid Files

Use the hybrid conversion script with suggested prefixes:

python3 ./meta-virtualization/scripts/oe-go-mod-fetcher-hybrid.py \
    --recipedir ./meta-virtualization/recipes-containers/myapp/ \
    --workdir ${WORKDIR} \
    --git "github.com/containerd,k8s.io,sigs.k8s.io"

This generates three new files:

• go-mod-hybrid-gomod.inc - SRC_URI entries for gomod:// fetcher (with inline checksums)
• go-mod-hybrid-git.inc - SRC_URI entries for git:// fetcher (VCS provenance)
• go-mod-hybrid-cache.inc - Module metadata for the git:// modules

Step 4: Configure Recipe for Mode Switching

Update your recipe to allow switching between modes:

# GO_MOD_FETCH_MODE: "vcs" (all git://) or "hybrid" (gomod:// + git://)
GO_MOD_FETCH_MODE ?= "vcs"

# VCS mode: all modules via git://
include ${@ "go-mod-git.inc" if d.getVar("GO_MOD_FETCH_MODE") == "vcs" else ""}
include ${@ "go-mod-cache.inc" if d.getVar("GO_MOD_FETCH_MODE") == "vcs" else ""}

# Hybrid mode: gomod:// for most, git:// for selected
include ${@ "go-mod-hybrid-gomod.inc" if d.getVar("GO_MOD_FETCH_MODE") == "hybrid" else ""}
include ${@ "go-mod-hybrid-git.inc" if d.getVar("GO_MOD_FETCH_MODE") == "hybrid" else ""}
include ${@ "go-mod-hybrid-cache.inc" if d.getVar("GO_MOD_FETCH_MODE") == "hybrid" else ""}

Step 5: Switch Modes

Switch between modes in local.conf:

# Use hybrid mode (faster)
GO_MOD_FETCH_MODE = "hybrid"

# Or use VCS mode (full provenance)
GO_MOD_FETCH_MODE = "vcs"

Then rebuild:

bitbake myapp

Hybrid Script Options

Option	Description
--recipedir	Recipe directory containing go-mod-git.inc and go-mod-cache.inc
--workdir	BitBake workdir for size calculations (optional)
--git "prefixes"	Comma-separated prefixes to keep as git:// (everything else becomes gomod://)
--gomod "prefixes"	Comma-separated prefixes to convert to gomod:// (everything else stays git://)
--no-checksums	Skip fetching SHA256 checksums (not recommended)
--list	List all modules with sizes
--recommend	Show size-based conversion recommendations

Example: k3s Hybrid Conversion

# 1. Ensure VCS mode works first
bitbake k3s

# 2. Get recommendations
bitbake k3s -c go_mod_recommend

# 3. Convert with recommended prefixes (keep containerd, k8s.io as git://)
python3 ./meta-virtualization/scripts/oe-go-mod-fetcher-hybrid.py \
    --recipedir ./meta-virtualization/recipes-containers/k3s/ \
    --git "github.com/containerd,k8s.io,sigs.k8s.io,github.com/rancher"

# 4. Enable hybrid mode
echo 'GO_MOD_FETCH_MODE = "hybrid"' >> conf/local.conf

# 5. Build in hybrid mode
bitbake k3s

Troubleshooting Hybrid Builds

"Permission denied" during do_unpack

Go's module cache creates read-only files. The go-mod-vcs.bbclass includes automatic permission fixes, but if you hit this on an existing build:

chmod -R u+w ${WORKDIR}/sources/
bitbake myapp

BitBake parse errors with special characters

Module paths like git.sr.ht/~sbinet/gg contain characters (~) that can cause BitBake parse errors in variable flag names. The hybrid script uses inline checksums (sha256sum=... in the SRC_URI) to avoid this issue.

---

Advanced: Manual Script Invocation

For cases where BitBake isn't available or you need more control:

# Direct generation from git (no BitBake needed)
python3 ./meta-virtualization/scripts/oe-go-mod-fetcher.py \
    --git-repo https://github.com/org/myapp.git \
    --git-ref abc123... \
    --recipedir ./meta-virtualization/recipes-containers/myapp/

# Using existing discovery cache
python3 ./meta-virtualization/scripts/oe-go-mod-fetcher.py \
    --discovered-modules ${TOPDIR}/go-mod-discovery/myapp/v1.0.0/modules.json \
    --git-repo https://github.com/org/myapp.git \
    --git-ref abc123... \
    --recipedir ./meta-virtualization/recipes-containers/myapp/

---

Getting Help

• Show available commands: bitbake myapp -c show_upgrade_commands
• Architecture details: See scripts/ARCHITECTURE.md
• Report issues: Check the generated files and error messages from discovery