poky/scripts/oe-setup-layers
Chuck Wolber b57d4de337 scripts/oe-setup-layers: Make efficiently idempotent
The effect of subsequent setup-layers executions is now either a NOOP
or the minimal set of changes required to ensure layers precisely match
the JSON configuration.

This change allows setup-layers to be incorporated into a team's
configuration management strategy. In particular, the configuration
JSON manages a "pinning policy" that documents the oversight of sources
of change (a requirement for embedded development in highly regulated
industries).

One model for this strategy would work as follows. Team level policy is
developed to regularly review upstream commits that occur between the
current upstream HEAD and the previously pinned revision. The JSON
configuration is periodically updated after a review, test, and approval
process. In the rare instance that an upstream change is considered
problematic, the bbappend mechanism can be used to make relevant
changes in the team's project repository. This approach also requires
that team developers regularly run the project repository copy of
setup-layers. This is most easily accomplished by including setup-layers
in a wrapper script that all team developers use to interact with the
bitbake tool suite (e.g. "bb bitbake foo-image"). Project level policy
and oversight is effectively "contained" within this wrapper script,
thereby reducing a significant source of human error.

Left unstated, but acknowledged here, are a number of nuances required
to successfully implement the above strategy.  The details are out of
scope for this explanation. What should be clear is that a larger
configuration management strategy can now benefit from the utility
provided by setup-layers.

Note: Neither the above configuration management strategy example nor
the change itself is intended to alter the original intent to use
"bitbake-layers create-layers-setup destdir" to keep pace with upstream
activity for those who wish to use it that way.

(From OE-Core rev: da2e01cacd98715318a5307fe0618dbca0cf1fe7)

Signed-off-by: Chuck Wolber <chuck.wolber@boeing.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-18 16:42:28 +00:00

5.0 KiB
Executable File

#!/usr/bin/env python3

Copyright OpenEmbedded Contributors

SPDX-License-Identifier: MIT

This file was copied from poky(or oe-core)/scripts/oe-setup-layers by running

bitbake-layers create-layers-setup destdir

It is recommended that you do not modify this file directly, but rather re-run the above command to get the freshest upstream copy.

This script is idempotent. Subsequent runs only change what is necessary to

ensure your layers match your configuration.

import argparse import json import os import subprocess

def _is_layer_git_repo(layerdir): git_dir = os.path.join(layerdir, ".git") if not os.access(git_dir, os.R_OK): return False try: return subprocess.check_output("git -C %s rev-parse --is-inside-git-dir" % git_dir, shell=True, stderr=subprocess.DEVNULL) except subprocess.CalledProcessError: return False

def _is_layer_at_rev(layerdir, rev): try: curr_rev = subprocess.check_output("git -C %s rev-parse HEAD" % layerdir, shell=True, stderr=subprocess.DEVNULL) if curr_rev.strip().decode("utf-8") == rev: return True except subprocess.CalledProcessError: pass return False

def _is_layer_at_remote_uri(layerdir, remote, uri): try: curr_uri = subprocess.check_output("git -C %s remote get-url %s" % (layerdir, remote), shell=True, stderr=subprocess.DEVNULL) if curr_uri.strip().decode("utf-8") == uri: return True except subprocess.CalledProcessError: pass return False

def _do_checkout(args, json): layers = json['sources'] for l_name in layers: l_data = layers[l_name] layerdir = os.path.abspath(os.path.join(args['destdir'], l_data['path']))

    if 'contains_this_file' in l_data.keys():
        force_arg = 'force_bootstraplayer_checkout'
        if not args[force_arg]:
            print('Note: not checking out source {layer}, use {layerflag} to override.'.format(layer=l_name, layerflag='--force-bootstraplayer-checkout'))
            continue
    l_remote = l_data['git-remote']
    rev = l_remote['rev']
    desc = l_remote['describe']
    if not desc:
        desc = rev[:10]
    branch = l_remote['branch']
    remotes = l_remote['remotes']

    print('\nSetting up source {}, revision {}, branch {}'.format(l_name, desc, branch))
    if not _is_layer_git_repo(layerdir):
        cmd = 'git init -q {}'.format(layerdir)
        print("Running '{}'".format(cmd))
        subprocess.check_output(cmd, shell=True)

    for remote in remotes:
        if not _is_layer_at_remote_uri(layerdir, remote, remotes[remote]['uri']):
            cmd = "git remote remove {} > /dev/null 2>&1; git remote add {} {}".format(remote, remote, remotes[remote]['uri'])
            print("Running '{}' in {}".format(cmd, layerdir))
            subprocess.check_output(cmd, shell=True, cwd=layerdir)

            cmd = "git fetch -q {} || true".format(remote)
            print("Running '{}' in {}".format(cmd, layerdir))
            subprocess.check_output(cmd, shell=True, cwd=layerdir)

    if not _is_layer_at_rev(layerdir, rev):
        cmd = "git fetch -q --all || true"
        print("Running '{}' in {}".format(cmd, layerdir))
        subprocess.check_output(cmd, shell=True, cwd=layerdir)

        cmd = 'git checkout -q {}'.format(rev)
        print("Running '{}' in {}".format(cmd, layerdir))
        subprocess.check_output(cmd, shell=True, cwd=layerdir)

parser = argparse.ArgumentParser(description="A self contained python script that fetches all the needed layers and sets them to correct revisions using data in a json format from a separate file. The json data can be created from an active build directory with 'bitbake-layers create-layers-setup destdir' and there's a sample file and a schema in meta/files/")

parser.add_argument('--force-bootstraplayer-checkout', action='store_true', help='Force the checkout of the layer containing this file (by default it is presumed that as this script is in it, the layer is already in place).')

try: defaultdest = os.path.dirname(subprocess.check_output('git rev-parse --show-toplevel', universal_newlines=True, shell=True, cwd=os.path.dirname(file))) except subprocess.CalledProcessError as e: defaultdest = os.path.abspath(".")

parser.add_argument('--destdir', default=defaultdest, help='Where to check out the layers (default is {defaultdest}).'.format(defaultdest=defaultdest)) parser.add_argument('--jsondata', default=file+".json", help='File containing the layer data in json format (default is {defaultjson}).'.format(defaultjson=file+".json"))

args = parser.parse_args()

with open(args.jsondata) as f: json = json.load(f)

supported_versions = ["1.0"] if json["version"] not in supported_versions: raise Exception("File {} has version {}, which is not in supported versions: {}".format(args.jsondata, json["version"], supported_versions))

_do_checkout(vars(args), json)