devtool: upgrade: improve performance and show progress when adding files

When devtool upgrade is upgrading to a new version where the source is
fetched as an archive (e.g. a tarball), we create a single commit in the
git repository that is the upgrade from the old version to the new. We
do this by extracting the old source, committing it, deleting all files,
copying in the new files, running git add on each new/changed/deleted
file, and then committing the result. When a lot of files have changed
in an upgrade (such as QEMU 2.8.1.1 -> 2.10.0) the penultimate step of
running git add it can take quite a long time; in order to reduce this
and show some feedback to the user, run git add with batches of 100
files at once and also show a progress bar. In a local test with the
aforementioned QEMU upgrade it took the time down from over 7 minutes
down to about 13 seconds.

Fixes [YOCTO #11948].

(From OE-Core rev: 8b184f6c874b60324ee107af53853687173d3434)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

scripts/lib/devtool/upgrade.py

@@ -251,8 +251,15 @@ def _extract_new_source(newpv, srctree, no_patch, srcrev, srcbranch, branch, kee
         _copy_source_code(tmpsrctree, srctree)
 
         (stdout,_) = __run('git ls-files --modified --others --exclude-standard')
-        for f in stdout.splitlines():
-            __run('git add -A "%s"' % f)
+        filelist = stdout.splitlines()
+        pbar = bb.ui.knotty.BBProgress('Adding changed files', len(filelist))
+        pbar.start()
+        batchsize = 100
+        for i in range(0, len(filelist), batchsize):
+            batch = filelist[i:i+batchsize]
+            __run('git add -A %s' % ' '.join(['"%s"' % item for item in batch]))
+            pbar.update(i)
+        pbar.finish()
 
         useroptions = []
         oe.patch.GitApplyTree.gitCommandUserOptions(useroptions, d=rd)