* guix/scripts/offload.scm (remove-gc-root): Rename to...
(remove-gc-roots): ... this.
[builder]: Use 'scandir' and remove all the files starting with
%GC-ROOT-FILE.
(transfer-and-offload): Adjust to renaming; remove
'false-if-exception' wraps.
Partially fixes <http://bugs.gnu.org/17212>.
* guix/utils.scm (errno): Move definition of 'bv' outside of the
procedure. Use 'bytevector-s32-native-ref' or
'bytevector-s64-native-ref' instead of 'bytevector-sint-ref'.
Problem reported by John Darrington <john@darrington.wattle.id.au>.
* guix/nar.scm (write-file): Filter out "." and ".." from the result of
'scandir'. Previously we did this by passing a suitable predicate.
* build-aux/hydra/demo-os.scm: New file.
* Makefile.am (EXTRA_DIST): Add it.
* build-aux/hydra/gnu-system.scm (qemu-jobs): New procedure.
(hydra-jobs): Use it.
* guix/scripts/system.scm (read-operating-system): Export.
Fixes the creation of single-package profiles, reported by Ludovic Courtès.
* guix/build/union.scm (union-build): Add new internal procedure
'union-of-directories' that always creates a directory, containing the code
previously used only to merge multiple directories. Call it from the
multiple-directory case in 'union' and from the top-level 'union-build'.
* guix/pki.scm (write-acl): New procedure.
(ensure-acl): Use it. Fixes a regression introduced in 39831f1,
whereby 'ensure-acl' would yield a wrong-type-arg error.
* guix/scripts/archive.scm (authorize-key): Use 'write-acl'.
Before that, there was a small time window during which the GC could
wipe the .drv (before 'guix build' has been called), or the build
result (before 'retrieve-files' has started.)
* guix/scripts/offload.scm (remote-pipe): Add #:quote? parameter and
honor it.
(%gc-root-file): New variable.
(register-gc-root, remove-gc-root): New procedures.
(offload): Adjust comment. Run 'guix build' with '-r %GC-ROOT-FILE'.
(transfer-and-offload): Call 'register-gc-root' before
sending (derivation-file-name DRV). Call 'remove-gc-root' after the
call to 'offload' or 'retrieve-files'.
(send-files): Call 'remote-pipe' with #:quote? #f.
(retrieve-files): Likewise.
* guix/build/union.scm: Rewrite; only 'file=?' remains unchanged. Remove
'tree-union' and 'delete-duplicate-leaves' exports. Merge inputs in a
breadth-first fashion. Follow symlinks for purposes of making decisions
about the merge.
* tests/union.scm: Remove tests of 'tree-union' and 'delete-duplicate-leaves'.
* guix/scripts/package.scm (search-path-environment-variables)[manifest-entry->package]:
Use 'find-best-packages-by-name' instead of 'find-packages-by-name'.
On a profile with 140 packages, this reduces execution time of this
procedure from 5.8 seconds to 2.9 seconds (50% improvement.)
* guix/scripts/package.scm (maybe-register-gc-root): New procedure.
* tests/guix-package.sh (profile): Grep the output of "guix gc
--list-live" in a couple of places.
* guix/pki.scm (acl-entry-sexp, acl-sexp): Remove.
(public-keys->acl, current-acl): Return a native sexp.
(acl->public-keys, authorized-key?): Expect ACL to be a native sexp.
* guix/scripts/archive.scm (authorize-key): Convert ACL to
canonical-sexp when writing it.
* guix/scripts/substitute-binary.scm (guix-substitute-binary) <--query>:
Cache the result of (current-acl); pass it to 'valid-narinfo?' calls.
This saves 12% wall-clock time for "guix build emacs -n".
* guix/nar.scm (restore-file-set)[assert-valid-signature]: Rewrite in
terms of 'signature-case'.
* guix/scripts/substitute-binary.scm (narinfo-signature->canonical-sexp):
Call 'leave' instead of 'raise' when SIGNATURE is invalid.
(&nar-signature-error, &nar-invalid-hash-error): Remove.
(assert-valid-signature): Add 'narinfo' parameter; remove 'port'.
Rewrite in terms of 'signature-case' and 'leave'. Mention NARINFO's
URI in error messages. Adjust caller.
(narinfo-sha256): New procedure.
(assert-valid-narinfo): Use it.
(valid-narinfo?): Rewrite using 'narinfo-sha256' and
'signature-case'.
* tests/substitute-binary.scm (assert-valid-signature,
test-error-condition): Remove.
("corrupt signature data", "unauthorized public key", "invalid
signature"): Remove.
* guix/scripts/substitute-binary.scm (narinfo-signature->canonical-sexp):
Catch 'gcry-error' around 'string->canonical-sexp' call, and re-raise
as a SRFI-35 &message and &nar-signature-error.
(narinfo-maker): Handle when SIGNATURE is #f or an invalid canonical
sexp.
(&nar-signature-error, &nar-invalid-hash-error): New variables.
(assert-valid-signature): Use them. Expect 'signature' to be a
canonical sexp.
(read-narinfo): Remove authentication and authorization checks.
(%signature-line-rx): New variable.
(assert-valid-narinfo, valid-narinfo?): New procedures.
(guix-substitute-binary): Wrap body in 'with-error-handling'.
[valid?]: New procedure.
<--query>: Show only store items of narinfos that match
'valid-narinfo?'.
<--substitute>: Call 'assert-valid-narinfo'.
* tests/substitute-binary.scm (test-error*): Use 'test-equal'.
(%keypair): Remove.
(%public-key, %private-key): Load from signing-key.{pub,sec}.
(signature-body): Add #:public-key parameter.
(call-with-narinfo): New procedure.
(with-narinfo): New macro.
("corrupt signature data", "unauthorized public key", "invalid
signature"): Make the first argument to 'assert-valid-signature' a
canonical sexp.
("invalid hash", "valid read-narinfo", "valid write-narinfo"):
Remove.
("query narinfo with invalid hash", "query narinfo signed with
authorized key", "query narinfo signed with unauthorized key",
"substitute, invalid hash", "substitute, unauthorized key"): New
tests.
* guix/scripts/substitute-binary.scm (<narinfo>)[uri-base]: New field.
(narinfo-maker): Pass CACHE-URL as the 'uri-base' value.
(string->narinfo): Add 'cache-uri' parameter.
(lookup-narinfo)[cache-entry]: Switch to version 1. Add 'cache-uri'
field. Adjust body accordingly.
(remove-expired-cached-narinfos): Switch to version 1 by default.
* guix/scripts/substitute-binary.scm (<narinfo>): Add the 'signature'
and 'contents' fields.
(narinfo-signature->canonical-sexp): New function.
(narinfo-maker): Add the 'signature' argument and use it.
(assert-valid-signature): New function.
(read-narinfo): Support the Signature field.
(write-narinfo): Use 'narinfo-contents'.
(%allow-unauthenticated-substitutes?): New variable.
* guix/base64.scm, tests/base64.scm, tests/substitute-binary.scm: New files.
* Makefile.am (SCM_TESTS): Add tests/base64.scm and
tests/substitute-binary.scm.
(MODULES): Add guix/base64.scm.
* test-env.in: Set 'GUIX_ALLOW_UNAUTHENTICATED_SUBSTITUTES'.
* guix/scripts/offload.scm (transfer-and-offload): Use 'upload' lock
instead of 'bandwidth' around 'send-files' calls, and 'download' lock
around 'retrieve-files' call.
* guix/scripts/offload.scm (remote-pipe): Remove '-z' lsh command line
argument. This makes transfers almost an order of magnitude slower.
OpenSSH's ssh(1) man page notes: "Compression is desirable on modem lines
and other slow connections, but will only slow down things on fast
networks." See also
<http://www.spikelab.org/blog/transfer-largedata-scp-tarssh-tarnc-compared.html>.
* guix/build/download.scm (http-fetch)[post-2.0.7?]: Use
'string->number' and numeric comparison. This fixes version
comparison with "2.0.10" and subsequent 2.0 releases.
* guix/utils.scm (call-with-decompressed-port,
call-with-compressed-output-port): New procedures.
* tests/utils.scm ("compressed-output-port + decompressed-port"):
Rewrite to use them.
* guix/scripts/authenticate.scm (guix-authenticate): Add clauses
for ("rsautl" "-sign" "-inkey" key) and ("rsautl" "-verify" "-inkey" _
"-pubin").
* tests/guix-authenticate.sh (hash): Add test using -sign and -verify in
a pipeline.
* guix/scripts/authenticate.scm (read-canonical-sexp): Change to expect
a port instead of a file name.
(read-hash-data): Likewise.
(sign-with-key, validate-signature): New procedures.
(guix-authenticate): Rewrite in terms of these two procedures.
* guix/utils.scm (decompressed-port, compressed-port): New procedures.
* guix/scripts/substitute-binary.scm (decompressed-port): Remove.
(guix-substitute-binary): Pass a symbol or #f as the first argument to
'decompress-port'.
* tests/utils.scm ("compressed-port, decompressed-port, non-file"): New
test.
* guix/utils.scm (filtered-port): Make sure the 'execl' child process
always exits, and does (primitive-_exit 1) upon execution failure.
Use 'primitive-_exit' in the 'dump-port' child process.
* tests/utils.scm ("filtered-port, does not exist"): New test.
* guix/pk-crypto.scm (gcrypt-version): New procedure.
* guix/scripts/archive.scm (%key-generation-parameters): New variable.
(%options) <generate-key>: Use it.
* tests/pk-crypto.scm ("sign + verify, Ed25519"): Skip if using gcrypt < 1.6.0.
* guix/scripts/offload.scm (with-error-to-port): New macro.
(remote-pipe): Add #:error-port parameter. Use 'with-error-to-port'
around 'open-pipe*' call.
(build-log-port): New procedure.
(offload): Change #:log-port to default to (build-log-port). Call
'remote-pipe' with #:error-port LOG-PORT.
* guix/scripts/substitute-binary.scm (fetch): Add #:quiet-404?
parameter. Upon &http-get-error, re-raise C if the QUIET-404? is
true and the code is 404.
(fetch-narinfo): Pass #:quiet-404? #t.
* guix/pk-crypto.scm (bytevector->hash-data): Add #:key-type parameter.
Use the 'pkcs1' flag when KEY-TYPE is 'rsa', and 'rfc6979' when
KEY-TYPE is 'ecc' or 'dsa'.
(key-type): New procedure.
* guix/scripts/authenticate.scm (read-hash-data): Add 'key-type'
parameter. Pass it to 'bytevector->hash-data'. Adjust caller
accordingly.
* tests/pk-crypto.scm (%ecc-key-pair): New variable.
("key-type"): New test.
("sign + verify"): Pass #:key-type to 'bytevector->hash-data'.
("sign + verify, Ed25519"): New test.
* gnu/packages/gcc.scm (gcc-4.7): Change /nix/store in comment.
* gnu/system/vm.scm (operating-system-default-contents):
Use (%store-prefix) instead of "/nix/store".
* guix/derivations.scm (derivation-path->output-path,
derivation-path->output-paths): Change to /gnu/store in docstring.
* gnu/packages/ld-wrapper.scm (%store-directory): Change the default to
/gnu/store.
* guix/build/utils.scm (%store-directory): New procedure.
(remove-store-references): Use it for the default value of 'store'.
* guix/packages.scm (patch-and-repack)[builder]: Change default store to
/gnu/store.
* guix/scripts/offload.scm (offload): Remove default value for
'build-timeout'. Pass '--timeout' to the remote 'guix build'
process.
(transfer-and-offload, process-request): Remove default value for
'build-timeout'.
* guix/scripts/offload.scm (transfer-and-offload): New procedure, with
core formerly in 'process-request'.
(choose-build-machine): Remove 'requirements' parameter.
(process-request): Reply 'decline' when none of MACHINES matches the
requirements, and 'postpone' when MACHINES are busy.
* guix/scripts/offload.scm (machine-choice-lock-file,
machine-slot-file, acquire-build-slot, release-build-slot): New
procedures.
(choose-build-machine): Operate with (machine-choice-lock-file)
taken. Acquire a build slot for each of MACHINES. Release those not
used.
* guix/scripts/offload.scm (lock-machine, unlock-machine): Remove.
(lock-file, unlock-file): New procedures.
(with-file-lock): New macro.
(with-machine-lock): Rewrite in terms of 'with-file-lock'.
* guix/scripts/offload.scm (machine-lock-file, lock-machine,
unlock-machine): New procedures.
(with-machine-lock): New macro.
(process-request): Wrap 'send-files' and 'retrieve-files' calls in
'with-machine-lock'.
* guix/http-client.scm (&http-get-error): New condition type.
(http-fetch): Raise it instead of using 'error'.
* guix/scripts/substitute-binary.scm (fetch) <http>: Wrap body into
'guard' form; gracefully handle 'http-get-error?' conditions.
* guix/scripts/offload.scm (machine-load, machine-less-loaded?,
machine-less-loaded-or-faster?): New procedures.
(choose-build-machine): Use 'machine-less-loaded-or-faster?' when
sorting. Return the head of MACHINES unless it's loaded is >= 2.
* guix/packages.scm (patch-and-repack)[numeric-extension?, tarxz-name]:
New procedures.
[builder]: Adjust to deal with SOURCE when it's a directory.
<body>: Use 'tarxz-name'. Always add (guix build utils) to
IMPORTED-MODULES.
* guix/build/gnu-build-system.scm (unpack): Check if SOURCE is a
directory, and copy it locally if it is.
* gnu/packages/libwebsockets.scm (libwebsockets)[arguments]: Remove
'unpack' phase.
* guix/nar.scm (write-file) <directory>: Pass 'string<?' as the second
argument to 'scandir'.
* tests/nar.scm ("write-file puts file in C locale collation order"):
New test.
* guix/derivations.scm (<derivation-output>): Add 'recursive?' field.
Adjust 'make-derivation-output' callers.
(%read-derivation) <fixed-output>: When HASH-ALGO starts with 'r:',
set the 'recursive?' field and drop 'r:' from the hash algo name.
(write-derivation)[write-output]: Write the algo as 'r:HASH-ALGO' when
the RECURSIVE? field is set.
(derivation-hash) <fixed-output>: Prepend "r:" when RECURSIVE? is set.
(fixed-output-path): New procedure.
(derivation): Add #:recursive? parameter. Use 'fixed-output-path' to
compute the output file name of a fixed output derivation.
(build-expression->derivation): Add #:recursive? parameter. Pass it
to 'derivation'.
* tests/derivations.scm ("fixed-output derivation, recursive",
"build-expression->derivation produces recursive fixed-output",
"build-expression->derivation uses recursive fixed-output"): New
tests.
* doc/guix.texi (Derivations): Document #:recursive? for 'derivation'.
Add #:recursive? for 'build-expression->derivation'.
* guix/scripts/hash.scm (show-help): Add --recursive.
(%options): Likewise.
(guix-hash)[file-hash]: New procedure. Honor --recursive.
Use it.
* guix/nar.scm (write-file): Add missing field to the &nar-error
condition raised upon unsupported file type; change its message to be
more descriptive.
* tests/guix-hash.sh: Add tests with -r.
* doc/guix.texi (Invoking guix hash): Document --recursive.
* guix/scripts/system.scm: New file.
* Makefile.am (MODULES): Add it.
* po/POTFILES.in: Likewise.
* doc/guix.texi (Using the Configuration System): Link to "Invoking guix
system".
(Invoking guix system): New node.
* guix/build/union.scm (file=?): Return #f if FILE1 and FILE2 are not
regular files. Fixes a bug whereby collisions among directories would
lead to the invocation of 'file=?' and thus 'call-with-input-file' on
directories. Reported by Mark H. Weaver <mhw@netris.org>.
Fixes <http://bugs.gnu.org/16703>.
Reported by Raimon Grau <raimonster@gmail.com>.
* guix/build/download.scm (http-fetch)[headers]: New variable.
Pass it as #:headers or #:extra-headers to 'http-get' and
'http-get*'.
* guix/monads.scm (text-file*): New procedure.
* tests/monads.scm ("text-file*"): New test.
* doc/guix.texi (The Store Monad): Change example since the previous one
would erroneously fail to retain a reference to Coreutils. Document
'text-file*'.
* gnu/system/linux-initrd.scm (qemu-initrd)[virtio-9p-modules]: New
variable.
[linux-modules]: Append VIRTIO-9P-MODULES when a 9p file system is in
MOUNTS.
* guix/build/linux-initrd.scm (mount-qemu-9p): New procedure.
(boot-system): Recognize '9p' in MOUNTS, and use 'mount-qemu-9p'.
* guix/build/linux-initrd.scm (make-essential-device-nodes): Rename
devices with major = 8 to /dev/sda*. Make /dev/vda* devices.
* gnu/system/vm.scm (qemu-image): Change '/dev/vda' to '/dev/sda'.
* gnu/system.scm (operating-system-derivation): Likewise.
* guix/build/linux-initrd.scm (boot-system): New procedure.
* gnu/system/linux-initrd.scm (qemu-initrd): Add keyword parameters
'guile-modules-in-chroot?' and 'mounts'. Change builder to simply
call 'boot-system'.
(gnu-system-initrd): Change to a simple call to 'qemu-initrd'.
* gnu/system/vm.scm (expression->derivation-in-linux-vm): Call
'qemu-initrd' with #:guile-modules-in-chroot?.
* guix/scripts/offload.scm (offload): Add 'log-port' keyword parameter.
Handle log display here. Return the result of (close-pipe pipe).
(process-request): Adjust 'offload' call site accordingly. Call
'retrieve-files' only when 'offload' returns zero; exit when 'offload'
returns non-zero.
* guix/derivations.scm (derivation): Add #:local-build? parameter and
honor it.
(build-expression->derivation): Likewise.
* doc/guix.texi (Derivations): Update documentation of these
procedures.
* nix/nix-daemon/guix-daemon.cc (GUIX_OPT_NO_BUILD_HOOK): New macro.
(options): Add '--no-build-hook'.
(parse_opt): Handle it.
(main)[HAVE_DAEMON_OFFLOAD_HOOK]: Set 'useBuildHook' by default.
Set $NIX_BUILD_HOOK to our offload hook unless otherwise specified.
[!HAVE_DAEMON_OFFLOAD_HOOK]: Clear 'useBuildHook'.
* pre-inst-env.in: Set and export NIX_BUILD_HOOK.
* nix/scripts/offload.in, guix/scripts/offload.scm: New files.
* guix/ui.scm (show-guix-help)[internal?]: Add "offload".
* config-daemon.ac: Call 'GUIX_CHECK_UNBUFFERED_CBIP'.
Instantiate 'nix/scripts/offload'. Set 'BUILD_DAEMON_OFFLOAD'
conditional, and optionally define 'HAVE_DEAMON_OFFLOAD_HOOK' cpp
macro.
* daemon.am (nodist_pkglibexec_SCRIPTS)[BUILD_DAEMON_OFFLOAD]: Add it.
* Makefile.am (MODULES)[BUILD_DAEMON_OFFLOAD]: Add
'guix/scripts/offload.scm'.
(EXTRA_DIST)[!BUILD_DAEMON_OFFLOAD]: Likewise.
* m4/guix.m4 (GUIX_CHECK_UNBUFFERED_CBIP): New macro.
* doc/guix.texi (Setting Up the Daemon): Move most of the body to...
(Build Environment Setup): ... this. New subsection.
(Daemon Offload Setup): New subsection.
* guix/store.scm (topologically-sorted): New procedure.
* tests/store.scm ("topologically-sorted, one item",
"topologically-sorted, several items", "topologically-sorted, more
difficult"): New tests.
* guix/pki.scm (signature-subject, signature-signed-data,
valid-signature?): New procedures.
* guix/scripts/authenticate.scm (guix-authenticate): Adjust to use
them.
* nix-upstream: Update sub-module.
* daemon.am (libutil_a_SOURCES): Add affinity.cc.
(libutil_headers): Add affinity.hh.
(libexec_PROGRAMS, nix_setuid_helper_SOURCES,
nix_setuid_helper_CPPFLAGS, nix_setuid_helper_LDADD): Remove.
* doc/guix.texi (Setting Up the Daemon): Remove paragraph about
'nix-setuid-helper'.
* guix/scripts/substitute-binary.scm (guix-substitute-binary): Exit 0 when
%CACHE-URL has an HTTP scheme and looking up its host fails.
Always print a newline to stdout when starting.
Ludovic Courtès