* guix/scripts/offload.scm (connect-to-remote-daemon)
(store-import-channel, store-export-channel, send-files)
(retrieve-files): Move to (guix ssh).
(nonce): Add optional 'name' parameter and use it.
(retrieve-files*): New procedure.
(transfer-and-offload): Use it instead of 'retrieve-files', and add
first parameter to 'send-files'.
(assert-node-can-import): Likewise.
(assert-node-can-export): Use 'retrieve-files' instead of
'store-export-channel'.
* guix/ssh.scm: New file.
* configure.ac: Use 'GUIX_CHECK_GUILE_SSH' and define 'HAVE_GUILE_SSH'
Automake conditional.
* Makefile.am (MODULES) [HAVE_GUILE_SSH]: Add guix/ssh.scm.
* guix/scripts/offload.scm (check-machine-availability): Add 'pred'
parameter and honor it.
(guix-offload): for the "test" sub-command, accept an extra 'regexp'
parameter. Pass a second argument to 'check-machine-availability'.
This fixes a regression introduced in
21531add32 whereby the build log would no
longer be sent to FD 4, thereby leading the daemon to not see the build
log.
* guix/scripts/offload.scm (transfer-and-offload): Parameterize
CURRENT-BUILD-OUTPUT-PORT.
* guix/scripts/offload.scm (assert-node-repl, assert-node-has-guix)
(nonce, assert-node-can-import, assert-node-can-export)
(check-machine-availability): New procedures.
(%random-state): New variable.
(guix-offload): Add case for "test".
* doc/guix.texi (Daemon Offload Setup): Document it. Remove obsolete
bit about remote invocation of 'guix build'.
This fixes a longstanding issue where 'choose-build-machine' would make
on average O(N log(N)) calls to 'machine-load', plus an extra call for
the selected machine, instead of N calls.
* guix/scripts/offload.scm (machine-load): Add comment.
(machine-power-factor, machine-less-loaded-or-faster?): Remove.
(choose-build-machine)[machines+slots]: Rename to...
[machines+slots+loads]: ... this.
[undecorate]: Adjust accordingly.
[machine-less-loaded-or-faster?]: New procedure.
Remove extra 'machine-load' call in body.
* guix/scripts/offload.scm (<build-machine>)[daemon-socket]: New field.
(connect-to-remote-daemon): New procedure.
(%gc-root-file, register-gc-root, remove-gc-roots, offload): Remove.
(transfer-and-offload): Rewrite using 'connect-to-remote-daemon' and
RPCs over SSH.
(store-import-channel, store-export-channel): New procedures.
(send-files, retrieve-files): Rewrite using these.
* guix/scripts/offload.scm (<build-machine>)[ssh-options]: Remove.
[host-key, host-key-type]: New fields.
(%lsh-command, %lshg-command, user-lsh-private-key): Remove.
(user-openssh-private-key, private-key-from-file*): New procedures.
(host-key->type+key, open-ssh-session): New procedures.
(remote-pipe): Remove 'mode' parameter. Rewrite in terms of
'open-ssh-session' etc. Update users.
(send-files)[missing-files]: Rewrite using the bidirectional channel
port.
Remove call to 'call-with-compressed-output-port'.
(retrieve-files): Remove call to 'call-with-decompressed-port'.
(machine-load): Remove exit status logic.
* doc/guix.texi (Requirements): Mention Guile-SSH.
(Daemon Offload Setup): Document 'host-key' and 'private-key'. Show the
default value on each @item line.
* m4/guix.m4 (GUIX_CHECK_GUILE_SSH): New macro.
* config-daemon.ac: Use 'GUIX_CHECK_GUILE_SSH'. Set
'HAVE_DAEMON_OFFLOAD_HOOK' as a function of that.
Suggested by Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>.
* guix/scripts/offload.scm (register-gc-root)[script]: Replace
'false-if-exception' with a finer-grain 'system-error handler.
Provide the name of MACHINE in 'leave' error message.
Suggested by Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>.
* guix/scripts/offload.scm (remote-pipe): Remove unneeded 'catch'.
(machine-load): Check the exit value upon (close-pipe pipe). Call
'warning' when it is non-zero.
* guix/scripts/offload.scm (machine-less-loaded?, machine-faster?):
Remove.
(machine-power-factor): New procedure.
(machine-less-loaded-or-faster?): Use it.
Fixes <http://bugs.gnu.org/18115>.
Reported by Mark H Weaver <mhw@netris.org>.
* guix/scripts/offload.scm (register-gc-root)[script]: Wrap 'symlink'
call in "catch 'system-error", and ignore EEXIST errors.
Fixes <http://bugs.gnu.org/18070>.
Reported by Andreas Enge <andreas@enge.fr>.
* guix/scripts/offload.scm (remote-pipe): Augment docstring.
(machine-load): Return +inf.0 instead of 1 if MACHINE does not respond
or responds badly.
* guix/scripts/system.scm (read-operating-system) <catch handler>: Add
case for 'syntax-error'. Correct message for default case.
* guix/scripts/offload.scm (build-machines) <catch handler>: Add case
for 'syntax-error'.
* tests/guix-system.sh: New file.
* Makefile.am (SH_TESTS): Add it.
Fixes situations where the remote 'guix build' is invoked before the
.drv has been completely copied, as reported at
<https://lists.gnu.org/archive/html/guix-devel/2014-04/msg00174.html>.
In some cases 'send-files' would return before the other end is done
importing the files, and so the subsequent 'guix build' invocation would
just miss the .drv file it refers to.
* guix/utils.scm (call-with-decompressed-port): Don't close PORT.
(call-with-compressed-output-port): Likewise.
* tests/utils.scm ("compressed-output-port + decompressed-port"): Adjust
accordingly.
* guix/scripts/offload.scm (send-files): Add explicit (close-pipe pipe)
call.
(retrieve-files): Likewise.
* guix/scripts/offload.scm (remove-gc-root): Rename to...
(remove-gc-roots): ... this.
[builder]: Use 'scandir' and remove all the files starting with
%GC-ROOT-FILE.
(transfer-and-offload): Adjust to renaming; remove
'false-if-exception' wraps.
Before that, there was a small time window during which the GC could
wipe the .drv (before 'guix build' has been called), or the build
result (before 'retrieve-files' has started.)
* guix/scripts/offload.scm (remote-pipe): Add #:quote? parameter and
honor it.
(%gc-root-file): New variable.
(register-gc-root, remove-gc-root): New procedures.
(offload): Adjust comment. Run 'guix build' with '-r %GC-ROOT-FILE'.
(transfer-and-offload): Call 'register-gc-root' before
sending (derivation-file-name DRV). Call 'remove-gc-root' after the
call to 'offload' or 'retrieve-files'.
(send-files): Call 'remote-pipe' with #:quote? #f.
(retrieve-files): Likewise.
* guix/scripts/offload.scm (transfer-and-offload): Use 'upload' lock
instead of 'bandwidth' around 'send-files' calls, and 'download' lock
around 'retrieve-files' call.
* guix/scripts/offload.scm (remote-pipe): Remove '-z' lsh command line
argument. This makes transfers almost an order of magnitude slower.
OpenSSH's ssh(1) man page notes: "Compression is desirable on modem lines
and other slow connections, but will only slow down things on fast
networks." See also
<http://www.spikelab.org/blog/transfer-largedata-scp-tarssh-tarnc-compared.html>.
* guix/scripts/offload.scm (with-error-to-port): New macro.
(remote-pipe): Add #:error-port parameter. Use 'with-error-to-port'
around 'open-pipe*' call.
(build-log-port): New procedure.
(offload): Change #:log-port to default to (build-log-port). Call
'remote-pipe' with #:error-port LOG-PORT.
* guix/scripts/offload.scm (offload): Remove default value for
'build-timeout'. Pass '--timeout' to the remote 'guix build'
process.
(transfer-and-offload, process-request): Remove default value for
'build-timeout'.
Ludovic Courtès