Commit Graph

77 Commits

Author SHA1 Message Date
Ludovic Courtès a4b42825a1 offload: Bail out when failing to register a GC root on the build machine.
* guix/scripts/offload.scm (register-gc-root): Call 'leave' when
  'close-pipe' returns non-zero.
2014-04-08 10:09:27 +02:00
Ludovic Courtès 4b00f3434e offload: Prevent the '.drv' and build result from being GC'd.
Before that, there was a small time window during which the GC could
wipe the .drv (before 'guix build' has been called), or the build
result (before 'retrieve-files' has started.)

* guix/scripts/offload.scm (remote-pipe): Add #:quote? parameter and
  honor it.
  (%gc-root-file): New variable.
  (register-gc-root, remove-gc-root): New procedures.
  (offload): Adjust comment.  Run 'guix build' with '-r %GC-ROOT-FILE'.
  (transfer-and-offload): Call 'register-gc-root' before
  sending (derivation-file-name DRV).  Call 'remove-gc-root' after the
  call to 'offload' or 'retrieve-files'.
  (send-files): Call 'remote-pipe' with #:quote? #f.
  (retrieve-files): Likewise.
2014-04-03 23:55:51 +02:00
Ludovic Courtès 36b5851df6 offload: Exit with code 100 upon build failures.
* guix/scripts/offload.scm (transfer-and-offload): Exit with code 100
  upon build failure.
2014-03-31 10:57:28 +02:00
Ludovic Courtès 07fb21b231 offload: Remove erroneous 'close-pipe' call.
* guix/scripts/offload.scm (send-files): Remove 'close-pipe' call from
  'guard' handler ('pipe' here referred to Guile's 'pipe' procedure.)
2014-03-26 16:27:46 +01:00
Ludovic Courtès 6c41cce0be offload: Wait for the processes involved in 'guix archive --missing'.
* guix/scripts/offload.scm (send-files): Keep the second return value of
  'filtered-port'.  Call 'waitpid' on it.
2014-03-26 16:27:46 +01:00
Ludovic Courtès 236e66481d offload: Allow one transfer in each direction simultaneously.
* guix/scripts/offload.scm (transfer-and-offload): Use 'upload' lock
  instead of 'bandwidth' around 'send-files' calls, and 'download' lock
  around 'retrieve-files' call.
2014-03-26 15:06:52 +01:00
Ludovic Courtès 3dfd8af534 offload: Disable SSH-level compression.
* guix/scripts/offload.scm (remote-pipe): Remove '-z' lsh command line
  argument.  This makes transfers almost an order of magnitude slower.
  OpenSSH's ssh(1) man page notes: "Compression is desirable on modem lines
  and other slow connections, but will only slow down things on fast
  networks."  See also
  <http://www.spikelab.org/blog/transfer-largedata-scp-tarssh-tarnc-compared.html>.
2014-03-26 15:06:52 +01:00
Ludovic Courtès 8b7af63754 offload: Compress files being sent/retrieved.
* guix/scripts/offload.scm (send-files): Add "xz -dc |" to the remote
  pipe command.  Pass PIPE through 'call-with-compressed-output-port'.
  Remove 'close-pipe' call.
  (retrieve-files): Add "| xz -c" to the remote pipe command.  Pass PIPE
  through 'call-with-decompressed-port'.  Remove 'close-pipe' call.
2014-03-24 22:20:54 +01:00
Ludovic Courtès d81195bffd offload: Send build logs to file descriptor 4.
* guix/scripts/offload.scm (with-error-to-port): New macro.
  (remote-pipe): Add #:error-port parameter.  Use 'with-error-to-port'
  around 'open-pipe*' call.
  (build-log-port): New procedure.
  (offload): Change #:log-port to default to (build-log-port).  Call
  'remote-pipe' with #:error-port LOG-PORT.
2014-03-19 23:12:06 +01:00
Ludovic Courtès 1a8ea0a188 offload: Fix 'choose-build-machine' for several machines.
* guix/scripts/offload.scm (choose-build-machine)[undecorate]: Turn into
  a two-argument procedure.
2014-03-13 22:57:21 +01:00
Ludovic Courtès 3c0e6e6080 offload: Convert the port number to a string when invoking lsh.
* guix/scripts/offload.scm (remote-pipe, send-files): Pass the result of
  'build-machine-port' to 'number->string'.
2014-03-13 22:46:32 +01:00
Ludovic Courtès cecd72d55a offload: Allow build machines to specify a port number.
* guix/scripts/offload.scm (<build-machine>): Add 'port' field.
  (remote-pipe, send-files): Use lsh's '-p' option when invoking it.
2014-03-13 21:58:04 +01:00
Ludovic Courtès 714084e6c0 offload: Honor absolute build timeouts.
* guix/scripts/offload.scm (offload): Remove default value for
  'build-timeout'.  Pass '--timeout' to the remote 'guix build'
  process.
  (transfer-and-offload, process-request): Remove default value for
  'build-timeout'.
2014-03-09 23:15:09 +01:00
Ludovic Courtès 59f704dff6 offload: Move macro definitions before use.
* guix/scripts/offload.scm (lock-file, unlock-file, with-file-lock,
  with-machine-lock, machine-slot-file, acquire-build-slot,
  release-build-slot): Move definitions above their first use.
2014-03-09 18:08:38 +01:00
Ludovic Courtès 88da0b6888 offload: Distinguish between 'decline' and 'postpone'.
* guix/scripts/offload.scm (transfer-and-offload): New procedure, with
  core formerly in 'process-request'.
  (choose-build-machine): Remove 'requirements' parameter.
  (process-request): Reply 'decline' when none of MACHINES matches the
  requirements, and 'postpone' when MACHINES are busy.
2014-03-09 14:06:35 +01:00
Ludovic Courtès 5cc569dc73 offload: Prevent locked files from being GC'd.
* guix/scripts/offload.scm (%slots): New variable.
  (choose-build-machine): Add SLOT to '%slots'.
2014-03-08 21:23:12 +01:00
Ludovic Courtès 7df3ade112 offload: Fix thinko.
* guix/scripts/offload.scm (choose-build-machine)[machine+slots]: Use
  'filter-map', not 'filter'.
2014-03-08 12:22:49 +01:00
Ludovic Courtès d652b85137 offload: Make 'parallel-builds' a hard limit.
* guix/scripts/offload.scm (machine-choice-lock-file,
  machine-slot-file, acquire-build-slot, release-build-slot): New
  procedures.
  (choose-build-machine): Operate with (machine-choice-lock-file)
  taken.  Acquire a build slot for each of MACHINES.  Release those not
  used.
2014-03-08 12:16:06 +01:00
Ludovic Courtès 4bf1eb4f88 offload: Further generalize lock files.
* guix/scripts/offload.scm (lock-machine, unlock-machine): Remove.
  (lock-file, unlock-file): New procedures.
  (with-file-lock): New macro.
  (with-machine-lock): Rewrite in terms of 'with-file-lock'.
2014-03-08 12:07:57 +01:00
Ludovic Courtès 178f5828eb offload: Generalize the machine lock mechanism.
* guix/scripts/offload.scm (lock-machine): Add 'hint' parameter.
  (unlock-machine): Remove 'machine' parameter.
  (with-machine-lock): Add 'hint' parameter, and pass it down.
  (process-request): Adjust uses of 'with-machine-lock' to pass the
  'bandwidth hint.
2014-03-08 11:29:52 +01:00
Ludovic Courtès f326fef8a8 offload: Serialize file transfers to build machines.
* guix/scripts/offload.scm (machine-lock-file, lock-machine,
  unlock-machine): New procedures.
  (with-machine-lock): New macro.
  (process-request): Wrap 'send-files' and 'retrieve-files' calls in
  'with-machine-lock'.
2014-03-06 21:42:24 +01:00
Ludovic Courtès 1f7fd80032 offload: Comment out attempt to set up an lsh gateway.
* guix/scripts/offload.scm (open-ssh-gateway): Comment out.
  (process-request): Remove call to 'open-ssh-gateway' and to 'kill'.
2014-03-01 12:24:39 +01:00
Ludovic Courtès aedbf9b873 offload: 'remote-pipe' uses the right SSH key.
* guix/scripts/offload.scm (remote-pipe): Pass -i when invoking
  %LSHG-COMMAND.
2014-03-01 12:15:47 +01:00
Ludovic Courtès 165f4b2add offload: Take the target machine load into account.
* guix/scripts/offload.scm (machine-load, machine-less-loaded?,
  machine-less-loaded-or-faster?): New procedures.
  (choose-build-machine): Use 'machine-less-loaded-or-faster?' when
  sorting.  Return the head of MACHINES unless it's loaded is >= 2.
2014-03-01 01:31:18 +01:00
Ludovic Courtès 92cb2e2888 offload: Have 'build-machines' honor its argument.
* guix/scripts/offload.scm (build-machines): Honor FILE.
2014-02-02 01:32:50 +01:00
Ludovic Courtès a76611c435 offload: Do not try to retrieve anything upon build failure.
* guix/scripts/offload.scm (offload): Add 'log-port' keyword parameter.
  Handle log display here.  Return the result of (close-pipe pipe).
  (process-request): Adjust 'offload' call site accordingly.  Call
  'retrieve-files' only when 'offload' returns zero; exit when 'offload'
  returns non-zero.
2014-01-27 21:32:59 +01:00
Ludovic Courtès 49e6291a7a Add 'guix offload' as a daemon build hook.
* nix/nix-daemon/guix-daemon.cc (GUIX_OPT_NO_BUILD_HOOK): New macro.
  (options): Add '--no-build-hook'.
  (parse_opt): Handle it.
  (main)[HAVE_DAEMON_OFFLOAD_HOOK]: Set 'useBuildHook' by default.
  Set $NIX_BUILD_HOOK to our offload hook unless otherwise specified.
  [!HAVE_DAEMON_OFFLOAD_HOOK]: Clear 'useBuildHook'.
* pre-inst-env.in: Set and export NIX_BUILD_HOOK.
* nix/scripts/offload.in, guix/scripts/offload.scm: New files.
* guix/ui.scm (show-guix-help)[internal?]: Add "offload".
* config-daemon.ac: Call 'GUIX_CHECK_UNBUFFERED_CBIP'.
  Instantiate 'nix/scripts/offload'.  Set 'BUILD_DAEMON_OFFLOAD'
  conditional, and optionally define 'HAVE_DEAMON_OFFLOAD_HOOK' cpp
  macro.
* daemon.am (nodist_pkglibexec_SCRIPTS)[BUILD_DAEMON_OFFLOAD]: Add it.
* Makefile.am (MODULES)[BUILD_DAEMON_OFFLOAD]: Add
  'guix/scripts/offload.scm'.
  (EXTRA_DIST)[!BUILD_DAEMON_OFFLOAD]: Likewise.
* m4/guix.m4 (GUIX_CHECK_UNBUFFERED_CBIP): New macro.
* doc/guix.texi (Setting Up the Daemon): Move most of the body to...
  (Build Environment Setup): ... this.  New subsection.
  (Daemon Offload Setup): New subsection.
2014-01-24 00:01:50 +01:00