Commit Graph

144 Commits

Author SHA1 Message Date
Ludovic Courtès e037e9dbec
store: Buffer RPC writes.
For a command like:

  guix build python2-numpy -n

this reduces the number of 'write' syscalls from 9.5K to 2.0K.

* guix/store.scm (<nix-server>)[buffer, flush]: New fields.
(open-connection): Adjust accordingly.  Call 'buffering-output-port' to
compute the two new fields.
(write-buffered-output, buffering-output-port): New procedures.
(operation): Write to (nix-server-output-port server).  Call
'write-buffered-output'.
2017-06-19 23:23:14 +02:00
Ludovic Courtès f4453df9a5
store: Add an RPC counter.
* guix/store.scm (%rpc-calls): New variable.
(show-rpc-profile, record-operation): New procedures.
(operation): Add call to 'record-operation'.
* guix/ui.scm (run-guix-command): Wrap COMMAND-MAIN in 'dynamic-wind'.
Run EXIT-HOOK.
2017-06-16 17:08:22 +02:00
Ludovic Courtès b46712159c
store: Speed up 'add-to-store'.
* guix/store.scm (add-to-store): Remove 'lstat' call.
2017-06-12 17:53:51 +02:00
Ludovic Courtès 950d51c9d9
store: Use 'TCP_NODELAY' when connecting to a daemon over PF_INET.
* guix/store.scm (open-inet-socket): Add 'cond-expand' form to define
'TCP_NODELAY' when needed.  Add call to 'setsockopt' after 'connect'.
2017-05-07 00:18:36 +02:00
Ludovic Courtès 396d12faea
store: Use 'write-bytevector' instead of hand-coded equivalent.
* guix/store.scm (process-stderr): Use 'write-bytevector' in the
%STDERR-READ case.
2017-05-04 18:05:05 +02:00
Ludovic Courtès cd041b268e
store: Add store path computation procedures.
* guix/derivations.scm (compressed-hash, store-path)
(output-path, fixed-output-path): Move to...
* guix/store.scm: ... here.
2017-05-04 18:05:05 +02:00
Ludovic Courtès dcb95c1fc9
monads: Add a template and specialization mechanism for monadic procedures.
* guix/monads.scm (%templates, %template-instances): New variables.
(register-template!, register-template-instance!): New procedures.
(template-directory, define-template): New macro.
(foldm, sequence, anym): Define using 'define-template'.  Avoid replace
ellipses with dots.
(mapm): Likewise, but do not use 'foldm'.
* guix/store.scm: Add 'template-directory' invocation.
2017-05-02 23:41:13 +02:00
Ludovic Courtès 285f63e805
store: Support 'ssh://' URIs in 'GUIX_DAEMON_SOCKET'.
This allows 'guix' commands to talk to a remote store over SSH.

* guix/store.scm (connect-to-daemon)[connect]: Call 'resolve-interface'
for unknown URI schemes.
* guix/store/ssh.scm: New file.
* Makefile.am (MODULES): Add it.
* doc/guix.texi (The Store): Document it.  Mark remote access as
experimental.
2017-04-21 19:17:36 +02:00
Ludovic Courtès 3dff90ce34
store: Add support for remote connections via 'guix://' URIs.
* guix/store.scm (open-inet-socket): New procedure.
(connect-to-daemon): Support the 'guix' URI scheme.
* doc/guix.texi (The Store): Document it.
2017-04-21 17:23:37 +02:00
Ludovic Courtès 3b5cd17a97
store: Add 'system-error-to-connection-error' macro.
* guix/store.scm (system-error-to-connection-error): New macro.
(open-unix-domain-socket): Use it instead of 'catch'.
2017-04-21 17:23:37 +02:00
Ludovic Courtès 1397b422e2
store: 'GUIX_DAEMON_SOCKET' can now be a URI.
* guix/store.scm (%daemon-socket-file): Rename to...
(%daemon-socket-uri): ... this.
(connect-to-daemon): New procedure.
(open-connection): Rename 'file' to 'uri'.  Use 'connect-to-daemon'
instead of 'open-unix-domain-socket'.
* guix/tests.scm (open-connection-for-tests): Rename 'file' to 'uri'.
* tests/guix-build.sh: Add tests.
* tests/store.scm ("open-connection with file:// URI"): New tests.
2017-04-21 17:23:37 +02:00
Ludovic Courtès 4c0c4db070
utils: Move base16 procedures to (guix base16).
* guix/utils.scm (bytevector->base16-string, base16-string->bytevector):
Move to...
* guix/base16.scm: ... here.  New file.
* tests/utils.scm ("bytevector->base16-string->bytevector"): Move to...
* tests/base16.scm: ... here.  New file.
* Makefile.am (MODULES): Add guix/base16.scm.
(SCM_TESTS): Add tests/base16.scm.
* build-aux/download.scm, guix/derivations.scm,
guix/docker.scm, guix/import/snix.scm, guix/pk-crypto.scm,
guix/scripts/authenticate.scm, guix/scripts/download.scm,
guix/scripts/hash.scm, guix/store.scm, tests/hash.scm,
tests/pk-crypto.scm: Adjust imports accordingly.
2017-03-16 22:50:14 +01:00
Ludovic Courtès 0d268c5d70
store: Add 'add-data-to-store'.
* guix/serialization.scm (write-bytevector): New procedure.
(write-string): Rewrite in terms of 'write-bytevector'.
* guix/store.scm (write-arg): Add 'bytevector' case.
(add-data-to-store): New procedure, from former 'add-text-to-store'.
(add-text-to-store): Rewrite in terms of 'add-data-to-store'.
* tests/store.scm ("add-data-to-store"): New test.
2017-01-30 10:52:45 +01:00
Ludovic Courtès 55b2d92145
Use 'mlambda' instead of 'memoize'.
* gnu/packages.scm (find-newest-available-packages): Use 'mlambda'
instead of (memoize (lambda ...) ...).
* gnu/packages/bootstrap.scm (package-with-bootstrap-guile): Likewise.
* guix/build-system/gnu.scm (package-with-explicit-inputs)[rewritten-input]:
Likewise.
* guix/build-system/python.scm (package-with-explicit-python)[transform]:
Likewise.
* guix/derivations.scm (derivation->string): Likewise.
* guix/gnu-maintenance.scm (gnu-package?): Likewise.
* guix/modules.scm (module-file-dependencies): Likewise.
* guix/scripts/graph.scm (standard-package-set): Likewise.
* guix/scripts/lint.scm (official-gnu-packages*): Likewise.
* guix/store.scm (store-regexp*): Likewise.
* guix/utils.scm (location): Likewise.
2017-01-28 18:55:20 +01:00
Ludovic Courtès f9704f179a
Add (guix memoization).
* guix/combinators.scm (memoize): Remove.
* guix/memoization.scm: New file.
* Makefile.am (MODULES): Add it.
* gnu/packages.scm, gnu/packages/bootstrap.scm,
guix/build-system/gnu.scm, guix/build-system/python.scm,
guix/derivations.scm, guix/gnu-maintenance.scm,
guix/import/cran.scm, guix/import/elpa.scm,
guix/modules.scm, guix/scripts/build.scm,
guix/scripts/graph.scm, guix/scripts/lint.scm,
guix/store.scm, guix/utils.scm: Adjust imports accordingly.
2017-01-28 18:55:20 +01:00
Ludovic Courtès deac976d3d
daemon: Client settings no longer override daemon settings.
Fixes <http://bugs.gnu.org/20217>.

* nix/libstore/worker-protocol.hh (PROTOCOL_VERSION): Bump to 0x161.
* nix/nix-daemon/nix-daemon.cc (performOp): "build-max-jobs",
"build-max-silent-time", and "build-cores" are no longer read upfront;
instead, read them from the key/value list at the end.
* nix/nix-daemon/guix-daemon.cc (main): Explicitly set
'settings.maxBuildJobs'.
* guix/store.scm (%protocol-version): Bump to #x161.
(set-build-options): #:max-build-jobs, #:max-silent-time, and
 #:build-cores now default to #f.  Adjust handshake to new protocol.
* tests/store.scm ("build-cores"): New test.
* tests/guix-daemon.sh: Add test for default "build-cores" value.
2017-01-15 15:43:22 +01:00
Ludovic Courtès 13d5e8dae5
store: 'open-connection' no longer raises '&nar-error' for protocol errors.
* guix/store.scm (open-connection): Guard body against 'nar-error?' and
re-raise as '&nix-connection-error'.
* tests/store.scm ("connection handshake error"): New test.
2016-12-08 23:12:11 +01:00
Ludovic Courtès 6374633b92
store: Increase buffering for the '%stderr-write' upcall.
* guix/store.scm (process-stderr) <%stderr-write>: Pass #:buffer-size to
'dump-port'.
2016-12-01 23:35:10 +01:00
Ludovic Courtès 6230d6f04f
store: 'open-connection' can taken an open port.
* guix/store.scm (open-unix-domain-socket): New procedure.
(open-connection): Add #:port parameter and honor it.
2016-11-25 23:44:20 +01:00
Ludovic Courtès e74f64b9e5
store: Add 'references*'.
* guix/store.scm (references*): New procedure.
* guix/profiles.scm (manifest-lookup-package)[references*]: Remove.
* guix/scripts/system.scm (references*): Remove.
* tests/gexp.scm ("gexp->file", "gexp->file + file-append")
("gexp->derivation", "gexp->derivation, cross-compilation")
("gexp->derivation, ungexp + ungexp-native")
("scheme-file", "text-file*", "mixed-text-file"): Remove 'references*'
instead of (store-lift references).
2016-11-19 18:44:53 +01:00
Ludovic Courtès f9aefa2d5f
daemon: Add 'built-in-builders' RPC.
* nix/libstore/builtins.cc (builtinBuilderNames): New function.
* nix/libstore/builtins.hh (builtinBuilderNames): New declaration.
* nix/libstore/worker-protocol.hh (PROTOCOL_VERSION): Bump to 0x160.
(WorkerOp)[wopBuiltinBuilders]: New value.
* nix/nix-daemon/nix-daemon.cc (performOp): Handle it.
* guix/store.scm (operation-id)[built-in-builders]: New value.
* guix/store.scm (read-arg): Add 'string-list'.
(built-in-builders): New procedure.
* tests/derivations.scm ("built-in-builders"): New test.
2016-11-16 18:19:54 +01:00
Ludovic Courtès 2535635f18
Use (ice-9 binary-ports) instead of (rnrs io ports).
This reduces the closure of (guix ui) from 123 to 106 modules.

* guix/derivations.scm: Use (ice-9 binary-ports) instead of (rnrs io
ports).
(map-derivation)[substitute-file]: Use 'read-string' instead of
'get-string-all'.
* guix/ftp-client.scm: Likewise.
* guix/hash.scm: Likewise.
* guix/http-client.scm: Likewise.
* guix/pki.scm (ensure-acl, current-acl): Likewise.
* guix/scripts/archive.scm (authorize-key)[read-key]: Likewise.
* guix/scripts/authenticate.scm (read-canonical-sexp)
(read-hash-data): Likewise.
* guix/scripts/download.scm: Likewise.
* guix/scripts/offload.scm (register-gc-root, remove-gc-roots)
(send-files): Likewise.
* guix/scripts/publish.scm (lazy-read-file-sexp): Likewise.
* guix/scripts/refresh.scm: Likewise.
* guix/scripts/substitute.scm (check-acl-initialized): Likewise.
* guix/serialization.scm (read-maybe-utf8-string): Likewise.
* guix/scripts/hash.scm (guix-hash): Use 'force-output' instead of
'flush-output-port'.
* guix/store.scm (process-stderr): Likewise.
* guix/tests.scm: Likewise.
* guix/utils.scm: Use (ice-9 binary-ports) and autoload (rnrs io ports)
for 'make-custom-binary-input-port'.
2016-10-19 15:54:10 +02:00
Ludovic Courtès f82ce8f6b1
build: Substitute URLs now default to "mirror.hydra.gnu.org" alone.
* config-daemon.ac: Remove "hydra.gnu.org" from 'guix_substitute_urls'.
* guix/store.scm (%default-substitute-urls): Remove "hydra.gnu.org".
2016-07-16 14:41:39 +02:00
Ludovic Courtès 95fa173ed1
store: 'register-path' no longer swallows 'system-error' exceptions.
* guix/store.scm (register-path): Do not catch 'system-error'.
2016-06-20 00:52:53 +02:00
Ludovic Courtès 1ec32f4a9d
store: Add #:select? parameter to 'add-to-store'.
* guix/store.scm (write-arg): Remove 'file' case.
(true): New procedure.
(add-to-store): Add #:select? parameter and honor it.  Use hand-coded
stub instead of 'operation'.
(interned-file): Add #:select? parameter and honor it.
* doc/guix.texi (The Store Monad): Adjust 'interned-file' documentation
accordingly.
2016-06-15 15:26:30 +02:00
Ludovic Courtès f6fee16e93
store: 'requisites' now takes a list of store items.
* guix/store.scm (fold-path): Change 'path' to 'paths' and adjust body
accordingly.
(requisites): Likewise.
* guix/scripts/environment.scm (inputs->requisites): Adjust
user accordingly.
* guix/scripts/size.scm (requisites*): Likewise.
* guix/scripts/gc.scm (guix-gc): Likewise.
* tests/store.scm ("requisites"): Likewise.
2016-05-24 23:35:28 +02:00
Ludovic Courtès aa8fff0ceb
store: Clarify 'query-path-hash' docstring.
* guix/store.scm (query-path-hash): Clarify docstring.
2016-05-17 14:02:48 +02:00
Ludovic Courtès 958dd3ce68
utils: Move combinators to (guix combinators).
* guix/utils.scm (compile-time-value, memoize, fold2)
(fold-tree, fold-tree-leaves): Move to...
* guix/combinators: ... here.  New file.
* tests/utils.scm ("fold2, 1 list", "fold2, 2 lists")
(fold-tree tests): Move to...
* tests/combinators.scm: ... here.  New file.
* Makefile.am (MODULES, SCM_TESTS): Add them.
* gnu/packages.scm, gnu/packages/bootstrap.scm,
gnu/services/herd.scm, guix/build-system/gnu.scm,
guix/build-system/python.scm, guix/derivations.scm,
guix/gnu-maintenance.scm, guix/import/elpa.scm,
guix/scripts/archive.scm, guix/scripts/build.scm,
guix/scripts/graph.scm, guix/scripts/lint.scm,
guix/scripts/size.scm, guix/scripts/substitute.scm,
guix/serialization.scm, guix/store.scm, guix/ui.scm: Adjust imports
accordingly.
2016-05-04 23:35:55 +02:00
Ludovic Courtès 38f50f49f3 substitute: Install the client's locale.
* guix/store.scm (set-build-options): Add #:locale parameter and honor
it.
* guix/scripts/substitute.scm (guix-substitute): Install the client's
locale.
2016-04-20 23:16:47 +02:00
Ludovic Courtès b0a6a97130 substitute: Honor the number of columns of the client terminal.
* guix/store.scm (set-build-options): Add #:terminal-columns parameter
and honor it.
* guix/scripts/substitute.scm (client-terminal-columns): New procedure.
(guix-substitute): Use it to parameterize 'current-terminal-columns'.
2016-04-15 00:32:18 +02:00
Ludovic Courtès d70533cbe9 store: Prepend mirror.hydra.gnu.org to %DEFAULT-SUBSTITUTE-URLS.
This allows GuixSD to default to the right list of URLs, with
mirror.hydra.gnu.org coming first.

Reported by Chris Marusich <cmmarusich@gmail.com>.

* guix/store.scm (%default-substitute-urls): Prepend
"mirror.hydra.gnu.org."
2016-03-28 22:25:42 +02:00
Ludovic Courtès df061d079b build: Default to "https://mirror.hydra.gnu.org/" for substitutes.
* config-daemon.ac: Check for (gnutls) and define 'GUIX_SUBSTITUTE_URLS'.
* nix/nix-daemon/guix-daemon.cc (main): Use GUIX_SUBSTITUTE_URLS.
* guix/store.scm (%default-substitute-urls): Use 'https' when (gnutls)
is available.
* doc/guix.texi (Binary Installation): Mention mirrors
(Invoking guix-daemon): Mention mirror.hydra.gnu.org.
(Substitutes): Mention mirrors.
(Invoking guix archive): Show https URLs.
2016-03-16 11:27:37 +01:00
Ludovic Courtès f09aea1b58 store: 'references/substitutes' caches its results.
* guix/store.scm (%reference-cache): New variable.
(references/substitutes): Use it.
2016-03-14 23:34:33 +01:00
Ludovic Courtès dd78e90a4d store: 'references/substitutes' correctly handles the order of substitutes.
Before that, 'references/substitutes' would assume that
'substitutable-path-info' would return things in the same order as its
arguments, which is not the case.  Thus, it would sometimes provide
incorrect reference information, occasionally leading to infinite
loop (because dependency information would denote cycles.)

Fixes <http://bugs.gnu.org/22914>.
Reported by Eric Bavier <ericbavier@openmailbox.org>.

* guix/store.scm (references/substitutes): Make ITEMS the first argument
of the loop; match on it.  Use 'any' to find a matching substitute.
(substitutable-path-info): Clarify docstring about ordering.
2016-03-05 22:53:40 +01:00
Ludovic Courtès 6581ec9ab9 store: Add 'references/substitutes'.
* guix/store.scm (references/substitutes): New procedure.
* tests/store.scm ("references/substitutes missing reference info")
("references/substitutes with substitute info"): New tests.
2016-03-05 00:19:10 +01:00
Ludovic Courtès 22572d56cb store: 'path-info-deriver' is #f when there is no deriver.
* guix/store.scm (read-path-info): Use #f when we get the empty string
for DERIVER.
* guix/scripts/publish.scm (narinfo-string): Adjust accordingly.
* tests/store.scm ("path-info-deriver"): New test.
2016-02-26 23:35:29 +01:00
Ludovic Courtès 06b76acceb store: Clarify documentation of 'valid-path?'.
* guix/store.scm (valid-path?): Improve docstring.
* doc/guix.texi (The Store): Update accordingly.
2016-02-25 12:44:41 +01:00
Ludovic Courtès 98a7b528d6 store: Add monadic access to '%current-system'.
* guix/store.scm (current-system, set-current-system): New procedures.
* tests/store.scm ("current-system"): New test.
2016-02-12 21:54:25 +01:00
Ludovic Courtès 2fba87ac7c store: Allow clients to request multiple builds.
* guix/store.scm (set-build-options): Add #:rounds parameter and honor it.
* tests/store.scm ("build multiple times"): New test.
2015-12-08 23:58:12 +01:00
Ludovic Courtès 2734cbb895 store: 'build-things' now supports older daemon protocols.
This is a followup to d203d3d.

* guix/store.scm (build-things): Add 'build/old'.  Use it when STORE's
minor version is less than 15.
2015-12-04 11:34:18 +02:00
Ludovic Courtès 07e70f4846 store: Add mode parameter to 'build-paths'.
* guix/store.scm (%protocol-version): Set minor to 15.
(build-mode): New enumerate type.
(build-things): Add 'mode' parameter; pass it to the RPC.
* tests/store.scm ("build-things, check mode"): New check.
2015-12-03 19:09:53 +02:00
Ludovic Courtès d203d3d4cb store: Update to the new daemon protocol.
* guix/store.scm (%protocol-version): Set minor to 14.
(open-connection): Add 'cpu-affinity' parameter and honor it.
2015-12-03 19:09:53 +02:00
Ludovic Courtès fb4bf72be3 store: Use the daemon's substitute URLs by default.
Partly fixes <http://bugs.gnu.org/20217>.

* guix/store.scm (set-build-options): Change #:substitute-urls to
  default to #f.  Send the 'substitute-urls' pair only if
  SUBSTITUTE-URLS is true.
* guix/scripts/build.scm (set-build-options-from-command-line): Do not
  default to %DEFAULT-SUBSTITUTE-URLS for #:substitute-urls.
* guix/scripts/size.scm (%default-options): Remove 'substitute-urls'.
2015-10-28 15:31:44 +01:00
Ludovic Courtès 16748d8015 store: Add 'query-failed-paths' and 'clear-failed-paths' RPCs.
Suggested by Mark H Weaver <mhw@netris.org>.

* guix/store.scm (query-failed-paths, clear-failed-paths): New
  procedures.
* tests/guix-daemon.sh: Add test with daemon started with
  --cache-failures.
2015-09-28 22:54:04 +02:00
Alex Kost 4f740b677a store: Add 'set-build-options*'.
Suggested by: Ludovic Courtès <ludo@gnu.org>.

* guix/store.scm (set-build-options*): New procedure.
2015-09-18 21:42:07 +03:00
Ludovic Courtès f8a8e0fe16 guix build: Add '--substitute-urls' client option.
* guix/scripts/build.scm (%standard-build-options,
  show-build-options-help): Add --substitute-urls=URLS.
  (set-build-options-from-command-line): Honor it.
* guix/store.scm (%default-substitute-urls): Make public.
* doc/guix.texi (Substitutes): Add xref to the client --substitute-urls
  option.
  (Invoking guix build): Document it.
  (Invoking guix-daemon): Add 'daemon-substitute-urls' anchor.
2015-07-13 18:29:01 +02:00
Ludovic Courtès a8afb9aed3 store: 'run-with-store' initializes %CURRENT-TARGET-SYSTEM to #f.
* guix/store.scm (run-with-store): Set %CURRENT-TARGET-SYSTEM to #f.
* tests/gexp.scm ("gexp->derivation vs. %current-target-system"): New test.
2015-07-08 00:06:45 +02:00
Ludovic Courtès 69792b285c store: Memoize 'add-to-store' based on the result of 'lstat', not 'stat'.
* guix/store.scm (add-to-store): Change 'stat' call to 'lstat'.  Clarify
  docstring.
2015-06-19 10:23:29 +02:00
Ludovic Courtès c63d94035f store: Add 'verify-store' RPC.
* guix/store.scm (operation-id): Add 'verify-store'.
  (verify-store): New procedure.
  (set-build-options): Adjust comment.
* tests/store.scm ("verify-store", "verify-store + check-contents"): New
  tests.
2015-06-06 19:05:25 +02:00
Cyrill Schenkel cdb5b075d5 gc: ignore trailing slash or subdirectories for `guix gc -d'
Fixes <http://bugs.gnu.org/19757>.

* guix/scripts/gc.scm (guix-gc): Convert paths to direct store paths.
* guix/store.scm (direct-store-path): Get rid of subdirectories in store path.
* tests/guix-gc.sh: New tests.

Co-authored-by: Ludovic Courtès <ludo@gnu.org>
2015-05-26 22:26:01 +02:00