diff --git a/readme.html b/readme.html new file mode 100644 index 0000000..897ebfb --- /dev/null +++ b/readme.html @@ -0,0 +1,2238 @@ + + + + + + + +ob-julia: high quality julia org-mode support + + + + + + + +
+

ob-julia: high quality julia org-mode support

+
+

Table of Contents

+
+ +
+
+ +
+

1 ob-julia

+
+

+See 3 for more details. +

+
+
+ +
+

2 How it works

+
+
    +
  1. Code block is saved to a temporary file (under /tmp)
    2. +
  2. Decide whether we need to start a new julia process or not +
      +
    1. If session is "none", don't use a session (code under /tmp will +be passed to julia -L initfile src-block.jl). This does not +require ess. The command is customized by +org-babel-julia-command)
      2. +
    2. If session is nil, use default session name +(customized by org-babel-julia-default-session)
      3. +
    3. If session has a values, use it's name
      4. +
    3. +
  3. Check if we want to use a session or not. Check if the session +exists. Start the session accordingly.
    4. +
  4. Is the evaluation async? +
      +
    1. YES: +
        +
      1. Register a filter function
        2. +
      2. Return immediately, printing a ref to the evaluation. +The ref is in the form: julia-async:$uuid:$tmpfile +
          +
        • uuid: identifier of this evaluation, used to find out where to +insert again results
          • +
        • tmpfile: the path where the output will be saved. This is +useful both for debugging purposes and so that we do not need +to store an object that maps computations to files. The +process-filter look for the uuid, and then inserts $tmpfile.
          • +
        3. +
      2. +
    2. NO: Run the code, wait for the results. Insert the results.
      3. +
    5. +
+
+
+ +
+

3 Implemented features

+
+
+
+

3.1 Session (:session none, :session, :session session-name)

+
+

+By default code is executed without a session. The advantage is that +you do not requires emacs-ess to run julia code with ob-julia. But +sessions (especially for julia, where speed comes from compiled code) +are available. The same behaviour is obtained by setting the :session +header to none. +

+ +

+You can enable sessions with the :session argument. Without +parameters, the session used is named after +org-babel-julia-default-session (*julia* by default). With a +parameter, the name is earmuffed (a star is prepended and appended). +

+ +

+The REPL is kept sane. There's no cluttering (you don't see all the +code executed that is required by ob-julia to have results), history +is preserved (you can C-S-up to see the list of org-src block +evaluated), and results are shown (this can be customized by +org-babel-julia-silent-repl). +

+
+
+ +
+

3.2 Async (:async :async yes, :async t)

+
+

+Async works both with and without :session. +

+ +

+The best combination of features is combining session with +:async. Async allows code evaluation in background (you can continue +using emacs while julia compute results). +

+ +

+You can change buffer or even narrow it while the evaluation +completes: the result is added automatically as soon as julia +finishes. Multiple evaluation are queued automatically (thanks to +ess). Cache is supported (evaluating the same code-block twice does +not re-trigger evaluation, even if the code is still running). +

+ +

+It's not possible to have async blocks with :results silent. I'm +currently using this to distinguish between active src block and +variable resolution (when a :var refer to an async block, the block +cannot be executed asynchonously. So we need to distinguish between +the two. This is the only way I was able to find, if you know better +please tell me). +

+ +
+
sleep(1)
+"It works!"
+
+
+ +
+It works!
+
+ + +
+
sleep(1)
+"It works!"
+
+
+ +
+It works!
+
+ + +

+Here the same, without the session +

+
+
sleep(1)
+"It works!"
+
+
+ +
+It works!
+
+ + +
+
sleep(1)
+"It works!"
+
+
+ +
+It works!
+
+ + + +

+Asynchronous evaluation is automatically disabled on export, or when a +code block depends on one (:var) +

+
+
+ +
+

3.3 Variables input (:var), Standard output

+
+
+
+

3.3.1 Inputs

+
+

+Those are example inputs that will be used later, to check whether +import+export pipeline works as expected. +

+ +

+A table +

+ + + + +++ ++ + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ab
11
22
  
44
+ +

+A matrix (no hline) +

+ + + + +++ ++ ++ ++ + + + + + + + + + + + + + + + +
1234
1234
+ +

+A column +

+ + + + +++ + + + + + + + + + + + + + + + + + +
1
2
3
4
+ +

+A row +

+ + + + +++ ++ ++ ++ + + + + + + + + +
1234
+ +

+A list +

+ +
    +
  • 1
    • +
  • 2
    • +
  • 3
    • +
  • 4
    • +
+
+ +
    +
  1. Table
    +
    +
    +
    table
+
    +
    + + + + +++ ++ + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    ab
    11
    22
      
    44
    + +

    +As you can see, the table automatically adds the hline after the +header. This is a heuristic that might fail (might be triggered for +matrix, might not trigger on tables), so you can manually +force/disable it with the :results table or :results matrix param. +

    + +
    +
    table
+
    +
    + + + + +++ ++ + + + + + + + + + + + + + + + + + + + + + + + + + + +
    ab
    11
    22
      
    44
    +
    +
    2. + +
  2. Row
    +
    +

    +Column, Rows, and Matrix export works just fine (tests in session sync, session async +and without session). +

    + +
    +
    row
+
    +
    + + + + +++ ++ ++ ++ + + + + + + + + +
    1234
    + +
    +
    row
+
    +
    + + + + +++ ++ ++ ++ + + + + + + + + +
    1234
    + +
    +
    row
+
    +
    + + + + +++ ++ ++ ++ + + + + + + + + +
    1234
    +
    +
    3. + +
  3. Column
    +
    +

    +Works both with synchronous evaluation +

    + +
    +
    column
+
    +
    + + + + +++ + + + + + + + + + + + + + + + + + +
    1
    2
    3
    4
    + +

    +asynchronous evaluation +

    + +
    +
    column
+
    +
    + + + + +++ + + + + + + + + + + + + + + + + + +
    1
    2
    3
    4
    + +

    +and without a session +

    + +
    +
    column
+
    +
    + + + + +++ + + + + + + + + + + + + + + + + + +
    1
    2
    3
    4
    +
    +
    4. + +
  4. Matrix
    +
    +

    +Sync +

    + +
    +
    matrix
+
    +
    + + + + +++ ++ ++ ++ + + + + + + + + + + + + + + + +
    1234
    1234
    + +

    +Just like for tables, you can control header hline line with the +results param. +

    + +
    +
    matrix
+
    +
    + + + + +++ ++ ++ ++ + + + + + + + + + + + + + + + + +
    1234
    1234
    + +

    +Async +

    + +
    +
    matrix
+
    +
    + + + + +++ ++ ++ ++ + + + + + + + + + + + + + + + +
    1234
    1234
    + +

    +No session +

    + +
    +
    matrix
+
    +
    + + + + +++ ++ ++ ++ + + + + + + + + + + + + + + + + +
    1234
    1234
    +
    +
    5. + +
  5. List
    +
    +

    +List are parsed as columns +

    + +
    +
    list
+
    +
    + +

    +:results list return the list (just like R). It's not perfect with +

    +
    +
    list
+
    +
    + +
      +
    • (1)
      • +
    • (2)
      • +
    • (3)
      • +
    • (4)
      • +
    +
    +
    6. + +
  6. Table
    +
    +

    +There are two ways in which tables can be passed to Julia: +

    +
      +
    • Array{NamedTuple}
      • +
    • Dictionary
      • +
    + +

    +I like the NamedTuple approach, but if you don't like it you can +customize the variable org-babel-julia-table-as-dict. In both cases, +if you :import DataFrames, you can construct a DataFrame from both. +

    + +

    +TOOD: I miss the julia code for printing Array{NamedTuple}. +

    + +
    +
    table
+
    +
    + + + + +++ ++ + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    ab
    11
    22
      
    44
    + +

    +Also, it's nice that a single NamedTuple can represent a table: +

    +
    +
    table[2]
+
    +
    + + + + +++ ++ + + + + + + + + + + + + +
    ab
    22
    +
    +
    7. +
+
+
+ +
+

3.4 Directory (:dir)

+
+

+Each source block is evaluated in it's :dir param +

+ +
+
pwd()
+
+
+ +
+/tmp
+
+ + +
+
pwd()
+
+
+ +
+/
+
+ + +

+If unspecified, the directory is session's one +

+
+
pwd()
+
+
+ +
+/home/nixo/dotfiles/emacs/.emacs.d/extra/ob-julia
+
+ + +

+Changing dir from julia code still works +

+
+
cd("/")
+realpath(".")
+
+
+ +
+/
+
+ + +

+but is ephemeral (like fort the :dir param) +

+
+
realpath(".")
+
+
+ +
+/home/nixo/dotfiles/emacs/.emacs.d/extra/ob-julia
+
+ + +

+This is obtained by wrapping the src block in a cd() call: +

+
+
cd(folder) do
+   block
+end
+
+
+
+
+ +
+

3.5 Error management

+
+

+If the block errors out, +

+ +
+
x
+
+
+ + + + +++ ++ + + + + + + +
ERRORUndefVarError(:x)
+ +
+
1 + "ciao"
+
+
+ + + + +++ ++ ++ ++ ++ + + + + + + + + + +
ERRORMethodError(+(1ciao)0x0000000000006426)
+ + + + +++ ++ ++ ++ ++ + + + + + + + + + +
ERRORMethodError(+(1ciao)0x0000000000006420)
+ +

+It works in async +

+
+
x
+
+
+ + + + +++ ++ + + + + + + +
ERRORUndefVarError(:x)
+ +

+On external process (sync) +

+
+
x
+
+
+ + + + +++ ++ + + + + + + +
ERRORUndefVarError(:x)
+ +

+and on external process (async) +

+
+
x
+
+
+ + + + +++ ++ + + + + + + +
ERRORUndefVarError(:x)
+ +

+Error management can still be improved for helping with debug (see +scimax). +

+
+
+ +
+

3.6 Using (:using) and Import (:import)

+
+

+To include dependencies, you can use :using and :import. +

+ +

+Because of how the julia code is actually implemented, in order to use +specialized exports (e.g., DataFrames, see ) you need the +modules to be available before the block gets evaluated. The problem +can be solved in 2 (or 3 ways): +

+
    +
  • Evaluating a block with using/import, then the other block
    • +
  • Using the header arguments
    • +
  • Fixing the Julia code :D
    • +
+ +

+to use :import, you need to pass arguments quoted: +

+
+:using DataFrames Query :import "FileIO: load" "Plots: plot"
+
+
+
+ +
+

3.7 Results (:results output, :results file, )

+
+

+The default is to return the value: +

+ +
+
10
+
+
+ +
+10
+
+ + +

+If results is output, it's included the stdout (what's printed in the +terminal). (This part still needs some work to be useful.) +

+ +
+
10
+
+
+ +
+
println(10)
+
+
+ +
+10
+
+ + +
+
println("a")
+
+"10"
+
+println("b")
+
+
+ +
+a
+b
+
+ + +

+Error (results output) +

+ +
+
This will throw an error
+
+
+ +
+Base.Meta.ParseError("extra token \"will\" after end of expression")
+
+ + +

+Another error (result ouptut) +

+
+
print(one(3,3))
+
+
+ +
+MethodError(one, (3, 3), 0x0000000000006426)
+
+ + +

+A matrix +

+
+
print(ones(3,3))
+
+
+ +
+[1.0 1.0 1.0; 1.0 1.0 1.0; 1.0 1.0 1.0]
+
+
+
+ +
+

3.8 Supported Types

+
+

+Adding new types is easy (you need to define an orgshow() function for +your type. See init.jl). There's a simple mechanism that allows to +define method on non-yet-existing types example. +

+ +

+The current version supports a couple of useful type: DataFrames and +Plots. ob-julia needs community support to add more types: please help! +

+
+
+ +
+

3.9 File output & Inlining

+
+

+There's native support for writing output to file. For unkown file +types, instead of inserting the output in the buffer it's written to the file. +

+ +
+
zeros(3,3)
+
+
+ +

+readme/output.html +

+ +
+
zeros(3,3)
+
+
+ +

+readme/output.csv +

+ +
+
Dict(10 => 10)
+
+
+ +

+readme/output.csv +

+ +

+Saving plots requires the Plots library. You can require it with the +:using header. There's the custom :size "tuple" header argument for +specifying the output size. It must be placed inside parentheses, and +it's evaluated as julia object (that means it can contains variables +and expressions). +

+ +
+
plot(matrix)
+
+
+ + +
+

output-plot.png +

+
+ +

+Matrix also has an automatic conversion (when Plots is loaded), so you +don't even need to pass it to the plot() function (there's a generic +fallback that tries to plot anything saved to png or svg). +

+ +
+
matrix
+
+
+ + +
+

+Sorry, your browser does not support SVG. +

+
+ +

+Plots can also manage errors (in a visually-disturbing way). +

+ +
+
this_is_undefined
+
+
+ + +
+

+Sorry, your browser does not support SVG. +

+
+ +
+
another_undefined_but_async
+
+
+ + +
+

output-undef.png +

+
+ +
+
DataFrame(x = 1:10, y = (0:9) .+ 'a')
+
+
+ + + + +++ ++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
xy
1a
2b
3c
4d
5e
6f
7g
8h
9i
10j
+ +
+
DataFrame(table)
+
+
+ +

+readme/output-table.csv +

+ +
+
data
+
+
+ + + + +++ ++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
xy
1a
2b
3c
4d
5e
6f
7g
8h
9i
10j
+
+ +
+

3.9.1 Inlining (:inline no, :inline, :inline format)

+
+

+Output to file and Inlining are different things but nicely fit +together to solve the same problem: inserting images or other objects +inside the buffer. +

+ +

+If your type can be represented inside the exported format (like +images as svg/base-64 encoded pngs inside .html, tex plots in a .tex +file), the :inline header is what you need. The behaviour changes +based depending on your interactive use and on the desired output +format. +

+ +

+TODO: right now :results raw is required on export. How do we fix it? +

+ +

+Examples: :inline keyword alone, in interactive evaluation, the output +inserted in the buffer is the usual. +

+
+
matrix
+
+
+ +

+
1234
1234
+

+ +

+But when you export the output to html, the output will be processed +by julia, and inserted in the buffer (or a different representation +for different export formats). This is not of much use with tables +(even if you can customize the export, e.g. by passing the :width +keyword), but is wonderful for pictures. If a result can be inserted +in multiple ways (picture in html can be both inline png or svg), you +can specify the desired format by passing the argument to the :inline +keyword (like :inline svg). In this case, the processed output is +inserted also in interactive sessions. +

+ +
+
matrix
+
+
+ +

+
1234
1234
+

+ +

+Plots default to inline png +

+ +
+
plot(matrix)
+
+
+ +

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +1.00 + + +1.25 + + +1.50 + + +1.75 + + +2.00 + + +1 + + +2 + + +3 + + +4 + + + + + + + + + +y1 + + + +y2 + + + +y3 + + + +y4 + + + +

+ + +

+But you can also force svg (Since it's multiline, :wrap it with :wrap html) +

+ +
+
plot(matrix)
+
+
+ +

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +1.00 + + +1.25 + + +1.50 + + +1.75 + + +2.00 + + +1 + + +2 + + +3 + + +4 + + + + + + + + + +y1 + + + +y2 + + + +y3 + + + +y4 + + + +

+
+
+
+
+ + +
+

4 Issues and Missing features

+
+
    +
  • No automated tests yet
    • +
  • Not tested with remote host
    • +
  • Variable resolution of src-block is synchronous. If your :async src +block depends on another :async src block, the latter is evaluated +synchronously, then former asynchonously. This could be implemented +by using a simple queue, where next item is started in the +process-filter and where variables starting with julia-async: are +replaced. Right now I don't feel the need (if results are cached, +things already feels good).
    • +
  • For async evaluation to work the session must be started from +ob-julia (or you must register the filter function manually, +undocumented).
    • +
  • :results output is implemented but untested. I rarely find it +useful.
    • +
  • import/using errors are not reported
    • +
+
+
+ +
+

5 Credits

+
+

+This project originally started as a fork of +astahlman/ob-async. However, because of changes in new version of Org +Mode, julia 1.0 release and unsupported features, I decided to start +over. +

+
+
+
+
+

Author: Nicolò Balzarotti

+

Created: 2019-10-29 Tue 11:56

+

Validate

+
+ + diff --git a/readme/output-matrix.svg b/readme/output-matrix.svg new file mode 100644 index 0000000..51766c6 --- /dev/null +++ b/readme/output-matrix.svg @@ -0,0 +1,153 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +1.00 + + +1.25 + + +1.50 + + +1.75 + + +2.00 + + +1 + + +2 + + +3 + + +4 + + + + + + + + + +y1 + + + +y2 + + + +y3 + + + +y4 + + diff --git a/readme/output-plot.png b/readme/output-plot.png new file mode 100644 index 0000000..8fad026 Binary files /dev/null and b/readme/output-plot.png differ diff --git a/readme/output-table.csv b/readme/output-table.csv new file mode 100644 index 0000000..026d3b7 --- /dev/null +++ b/readme/output-table.csv @@ -0,0 +1,5 @@ +a,b +1,1 +2,2 +, +4,4 \ No newline at end of file diff --git a/readme/output-undef.png b/readme/output-undef.png new file mode 100644 index 0000000..6848523 Binary files /dev/null and b/readme/output-undef.png differ diff --git a/readme/output-undef.svg b/readme/output-undef.svg new file mode 100644 index 0000000..616acfa --- /dev/null +++ b/readme/output-undef.svg @@ -0,0 +1,27 @@ + + + + + + + + + + + + + + + + + + + + +ERROR: UndefVarError(:this_is_undefined) + + diff --git a/readme/output.csv b/readme/output.csv new file mode 100644 index 0000000..8e6a436 --- /dev/null +++ b/readme/output.csv @@ -0,0 +1 @@ +Dict(10=>10) \ No newline at end of file diff --git a/readme/output.html b/readme/output.html new file mode 100644 index 0000000..4f75efb --- /dev/null +++ b/readme/output.html @@ -0,0 +1 @@ +
0.00.00.0
0.00.00.0
0.00.00.0
\ No newline at end of file