ob-julia: high quality julia org-mode support

• ob-julia
• How it works
• Implemented features
  • Session (:session none, :session, :session session-name)
  • Async (:async :async yes, :async t)
  • Variables input (:var =:let), Standard output
    • Inputs
      • Table
      • Row
      • Column
      • Matrix
      • List
      • Table
  • Directory (:dir)
  • Error management
  • Using (:using) and Import (:import)
  • Results (:results output, :results file, )
  • Supported Types
  • File output & Inlining
    • Inlining (:inline no, :inline, :inline format)
• Issues and Missing features
• Credits

ob-julia

See /nixo/ob-julia/src/branch/master/Implemented%20features for more details.

How it works

1. Code block is saved to a temporary file (under /tmp)

2. Decide whether we need to start a new julia process or not

   1. If session is "none", don't use a session (code under /tmp will be passed to julia -L initfile src-block.jl). This does not require ess. The command is customized by org-babel-julia-command)
   2. If session is nil, use default session name (customized by org-babel-julia-default-session)
   3. If session has a values, use it's name
3. Check if we want to use a session or not. Check if the session exists. Start the session accordingly. During startup, the file defined in ob-julia-startup-script is loaded. This file must define functions like OrgBabelFormat, to let emacs send code to execute. See init.jl for details.

4. Is the evaluation async?

   1. YES:

      1. Register a filter function

      2. Return immediately, printing a ref to the evaluation. The ref is in the form: julia-async:$uuid:$tmpfile

         • uuid: identifier of this evaluation, used to find out where to insert again results
         • tmpfile: the path where the output will be saved. This is useful both for debugging purposes and so that we do not need to store an object that maps computations to files. The process-filter look for the uuid, and then inserts $tmpfile.
   2. NO: Run the code, wait for the results. Insert the results.

Implemented features

Session (:session none, :session, :session session-name)

By default code is executed without a session. The advantage is that you do not requires emacs-ess to run julia code with ob-julia. But sessions (especially for julia, where speed comes from compiled code) are available. The same behaviour is obtained by setting the :session header to none.

You can enable sessions with the :session argument. Without parameters, the session used is named after org-babel-julia-default-session (*julia* by default). With a parameter, the name is earmuffed (a star is prepended and appended).

The REPL is kept sane. There's no cluttering (you don't see all the code executed that is required by ob-julia to have results), history is preserved (you can C-S-up to see the list of org-src block evaluated), and results are shown (this can be customized by org-babel-julia-silent-repl).

Async (:async :async yes, :async t)

Async works both with and without :session.

The best combination of features is combining session with :async. Async allows code evaluation in background (you can continue using emacs while julia compute results).

You can change buffer or even narrow it while the evaluation completes: the result is added automatically as soon as julia finishes. Multiple evaluation are queued automatically (thanks to ess). Cache is supported (evaluating the same code-block twice does not re-trigger evaluation, even if the code is still running).

It's not possible to have async blocks with :results silent. I'm currently using this to distinguish between active src block and variable resolution (when a :var refer to an async block, the block cannot be executed asynchonously. So we need to distinguish between the two. This is the only way I was able to find, if you know better please tell me).

sleep(1)
"It works!"

sleep(1)
"It works!"

Here the same, without the session

sleep(1)
"It works!"

sleep(1)
"It works!"

Asynchronous evaluation is automatically disabled on export, or when a code block depends on one (:var)

Variables input (:var =:let), Standard output

As usual, you can set variables with the :var header. To make ob-julia behave like the julia REPL, variables are set in the global scope. If you want a block to be isolated, you can use the extra :let header argument: with this, the src block is inside a let block:

let vars
  src_block
end

Inputs

Those are example inputs that will be used later, to check whether import+export pipeline works as expected.

A table

a	b
1	1
2	2
4	4

A matrix (no hline)

1	2	3	4
1	2	3	4

A column

1

2

3

4

A row

1

2

3

4

A list

• 1
• 2
• 3
• 4

Table

table

As you can see, the table automatically adds the hline after the header. This is a heuristic that might fail (might be triggered for matrix, might not trigger on tables), so you can manually force/disable it with the :results table or :results matrix param.

table

Row

Column, Rows, and Matrix export works just fine (tests in session sync, session async and without session).

row

row

row

Column

Works both with synchronous evaluation

column

asynchronous evaluation

column

and without a session

column

Matrix

Sync

matrix

Just like for tables, you can control header hline line with the results param.

matrix

Async

matrix

No session

matrix

List

List are parsed as columns

list

:results list return the list (just like R). It's not perfect with

list

Table

There are two ways in which tables can be passed to Julia:

• Array{NamedTuple}
• Dictionary

I like the NamedTuple approach, but if you don't like it you can customize the variable org-babel-julia-table-as-dict. In both cases, if you :import DataFrames, you can construct a DataFrame from both.

TOOD: I miss the julia code for printing Array{NamedTuple}.

table

Also, it's nice that a single NamedTuple can represent a table:

table[2]

Directory (:dir)

Each source block is evaluated in it's :dir param

pwd()

pwd()

If unspecified, the directory is session's one

pwd()

Changing dir from julia code still works

cd("/")
realpath(".")

but is ephemeral (like fort the :dir param)

realpath(".")

This is obtained by wrapping the src block in a cd() call:

cd(folder) do
   block
end

Error management

If the block errors out,

x

1 + "ciao"

It works in async

x

On external process (sync)

x

and on external process (async)

x

Error management can still be improved for helping with debug (see scimax).

Using (:using) and Import (:import)

To include dependencies, you can use :using and :import.

Because of how the julia code is actually implemented, in order to use specialized exports (e.g., DataFrames, see ) you need the modules to be available before the block gets evaluated. The problem can be solved in 2 (or 3 ways):

• Evaluating a block with using/import, then the other block
• Using the header arguments
• Fixing the Julia code :D

to use :import, you need to pass arguments quoted:

:using DataFrames Query :import "FileIO: load" "Plots: plot"

Results (:results output, :results file, )

The default is to return the value:

10

If results is output, it's included the stdout (what's printed in the terminal). (This part still needs some work to be useful.)

10

println(10)

println("a")

"10"

println("b")

Error (results output)

This will throw an error

Another error (result ouptut)

print(one(3,3))

A matrix

print(ones(3,3))

Supported Types

Adding new types is easy (you need to define an orgshow() function for your type. See init.jl). There's a simple mechanism that allows to define method on non-yet-existing types example.

The current version supports a couple of useful type: DataFrames and Plots. ob-julia needs community support to add more types: please help!

File output & Inlining

There's native support for writing output to file. For unkown file types, instead of inserting the output in the buffer it's written to the file.

zeros(3,3)

zeros(3,3)

sleep(1)
Dict(10 => 10)

Saving plots requires the Plots library. You can require it with the :using header. There's the custom :size "tuple" header argument for specifying the output size. It must be placed inside parentheses, and it's evaluated as julia object (that means it can contains variables and expressions).

plot(matrix)

Matrix also has an automatic conversion (when Plots is loaded), so you don't even need to pass it to the plot() function (there's a generic fallback that tries to plot anything saved to png or svg).

matrix

Plots can also manage errors (in a visually-disturbing way).

this_is_undefined

another_undefined_but_async

DataFrame(x = 1:10, y = (0:9) .+ 'a')

DataFrame(table)

data

Inlining (:inline no, :inline, :inline format)

Output to file and Inlining are different things but nicely fit together to solve the same problem: inserting images or other objects inside the buffer.

If your type can be represented inside the exported format (like images as svg/base-64 encoded pngs inside .html, tex plots in a .tex file), the :inline header is what you need. The behaviour changes based depending on your interactive use and on the desired output format.

TODO: right now :results raw is required on export. How do we fix it?

Examples: :inline keyword alone, in interactive evaluation, the output inserted in the buffer is the usual.

matrix

But when you export the output to html, the output will be processed by julia, and inserted in the buffer (or a different representation for different export formats). This is not of much use with tables (even if you can customize the export, e.g. by passing the :width keyword), but is wonderful for pictures. If a result can be inserted in multiple ways (picture in html can be both inline png or svg), you can specify the desired format by passing the argument to the :inline keyword (like :inline svg). In this case, the processed output is inserted also in interactive sessions.

matrix

Plots default to inline png

plot(matrix)

But you can also force svg (Since it's multiline, :wrap it with :wrap html)

plot(matrix)

sleep(10)
rand(100,100)

/nixo/ob-julia/src/branch/master/deleteme.csv

Issues and Missing features

• No automated tests yet
• Not tested with remote host
• Variable resolution of src-block is synchronous. If your :async src block depends on another :async src block, the latter is evaluated synchronously, then former asynchonously. This could be implemented by using a simple queue, where next item is started in the process-filter and where variables starting with julia-async: are replaced. Right now I don't feel the need (if results are cached, things already feels good).
• For async evaluation to work the session must be started from ob-julia (or you must register the filter function manually, undocumented).
• :results output is implemented but untested. I rarely find it useful.
• import/using errors are not reported

Credits

This project originally started as a fork of astahlman/ob-async. However, because of changes in new version of Org Mode, julia 1.0 release and unsupported features, I decided to start over.