Merge pull request #353 from JunoLab/avi/rmarkdownchunks

RMarkdown style chunk options

doc/make.jl

@@ -14,8 +14,8 @@ makedocs(
         "getting_started.md",
         "usage.md",
         "publish.md",
-        "header.md",
         "chunk_options.md",
+        "header.md",
         "notebooks.md",
         "function_index.md",
     ],

doc/src/chunk_options.md

@@ -1,43 +1,65 @@
-# Chunk Options
+# [Chunk Options](@id chunk-options)
 
-I've mostly followed [Knitr](http://yihui.name/knitr/options)'s naming for chunk options, but not all options are implemented.
-
-Options are separated using ";" and need to be valid Julia expressions. Example: markdown code chunk that saves and displays a 12 cm wide image and hides the source code:
-
-`julia; out_width="12cm"; echo=false`
-
-Weave currently supports the following chunk options with the following defaults:
+You can use chunk options to configure how each chunk is evaluated, rendered, etc.
+Most of the ideas came from [chunk options in RMarkdown](http://yihui.name/knitr/options).
 
 
-## Options for Code
+## Syntax
+
+Chunk options come after code chunk starter at the top of [code chunk](@ref code-chunks).
+There are two (slightly) different syntax to write them:
+- (Julia's toplevel expression) options are separated by semicolon (`;`)
+- (RMarkdown style) options are separated by comma (`,`)
+
+Let's take a look at examples. All the following code chunk header are valid,
+and so configured to hide the source code from generated output (`echo = false`)
+and displays figures with 12cm width (`out_width = "12cm"`):
+```md
+ ```julia; echo = false; out_width = "12cm"
+
+ ```{julia; echo = false; out_width = "12cm"}
+
+ ```julia, echo = false, out_width = "12cm"
+
+ ```{julia, echo = false, out_width = "12cm"}
+```
+
+
+## Weave Chunk Options
+
+Weave currently supports the following chunk options:
+we've mostly followed [RMarkdown's namings](http://yihui.name/knitr/options), but not all options are implemented.
+
+### Evaluation
+
+- `eval = true`: Evaluate the code chunk. If `false` the chunk won’t be executed.
+- `cache = false`: Cache results, depending on `cache` parameter on [`weave`](@ref) function.
+- `tangle = true`: Set tangle to `false` to exclude chunk from tangled code.
+
+### Rendering
 
 - `echo = true`: Echo the code in the output document. If `false` the source code will be hidden.
 - `results = "markup"`: The output format of the printed results. `"markup"` for literal block, `"hidden"` for hidden results, or anything else for raw output (I tend to use `"tex"` for Latex and `"rst"` for rest). Raw output is useful if you want to e.g. create tables from code chunks.
-- `eval = true`: Evaluate the code chunk. If `false` the chunk won’t be executed.
 - `term = false`: If `true` the output emulates a REPL session. Otherwise only stdout and figures will be included in output.
-- `label = nothing`: Chunk label, will be used for figure labels in Latex as `fig:label`.
 - `wrap = true`: Wrap long lines from output.
 - `line_width = 75`: Line width for wrapped lines.
-- `cache = false`: Cache results, depending on `cache` parameter on `weave` function.
 - `hold = false`: Hold all results until the end of the chunk.
-- `tangle = true`: Set tangle to `false` to exclude chunk from tangled code.
 
+### Figures
 
-## Options for Figures
-
+- `label = nothing`: Chunk label, will be used for figure labels in Latex as `fig:label`.
 - `fig_width = 6`: Figure width passed to plotting library.
 - `fig_height = 4`: Figure height passed to plotting library.
 - `out_width`: Width of saved figure in output markup e.g. `"50%"`, `"12cm"`, `0.5\linewidth`
 - `out_height`: Height of saved figure in output markup
 - `dpi = 96`: Resolution of saved figures.
 - `fig_cap`: Figure caption.
-- `label`: Chunk label, will be used for figure labels in Latex as fig:label
 - `fig_ext`: File extension (format) of saved figures.
 - `fig_pos = "!h"`: Figure position in Latex, e.g.: `"ht"`.
 - `fig_env = "figure"`: Figure environment in Latex.
 
 
-## Set Default Chunk Options
+## Default Chunk Options
 
 You can set the default chunk options (and `weave` arguments) for a document using `weave_options` key in YAML [Header Configuration](@ref).
 E.g. to set the default `out_width` of all figures you can use:

doc/src/index.md

@@ -30,8 +30,8 @@ Pages = [
     "getting_started.md",
     "usage.md",
     "publish.md",
-    "header.md",
     "chunk_options.md",
+    "header.md",
     "notebooks.md",
     "function_index.md",
 ]

doc/src/usage.md

@@ -73,62 +73,69 @@ ext == ".ipynb" && return "notebook"
 return "noweb"
 ```
 
-## Documentation Chunks
+
+### Documentation Chunks
 
 In markdown and Noweb input formats documentation chunks are the parts that aren't inside code delimiters. Documentation chunks can be written with several different markup languages.
 
-## Code Chunks
 
-### Markdown Format
+### [Code Chunks](@id code-chunks)
 
-Markdown code chunks are defined using fenced code blocks with options following on the same line. e.g. to hide code from output you can use:
+Code chunks are written in different ways in different formats.
 
-```
- ```julia; echo=false
+#### Markdown Format
+
+Weave code chunks are defined using fenced code blocks, same as with [common markdown](https://spec.commonmark.org/0.29/#fenced-code-blocks):
+```markdown
+ ```julia
+ code
+ ...
+ ```
 ```
 
-[Sample document]( https://github.com/mpastell/Weave.jl/blob/master/examples/FIR_design.jmd)
+Weave code chunks can optionally be followed by [chunk options](@ref) on the same line.
+E.g. the chunk below will hide code itself from generated output:
+```markdown
+ ```julia, echo = false
+ code
+ ...
+ ```
+```
 
-## [Inline Code](@id inline-code)
+#### Noweb Format
+
+Code chunks start with a line marked with `<<>>=` or `<<options>>=` and end with line marked with `@`.
+The code between the start and end markers is executed and the output is captured to the output document.
+
+### [Inline Code](@id inline-code)
 
 You can also add inline code to your documents using
-
 ```
 `j juliacode`
 ```
-
 or
-
 ```
 ! juliacode
 ```
+syntax.
 
-syntax. Using the `j code` syntax you can insert code anywhere in a line and with
-the `!` syntax the whole line after `!` will be executed. The code will be replaced
-with captured output in the weaved document.
-
-If the code produces figures the filename or base64 encoded string will be
-added to output e.g. to include a Plots figure in markdown you can use:
+The former syntax allows you to insert code _anywhere_ in a line
+while the `!` syntax treats the whole line as code,
+and the code will be replaced with captured output in the weaved document.
 
+If the code produces figures, the filename or base64 encoded string will be added to output,
+e.g. to include a Plots figure in markdown you can use:
 ```
 ![A plot](`j plot(1:10)`)
 ```
-
-or to produce any html output:
-
+or to produce any HTML output:
 ```
 ! display("text/html", "Header from julia");
 ```
 
-
-### Noweb Format
-
-Code chunks start with a line marked with `<<>>=` or `<<options>>=` and end with line marked with `@`. The code between the start and end markers is executed and the output is captured to the output document. See [chunk options](../chunk_options/).
-
-
 ### Script Format
 
-Weave also support script input format with a markup in comments.
+Weave also supports script input format with a markup in comments.
 These scripts can be executed normally using Julia or published with Weave.
 
 Lines starting with `#'`, `#%%` or `# %%` are treated as document.
@@ -143,7 +150,11 @@ hoge # some code comes here
 The format is identical to [Pweave](http://mpastell.com/pweave/pypublish.html) and the concept is similar to publishing documents with MATLAB or using Knitr's [spin](http://yihui.name/knitr/demo/stitch/).
 Weave will remove the first empty space from each line of documentation.
 
-[See sample document:](https://github.com/mpastell/Weave.jl/blob/master/examples/FIR_design.jl)
+!!! tip
+    - Here are sample documents:
+      + [markdown format](https://github.com/mpastell/Weave.jl/blob/master/examples/FIR_design.jmd)
+      + [script format](https://github.com/mpastell/Weave.jl/blob/master/examples/FIR_design.jl)
+    - [Details about chunk options](@ref chunk-options)
 
 
 ## Configuration via YAML Header

src/Weave.jl

@@ -19,6 +19,7 @@ function __init__()
     @require Gadfly = "c91e804a-d5a3-530f-b6f0-dfbca275c004" include("gadfly.jl")
 end
 
+# utilitity functions
 @static @isdefined(isnothing) || begin
     isnothing(::Any) = false
     isnothing(::Nothing) = true

src/reader/markdown.jl

@@ -5,7 +5,7 @@ function parse_markdown(document_body; is_pandoc = false)
         r"^<<(?<options>.*?)>>=\s*$",
         r"^@\s*$"
     else
-        r"^[`~]{3}(?:\{?)julia(?:;?)\s*(?<options>.*?)(\}|\s*)$",
+        r"^[`~]{3}(\{?)julia\s*([;,]?)\s*(?<options>.*?)(\}|\s*)$",
         r"^[`~]{3}\s*$"
     end
     return header, parse_markdown_body(document_body, code_start, code_end, offset)
@@ -55,22 +55,17 @@ function parse_markdown_body(document_body, code_start, code_end, offset)
     content = ""
     start_line = offset
 
-    options = Dict()
+    options = OptionDict()
     option_string = ""
+
     chunks = WeaveChunk[]
     for (line_no, line) in enumerate(lines)
         m = match(code_start, line)
         if !isnothing(m) && state === :doc
             state = :code
-            option_string = isnothing(m[:options]) ? "" : strip(m[:options])
 
-            options = Dict{Symbol,Any}()
-            if !isempty(option_string)
-                expr = Meta.parse(option_string)
-                Base.Meta.isexpr(expr, :(=)) && (options[expr.args[1]] = expr.args[2])
-                Base.Meta.isexpr(expr, :toplevel) &&
-                    map(pushopt, fill(options, length(expr.args)), expr.args)
-            end
+            option_string = isnothing(m[:options]) ? "" : strip(m[:options])
+            options = parse_options(option_string)
             haskey(options, :label) && (options[:name] = options[:label])
             haskey(options, :name) || (options[:name] = nothing)
 

src/reader/reader.jl

@@ -64,12 +64,6 @@ function parse_doc(document, informat)
         error("unsupported input format given: $informat")
 end
 
-function pushopt(options::Dict, expr::Expr)
-    if Base.Meta.isexpr(expr, :(=))
-        options[expr.args[1]] = expr.args[2]
-    end
-end
-
 # inline
 # ------
 
@@ -100,6 +94,34 @@ end
 
 parse_inline(str) = Inline[InlineText(str, 1)]
 
+# options
+# -------
+
+const OptionDict = Dict{Symbol,Any}
+
+function parse_options(str)::OptionDict
+    str = string('(', str, ')')
+    ex = Meta.parse(str)
+    nt = if Meta.isexpr(ex, (
+        :block, # "(k1 = v1; k2 = v2, ...)"
+        :tuple, # "(k1 = v1, k2 = v2, ...)"
+    ))
+        eval(Expr(:tuple, filter(is_valid_kv, ex.args)...))
+    elseif is_valid_kv(ex) # "(k = v)"
+        eval(Expr(:tuple, ex))
+    else
+        NamedTuple{}()
+    end
+    return dict(nt)
+end
+
+is_valid_kv(x) = Meta.isexpr(x, :(=))
+dict(nt) = Dict((k => v for (k,v) in zip(keys(nt), values(nt)))...)
+nt(dict) = NamedTuple{(Symbol.(keys(dict))...,)}((collect(values(dict))...,))
+
+# each input format
+# -----------------
+
 include("markdown.jl")
 include("script.jl")
 include("notebook.jl")

src/reader/script.jl

@@ -12,14 +12,14 @@ function parse_script(document_body)
     code_no = 0
     start_line = 1
 
-    options = Dict{Symbol,Any}()
+    options = OptionDict()
     option_string = ""
-    chunks = WeaveChunk[]
 
+    chunks = WeaveChunk[]
     for (line_no, line) in enumerate(lines)
         if (m = match(doc_line, line)) !== nothing && (m = match(opt_line, line)) === nothing
             line = replace(line, doc_start => "", count = 1)
-            startswith(line, ' ') && (line = line[2:end])
+            startswith(line, ' ') && (line = replace(line, ' ' => "", count = 1))
             if state === :code && !isempty(strip(content))
                 push!(chunks, CodeChunk(string('\n', strip(content)), code_no += 1, start_line, option_string, options))
                 content = ""
@@ -39,13 +39,7 @@ function parse_script(document_body)
             end
 
             option_string = replace(line, opt_start => "", count = 1)
-            options = Dict{Symbol,Any}()
-            if !isempty(option_string)
-                expr = Meta.parse(option_string)
-                Base.Meta.isexpr(expr, :(=)) && (options[expr.args[1]] = expr.args[2])
-                Base.Meta.isexpr(expr, :toplevel) &&
-                    map(pushopt, fill(options, length(expr.args)), expr.args)
-            end
+            options = parse_options(option_string)
             haskey(options, :label) && (options[:name] = options[:label])
             haskey(options, :name) || (options[:name] = nothing)
 

src/types.jl

@@ -25,30 +25,31 @@ struct ChunkOutput
 end
 
 mutable struct CodeChunk <: WeaveChunk
-    content::AbstractString
+    content::String
     number::Int
     result_no::Int
     start_line::Int
-    optionstring::AbstractString
+    optionstring::String
     options::Dict{Symbol,Any}
     output::AbstractString
     rich_output::AbstractString
-    figures::Vector{AbstractString}
+    figures::Vector{String}
     result::Vector{ChunkOutput}
-    function CodeChunk(content, number, start_line, optionstring, options)
-        new(
-            string(rstrip(content), '\n'), # normalize end of chunk
-            number,
-            0,
-            start_line,
-            optionstring,
-            options,
-            "",
-            "",
-            AbstractString[],
-            ChunkOutput[],
-        )
-    end
+end
+
+function CodeChunk(content, number, start_line, optionstring, options)
+    return CodeChunk(
+        string(rstrip(content), '\n'), # normalize end of chunk)
+        number,
+        0,
+        start_line,
+        optionstring,
+        options,
+        "",
+        "",
+        AbstractString[],
+        ChunkOutput[]
+    )
 end
 
 mutable struct DocChunk <: WeaveChunk

test/chunk_options.jl

@@ -1,20 +0,0 @@
-using Weave
-using Test
-
-cleanup = true
-
-VER = "$(VERSION.major).$(VERSION.minor)"
-
-Weave.push_preexecute_hook(identity)
-weave("documents/chunk_options.noweb")
-Weave.pop_preexecute_hook(identity)
-result =  read("documents/chunk_options.md", String)
-ref =  read("documents/chunk_options_ref.md", String)
-@test result == ref
-cleanup && rm("documents/chunk_options.md")
-
-tangle("documents/chunk_options.noweb", out_path = "documents/tangle")
-result =  read("documents/tangle/chunk_options.jl", String)
-ref =  read("documents/tangle/chunk_options.jl.ref", String)
-@test ref == result
-cleanup && rm("documents/tangle/chunk_options.jl")

test/end2end.jl

@@ -16,3 +16,17 @@ rm(out)
 out = weave(joinpath(@__DIR__, "documents", "markdown_beamer.jmd"), doctype="md2tex", template=tpl)
 @test read(out, String) == read(out*".ref", String)
 rm(out)
+
+
+@testset "chunk options" begin
+
+result =  read("documents/chunk_options.md", String)
+ref =  read("documents/chunk_options_ref.md", String)
+@test result == ref
+
+tangle("documents/chunk_options.noweb", out_path = "documents/tangle")
+result =  read("documents/tangle/chunk_options.jl", String)
+ref =  read("documents/tangle/chunk_options.jl.ref", String)
+@test ref == result
+
+end

test/runtests.jl

@@ -16,10 +16,6 @@ macro jmd_str(s) mock_doc(s) end
 
 
 @testset "Weave" begin
-    @testset "Chunk options" begin
-        include("chunk_options.jl")
-    end
-
     @testset "module evaluation" begin
         include("test_module_evaluation.jl")
     end
@@ -32,6 +28,10 @@ macro jmd_str(s) mock_doc(s) end
         include("test_inline.jl")
     end
 
+    @testset "chunk options" begin
+        include("test_chunk_options.jl")
+    end
+
     @testset "error rendering" begin
         include("test_error_rendering.jl")
     end
@@ -64,14 +64,16 @@ macro jmd_str(s) mock_doc(s) end
     # end
 
     # trigger only on CI
-    get(ENV, "CI", nothing) == "true" && begin
+    if get(ENV, "CI", nothing) == "true"
         @testset "Plots" begin
             include("plotsjl_test.jl")
         end
-        
+
         @testset "Gadfly" begin
             include("gadfly_formats.jl")
         end
+    else
+        @info "skipped Plots.jl and Gadfly.jl integration test"
     end
 
     try

test/test_chunk_options.jl

@@ -0,0 +1,100 @@
+using Weave: parse_options, parse_markdown
+
+
+@static @isdefined(hasproperty) || (hasproperty(x, s::Symbol) = s in propertynames(x))
+
+@testset "`parse_options`" begin
+
+# general
+@test isempty(parse_options(""))
+@test (:opt => nothing) in parse_options("opt = nothing")
+
+# Weave style -- semicolon separated
+let opts = parse_options("opt1 = 1; opt2 = \"2\"")
+    @test (:opt1 => 1) in opts
+    @test (:opt2 => "2") in opts
+end
+# robust parsing
+@test let opts = parse_options("invalid; valid = nothing")
+    @test (:valid => nothing) in opts
+    true
+end
+
+# RMarkdown style -- comma separated
+let opts = parse_options("opt1 = 1, opt2 = \"2\"")
+    @test (:opt1 => 1) in opts
+    @test (:opt2 => "2") in opts
+end
+# robust parsing
+@test let opts = parse_options("invalid, valid = nothing")
+    @test (:valid => nothing) in opts
+    true
+end
+
+end
+
+
+@testset "`parse_markdown` Julia markdown format" begin
+
+get_options(str) =
+    (chunk = first(last(parse_markdown(str))); @test hasproperty(chunk, :options); chunk.options)
+
+# Julia markdown
+@test get_options("```julia\n```") |> length === 1
+@test get_options("```julia \n```") |> length === 1
+@test get_options("```{julia}\n```") |> length === 1
+@test get_options("```{julia }\n```") |> length === 1
+
+# Weave style -- semicolon separated
+@test get_options("```julia;\n```") |> length === 1
+@test get_options("```julia ;\n```") |> length === 1
+@test get_options("```julia; \n```") |> length === 1
+@test (:opt => nothing) in get_options("```julia; opt = nothing\n```")
+@test (:opt => nothing) in get_options("```{julia; opt = nothing}\n```")
+let opts = get_options("```julia; opt1 = 1; opt2 = \"2\"\n```")
+    @test (:opt1 => 1) in opts
+    @test (:opt2 => "2") in opts
+end
+let opts = get_options("```{julia; opt1 = 1; opt2 = \"2\"}\n```")
+    @test (:opt1 => 1) in opts
+    @test (:opt2 => "2") in opts
+end
+
+# RMarkdown style -- comma separated
+@test get_options("```julia,\n```") |> length === 1
+@test get_options("```julia ,\n```") |> length === 1
+@test get_options("```julia, \n```") |> length === 1
+@test (:opt => nothing) in get_options("```julia, opt = nothing\n```")
+@test (:opt => nothing) in get_options("```{julia, opt = nothing}\n```")
+let opts = get_options("```{julia, opt1 = 1, opt2 = \"2\"}\n```")
+    @test (:opt1 => 1) in opts
+    @test (:opt2 => "2") in opts
+end
+let opts = get_options("```julia, opt1 = 1, opt2 = \"2\"\n```")
+    @test (:opt1 => 1) in opts
+    @test (:opt2 => "2") in opts
+end
+
+end
+
+
+@testset "`parse_markdown` pandoc format" begin
+
+get_options(str) =
+    (chunk = first(last(parse_markdown(str; is_pandoc = true))); @test hasproperty(chunk, :options); chunk.options)
+
+@test get_options("<<>>=\n@") |> length === 1
+@test (:opt => nothing) in get_options("<<opt = nothing>>=\n@")
+let opts = get_options("<<opt1 = 1; opt2 = \"2\">>=\n@")
+    @test (:opt1 => 1) in opts
+    @test (:opt2 => "2") in opts
+end
+let opts = get_options("<<opt1 = 1, opt2 = \"2\">>=\n@")
+    @test (:opt1 => 1) in opts
+    @test (:opt2 => "2") in opts
+end
+
+end
+
+
+# TODO: tests for `"script"` format ?