Skip to content

Commit

Permalink
Streamline adapter API; deal with vectors of substrings; improve desc…
Browse files Browse the repository at this point in the history 
…ription format.
  • Loading branch information
@orenbenkiki
orenbenkiki committed Jun 11, 2024
1 parent ebc52d6 commit 9feb3f9
Show file tree
Hide file tree
Showing 23 changed files with 330 additions and 293 deletions.
2 changes: 1 addition & 1 deletion docs/v0.1.0/.documenter-siteinfo.json
View file
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"documenter":{"julia_version":"1.10.0","generation_timestamp":"2024-06-08T17:04:59","documenter_version":"1.4.1"}}
{"documenter":{"julia_version":"1.10.0","generation_timestamp":"2024-06-11T13:43:45","documenter_version":"1.4.1"}}
123 changes: 34 additions & 89 deletions docs/v0.1.0/adapters.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -273,32 +273,25 @@ <h1 id="Adapters">
<pre>
<code class="language-julia hljs">adapter(
computation::Function,
view::Union{DafWriter, DafReadOnly},
[name::Maybe{AbstractString} = nothing,
capture=MemoryDaf,
axes::Maybe{ViewAxes} = nothing,
data::Maybe{ViewData} = nothing,
daf::DafWriter;
input_axes::Maybe{ViewAxes} = nothing,
input_data::Maybe{ViewData} = nothing,
capture = MemoryDaf,
output_axes::Maybe{ViewAxes} = nothing,
output_data::Maybe{ViewData} = nothing,
empty::Maybe{EmptyData} = nothing,
relayout::Bool = true,
overwrite::Bool = false]
overwrite::Bool = false,
)::Any
</code>
</pre>
<p>Invoke a
<code>computation
</code> on a
<code>view
</code> data set and return the result; copy a
<a href="views.html#Daf.Views.viewer">
<code>viewer
</code>
</a> of the updated data set into the base
<code>Daf
</code> data of the view. If specified, the
<code>name
</code> is used as a prefix for all the names; otherwise, the
<code>view
</code> name is used as the prefix.
</code> on a view of some
<code>daf
</code> data and return the result; copy a view of the results into the base
<code>daf
</code> data.
</p>
<p>If you have some
<code>Daf
Expand All @@ -315,15 +308,13 @@ <h1 id="Adapters">
</code> as follows:
</p>
<ul>
<li>Create a (read-only)
<code>view
</code> of your data which presents the data properties under the names expected by the
<li>Create a (read-only) view of your data which presents the data properties under the names expected by the
<code>computation
</code>, using
<a href="views.html#Daf.Views.viewer">
<code>viewer
</code>
</a>. If the
<code>input_axes
</code> and
<code>input_data
</code>. If the
<code>computation
</code> was annotated by
<a href="computations.html#Daf.Computations.@computation">
Expand All @@ -335,51 +326,25 @@ <h1 id="Adapters">
</code>
</a> will be explicitly documented so you will know exactly what to provide.
</li>
<li>Pass this
<code>view
</code> to
<code>adapter
</code>, which will invoke the
<code>computation
</code> with a (writable)
<code>adapted
</code> version of the data (created using
<a href="chains.html#Daf.Chains.chain_writer">
<code>chain_writer
</code>
</a> and a new
<a href="formats.html#Daf.Formats.DafWriter">
<code>DafWriter
</code>
</a> to
<li>Chain this read-only view with an empty
<code>capture
</code> the output; by default, this will be a
</code> writable data set (by default,
<a href="memory_format.html#Daf.MemoryFormat.MemoryDaf">
<code>MemoryDaf
</code>
</a>), but it can be any function that takes a
<code>name
</code> (named) parameter and returns a
<code>DafWriter
</code>.
</a>) and pass the result to the
<code>computation
</code> as the &quot;adapted&quot; data set.
</li>
<li>Once the
<code>computation
</code> is done, create a new view of the output, which presents the subset of the output data properties you are interested in, with the names you would like to store them as. Again, if the computation was annotated by
<a href="computations.html#Daf.Computations.@computation">
<code>@computation
</code>
</a>, then its
<a href="contracts.html#Daf.Contracts.Contract">
<code>Contract
</code>
</a> will be explicitly documented so you will know exactly what to expect.
</li>
<li>Copy this output view data into the base
<code>Daf
</code> data of the
<code>view
</code> (using
</code> is done, use the
<code>output_axes
</code> and
<code>output_data
</code> to create a view of the output, and copy this subset to the original
<code>daf
</code> data set, using (using
<a href="copies.html#Daf.Copies.copy_all!">
<code>copy_all!
</code>
Expand All @@ -396,26 +361,6 @@ <h1 id="Adapters">
</code>).
</li>
</ul>
<div class="admonition is-info">
<header class="admonition-header">Note
</header>
<div class="admonition-body">
<p>If the names of the properties in the input already match the contract of the
<code>computation
</code>, you can pass the data set directly as the
<code>view
</code>. The call to
<code>adapter
</code> may still be needed to filter or rename the
<code>computation
</code>&#39;s output. If the outputs can also be used as-is, then there&#39;s no need to invoke
<code>adapter
</code>; directly apply the
<code>computation
</code> to the data and be done.
</p>
</div>
</div>
<p>Typically the code would look something like this:
</p>
<pre>
Expand All @@ -425,10 +370,10 @@ <h1 id="Adapters">
# under a different name.

result = adapter(
viewer(daf; ...), # How to view the input in the way expected by the computation.
name = &quot;example&quot;, # A name to use to generate the temporary `Daf` data names.
axes = ..., data = ..., # How and what to view from the output for copying back into `daf`.
empty = ..., # If the input view specifies a subset of some axes.
daf; # The Daf data set we want to apply the computation to.
input_axes = ..., input_data = ..., # How and what to provide as input to the computation.
output_axes = ..., output_data = ..., # How and what to copy back as output of the computation.
empty = ..., # If the input view specifies a subset of some axes.
) do adapted # The writable adapted data we can pass to the computation.
computation(adapted, ...) # Actually do the computation.
return ... # An additional result outside `daf`.
Expand All @@ -442,7 +387,7 @@ <h1 id="Adapters">
<a href="computations.html#Daf.Computations.@computation">
<code>@computation
</code>
</a> functions to use clear generic names for their inputs and outputs, and still apply them to arbitrary data sets using more specific names. One can even invoke the same computation with different parameter values, and store the different results in the same data set under different names.
</a> functions to use clear generic names for their inputs and outputs, and still apply them to arbitrary data sets that use more specific names. One can even invoke the same computation with different parameter values, and store the different results in the same data set under different names.
</p>
</div>
</section>
Expand Down
4 changes: 2 additions & 2 deletions docs/v0.1.0/formats.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -621,7 +621,7 @@ <h3 id="Description">
<section>
<div>
<pre>
<code class="language-julia hljs">format_description_header(format::FormatReader, lines::Vector{String})::Nothing
<code class="language-julia hljs">format_description_header(format::FormatReader, lines::Vector{String}, deep::Bool)::Nothing
</code>
</pre>
<p>Allow a
Expand All @@ -647,7 +647,7 @@ <h3 id="Description">
<section>
<div>
<pre>
<code class="language-julia hljs">format_description_footer(format::FormatReader, lines::Vector{String})::Nothing
<code class="language-julia hljs">format_description_footer(format::FormatReader, lines::Vector{String}, cache::Bool, deep::Bool)::Nothing
</code>
</pre>
<p>Allow a
Expand Down
6 changes: 6 additions & 0 deletions docs/v0.1.0/matrix_layouts.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -733,6 +733,12 @@ <h2 id="Changing-layout">
<code>PyArray
</code> will still share the underlying buffer. Sigh.
</li>
<li>Copying a vector of anything derived from
<code>AbstractString
</code> returns a vector of
<code>AbstractString
</code>.
</li>
</ul>
</div>
</section>
Expand Down
2 changes: 1 addition & 1 deletion docs/v0.1.0/search_index.js
View file

Large diffs are not rendered by default.

128 changes: 33 additions & 95 deletions src/adapters.jl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,19 +22,19 @@ using ..Views
"""
adapter(
computation::Function,
view::Union{DafWriter, DafReadOnly},
[name::Maybe{AbstractString} = nothing,
capture=MemoryDaf,
axes::Maybe{ViewAxes} = nothing,
data::Maybe{ViewData} = nothing,
daf::DafWriter;
input_axes::Maybe{ViewAxes} = nothing,
input_data::Maybe{ViewData} = nothing,
capture = MemoryDaf,
output_axes::Maybe{ViewAxes} = nothing,
output_data::Maybe{ViewData} = nothing,
empty::Maybe{EmptyData} = nothing,
relayout::Bool = true,
overwrite::Bool = false]
overwrite::Bool = false,
)::Any
Invoke a `computation` on a `view` data set and return the result; copy a [`viewer`](@ref) of the updated data set into
the base `Daf` data of the view. If specified, the `name` is used as a prefix for all the names; otherwise, the `view`
name is used as the prefix.
Invoke a `computation` on a view of some `daf` data and return the result; copy a view of the results into the base
`daf` data.
If you have some `Daf` data you wish to run a `computation` on, you need to deal with name mismatches. That is, the names
of the input and output data properties of the `computation` may be different from these used in your data. In addition,
Expand All @@ -44,27 +44,15 @@ irrelevant properties.
To address these issues, the common idiom for applying computations to `Daf` data is to use the `adapter` as
follows:
- Create a (read-only) `view` of your data which presents the data properties under the names expected by the
`computation`, using [`viewer`](@ref). If the `computation` was annotated by [`@computation`](@ref), then its
[`Contract`](@ref) will be explicitly documented so you will know exactly what to provide.
- Pass this `view` to `adapter`, which will invoke the `computation` with a (writable) `adapted` version of the
data (created using [`chain_writer`](@ref) and a new [`DafWriter`](@ref) to `capture` the output; by default, this
will be a [`MemoryDaf`](@ref)), but it can be any function that takes a `name` (named) parameter and returns a
`DafWriter`.
- Once the `computation` is done, create a new view of the output, which presents the subset of the output data
properties you are interested in, with the names you would like to store them as. Again, if the computation was
annotated by [`@computation`](@ref), then its [`Contract`](@ref) will be explicitly documented so you will know
exactly what to expect.
- Copy this output view data into the base `Daf` data of the `view` (using [`copy_all!`](@ref), `empty`, `relayout`
- Create a (read-only) view of your data which presents the data properties under the names expected by the
`computation`, using `input_axes` and `input_data`. If the `computation` was annotated by [`@computation`](@ref),
then its [`Contract`](@ref) will be explicitly documented so you will know exactly what to provide.
- Chain this read-only view with an empty `capture` writable data set (by default, [`MemoryDaf`](@ref)) and pass the
result to the `computation` as the "adapted" data set.
- Once the `computation` is done, use the `output_axes` and `output_data` to create a view of the
output, and copy this subset to the original `daf` data set, using (using [`copy_all!`](@ref), `empty`, `relayout`
(default: `true`) and `overwrite` (default: `false`).
!!! note
If the names of the properties in the input already match the contract of the `computation`, you can pass the data
set directly as the `view`. The call to `adapter` may still be needed to filter or rename the `computation`'s
output. If the outputs can also be used as-is, then there's no need to invoke `adapter`; directly apply the
`computation` to the data and be done.
Typically the code would look something like this:
```
Expand All @@ -74,10 +62,10 @@ daf = ... # Some input `Daf` data we wish to compute on.
# under a different name.
result = adapter(
viewer(daf; ...), # How to view the input in the way expected by the computation.
name = "example", # A name to use to generate the temporary `Daf` data names.
axes = ..., data = ..., # How and what to view from the output for copying back into `daf`.
empty = ..., # If the input view specifies a subset of some axes.
daf; # The Daf data set we want to apply the computation to.
input_axes = ..., input_data = ..., # How and what to provide as input to the computation.
output_axes = ..., output_data = ..., # How and what to copy back as output of the computation.
empty = ..., # If the input view specifies a subset of some axes.
) do adapted # The writable adapted data we can pass to the computation.
computation(adapted, ...) # Actually do the computation.
return ... # An additional result outside `daf`.
Expand All @@ -88,80 +76,30 @@ end
```
This idiom allows [`@computation`](@ref) functions to use clear generic names for their inputs and outputs, and still
apply them to arbitrary data sets using more specific names. One can even invoke the same computation with different
apply them to arbitrary data sets that use more specific names. One can even invoke the same computation with different
parameter values, and store the different results in the same data set under different names.
"""
@logged function adapter(
computation::Function,
view::Union{DafWriter, DafReadOnly};
name::Maybe{AbstractString} = nothing,
daf::DafWriter;
input_axes::Maybe{ViewAxes} = nothing,
input_data::Maybe{ViewData} = nothing,
capture = MemoryDaf,
axes::Maybe{ViewAxes} = nothing,
data::Maybe{ViewData} = nothing,
output_axes::Maybe{ViewAxes} = nothing,
output_data::Maybe{ViewData} = nothing,
empty::Maybe{EmptyData} = nothing,
relayout::Bool = true,
overwrite::Bool = false,
)::Any
writer = capture(; name = get_adapter_capture_name(view; name = name))
adapted = get_adapter_input(view; name = name, writer = writer)
base_name = daf.name
@assert input_axes !== nothing || input_data !== nothing || output_axes !== nothing || output_data !== nothing "no-op adapter"
input = viewer(daf; axes = input_axes, data = input_data, name = base_name * ".input")
captured = capture(; name = base_name * ".capture")
adapted = chain_writer([input, captured]; name = base_name * ".adapted")
result = computation(adapted)
copy_adapter_output(
view,
adapted;
name = name,
axes = axes,
data = data,
empty = empty,
relayout = relayout,
overwrite = overwrite,
)
output = viewer(adapted; axes = output_axes, data = output_data, name = base_name * ".output")
copy_all!(; source = output, destination = daf, empty = empty, relayout = relayout, overwrite = overwrite)
return result
end

function get_adapter_capture_name(view::Union{DafWriter, DafReadOnly}; name::Maybe{AbstractString})::AbstractString
_, prefix = get_base(view, name)
return prefix * ".capture"
end

@logged function get_adapter_input(
view::Union{DafWriter, DafReadOnly};
name::Maybe{AbstractString},
writer::DafWriter,
)::DafWriter
_, prefix = get_base(view, name)
return chain_writer([view, writer]; name = "$(prefix).adapted")
end

@logged function copy_adapter_output(
view::Union{DafWriter, DafReadOnly},
adapted::DafWriter;
name::Maybe{AbstractString},
axes::Maybe{ViewAxes},
data::Maybe{ViewData},
empty::Maybe{EmptyData},
relayout::Bool,
overwrite::Bool,
)::Nothing
destination, prefix = get_base(view, name)
output = viewer(adapted; name = "$(prefix).output", axes = axes, data = data)
copy_all!(; source = output, destination = destination, empty = empty, relayout = relayout, overwrite = overwrite)
return nothing
end

function get_base(view::Union{DafWriter, DafReadOnly}, name::Maybe{AbstractString})::Tuple{DafWriter, AbstractString}
base = view
if base isa DafReadOnly
base = view.daf
@assert base isa DafWriter && !base.internal.is_frozen
end

if name === nothing
prefix = base.name
else
prefix = name # untested
end

return (base, prefix)
end

end # module
Loading

0 comments on commit 9feb3f9

Please sign in to comment.