CLI Tooling: Import/export tool.

[matthewhammer]

Simple CLI tool for importing and exporting data from a Candid Spaces canister.

The Rust crate may provide almost enough.

But, we need to "re-encode" and "re-decode" the Candid data as a Candid AST, as the API accepts and emits these ASTs, as to be generic and multi-purpose.

That is to say,

• for importing into the canister with the tool (from a local machine), we want to read ordinary Candid data as human readable text or in serialized binary form, and then represent and import this data as Candid value AST, sent to the canister (see put).

• for exporting, we want to reverse the process, converting a Candid value AST into either ordinary text or serialized binary form (see getFullImage and getSubImage, which respond with the same data types accepted by put).

Perhaps this is best done as a feature added to either the Candid tools, or ic-repl, since those are the tools that already exist?

Thoughts @chenyan-dfinity?

[matthewhammer]

One may wonder:

Why does Candid Spaces convert Candid data into an AST for the put and getImage APIs?

If we want to transfer arbitrary data, we can just send blob over the wire, and let the canister decode to its proper type?

These are valid and natural questions.

We could write a Candid serialization/deserialization implementation in Motoko, that converts from Blob into Candid AST and back. This route seems even better (more useful) but also seemed like more work, in the short term, than insisting that the user do the parsing into and from an AST.

Then, I thought, perhaps the local Candid tools could do that extra step, since the canister may want to accept and store the AST directly (like in my case, here).

[chenyan-dfinity]

We could write a Candid serialization/deserialization implementation in Motoko, that converts from Blob into Candid AST and back. This route seems even better (more useful) but also seemed like more work, in the short term, than insisting that the user do the parsing into and from an AST.

This is actually the "shortcut", see https://github.com/dfinity/motoko/blob/master/test/run/idl.mo. The real solution is generics in Candid :)

[matthewhammer]

• Thanks for that link for the shortcut! Yes, using those utilities would be helpful, but they aren't the same as what a want, which is what Andreas has been calling "type dynamic" in our meetings, AFAIK. In particular, for candid spaces to accept any candid value stream but also see its structure (for queries, etc.), I want to inspect each Blob-based value without knowing its type, a priori, like in a dynamically-typed language. I think that idl primitive stuff is not exposing enough type information, and perhaps there just isn't enough in Candid, since we only include the hashes of fields.
• Generics in Candid would be helpful for some projects, but this one (candid spaces) is trying to accept any valid Candid, so having generics isn't quite as useful as it would be for a key-value store with specific key and value types.

[chenyan-dfinity]

for candid spaces to accept any candid value stream but also see its structure (for queries, etc.), I want to inspect each Blob-based value without knowing its type

I don't know if that's possible in a strongly typed language like Motoko. As you mentioned, we need the type information as an input to deserialization. The deserialization expects two types: the wire type (comes from the message) and the expected type. With only the wire type, it's like using the candid value untyped.

[chenyan-dfinity]

Actually, when you receive the message, the sender should know the candid type. You can ask the sender to get the type by calling the candid_tmp_hack. I think your use case is similar to Paul's qr scanner: https://github.com/ninegua/ic-qr-scanner

[matthewhammer]

I think for now, the easiest thing is to finish the tool that I am creating to send the AST over the wire, with full field names.

See #10 for a draft PR.

I realize that the wire format is inefficient, but that's not the main issue right now. Later, Candid Spaces can offer a more efficient put operation that accepts the (ordinary candid) Blob format whose decoding we are debating now. Somehow, we need to send the type with the Blob, or get it from somewhere. I don't think it's generally the case that every candid value has a service or a type to go along with it, but I see that in many cases, they might. In the short term, #10 plans to reuse the functionality of parsing and type-checking from the candid crate, and then add a minimal layer to send the parsed AST over the wire. Almost there now.

[matthewhammer]

I finished and merged #10

It creates a simple tool caniput using the existing candid crate. Its chief purpose is to put data into candid spaces, and provide the text/binary parsing that I don't want to implement in Motoko, at least not yet. For now, the tool permits me to reuse that functionality of the candid crate, and do it "client side" rather than "server side" (for now).

Now, with a bit more polish, caniput can transmit files and binary data too. (Not yet in #10)

[chenyan-dfinity]

Related: dfinity/candid#245

I still don't quite see the exact requirement here. Does one of the ideas listed above fits your need? Feel free to add more ideas there.