Prettify arrays in the console output

[Totktonada]

First, two examples.

Examples

1. Space format.

   # tarantool> box.space.s:format()
   ---
   - [{'type': 'unsigned', 'name': 'id', 'version': 2}, {'type': 'string', 'name': 'd',
       'is_nullable': true}, {'type': 'map', 'name': 'x', 'is_nullable': true}]
   ...

2. Crud select metadata.

   # tarantool> crud.select('developers', nil, {first = 0})
   ---
   - metadata: [{'name': 'id', 'type': 'unsigned'}, {'name': 'bucket_id', 'type': 'unsigned'},
       {'name': 'name', 'type': 'string'}, {'name': 'surname', 'type': 'string'}, {'name': 'age',
         'type': 'number'}]
     rows: []
   - null
   ...

What is going on? Why it looks so messy?

Without array_mt

Those Lua tables are decoded from msgpack. The msgpack decoder adds a metatable to each table: {__serialize = 'seq'} for arrays and {__serialize = 'map'} for maps¹ (see msgpack.cfg({decode_save_metatables = <boolean>})).

How the examples above would look without the metatable on the array?

1. Space format.

   # tarantool> setmetatable((box.space.s:format()), nil)
   ---
   - - {'type': 'unsigned', 'name': 'id', 'version': 2}
     - {'type': 'string', 'name': 'd', 'is_nullable': true}
     - {'type': 'map', 'name': 'x', 'is_nullable': true}
   ...

2. Crud select metadata.

   # tarantool> function drop_metatable_on_select(...) setmetatable((...).metadata, nil) return ... end
   # tarantool> drop_metatable_on_select(crud.select('developers', nil, {first = 0}))
   ---
   - metadata:
     - {'name': 'id', 'type': 'unsigned'}
     - {'name': 'bucket_id', 'type': 'unsigned'}
     - {'name': 'name', 'type': 'string'}
     - {'name': 'surname', 'type': 'string'}
     - {'name': 'age', 'type': 'number'}
     rows: []
   - null
   ...

So the obvious solution would be don't add those metatables. The drawback here is that in this case we can loss information whether the Lua table should be interpreted as an array or a map. It may be crucial at encoding to msgpack back.

(The similar idea is to take knowledge how our serializers work and use it in deserializers: add the metatables only when the Lua table would be interpreted differently without it. Possibly would require more CPU cycles in deserializers. Possibly would save some CPU cycles on access to fields/elements of the resulting tables.)

However it is interesting how the examples would look without the metatables at all (both for arrays and for maps).

Without array_mt and map_mt

1. Space format.

   # tarantool> msgpack.cfg({decode_save_metatables = false})
   # tarantool> box.space.s:format()
   ---
   - - type: unsigned
       name: id
       version: 2
     - type: string
       name: d
       is_nullable: true
     - type: map
       name: x
       is_nullable: true
   ...

2. Crud select metadata.

   # tarantool> msgpack.cfg({decode_save_metatables = false}) -- does not work, hmm, why?
   # tarantool> msgpack.array_mt.__serialize = nil
   # tarantool> msgpack.map_mt.__serialize = nil
   # tarantool> crud.select('developers', nil, {first = 0})
   ---
   - metadata:
     - name: id
       type: unsigned
     - name: bucket_id
       type: unsigned
     - name: name
       type: string
     - name: surname
       type: string
     - name: age
       type: number
     rows: []
   - null
   ...

seq vs sequence

We can look at this from another angle. I mentioned two __serialize values: 'seq' and 'map'. There are four ones in fact (plus one alias).

src/lua/serializer.h#L122-L132:

/**
 * Common configuration options for Lua serializers (MsgPack, YAML, JSON)
 */
struct luaL_serializer {
	<...>
	/**
	 * Enables __serialize meta-value checking:
	 *
	 *  + 'seq', 'sequence', 'array' - table encoded as an array
	 *  + 'map', 'mappping' - table encoded as a map.
	 *    'seq' or 'map' also enable flow (compact) mode for YAML serializer
	 *    (flow="[1,2,3]" vs block=" - 1\n - 2\n - 3\n").
	 *  + function - the meta-method is called to unpack serializable
	 *    representation of table, cdata or userdata objects.
	 */
	int encode_load_metatables;
	<...>
	/** Save __serialize meta-value for decoded arrays and maps */
	int decode_save_metatables;
	<...>
};

The YAML serializer may be instructed, whether one-line or block serialization should be used. Considering the examples above we can judge that msgpack.array_mt.__serialize = 'sequence' would make things nice (the output would be the same as without array_mt).

Let's consider the example with a mapping at the outmost level instead of an array.

# tarantool> (msgpack.decode(msgpack.encode({foo = {1, 2, 3}, bar = {4, 5, 6}, baz = {7, 8, 9}})))
---
- {'bar': [4, 5, 6], 'foo': [1, 2, 3], 'baz': [7, 8, 9]}
...

Here the best result would be with msgpack.map_mt.__serialize = 'mapping'.

# tarantool> msgpack.map_mt.__serialize = 'mapping'
# tarantool> (msgpack.decode(msgpack.encode({foo = {1, 2, 3}, bar = {4, 5, 6}, baz = {7, 8, 9}})))
---
- bar: [4, 5, 6]
  foo: [1, 2, 3]
  baz: [7, 8, 9]
...

Conclusion

In my understanding, it is the deal of YAML serializer how to present data in the most readable way. I guess a nesting level plus an amount of elements would give a good heuristic. It worth to check what other formatters do.

Summary:

1. It worth to investigate heuristics in YAML encoders/formatters.
2. We should consider ignoring of the difference between seq/sequence (and its array alias) and map/mapping in the YAML serializer and do the best based on source data.
3. Just msgpack.array_mt.__serialize = 'sequence' would make things better at the first glance.

(It is not an in-depth investigation, just brief glance. I can miss details about existing YAML serializer heuristics and seq/sequence plus map/mapping interpretation. I doubt that manual setting of one-line/block formatting makes any sense, but I can miss places, where it does.)

Footnotes

1. And __newindex, which drops the metatable if a new element/field is added. The tables can be accessed as msgpack.array_mt and msgpack.map_mt². ↩

2. I doubt that it is documented API, don't lean on it in a modules/application. ↩