Node development mode

[jrabary]

After using Dora for a while, we found it quite challenging to code and debug a node because we always need to run the entire dataflow to check if a node works. This approach is not scalable, and it would be much more efficient if we could test nodes in isolation. Currently, our workaround is to mock the node object, allowing us to perform unit testing on individual nodes. It would be highly beneficial if Dora could natively support a feature for testing nodes in isolation.

[haixuanTao]

I understand.

Thanks for reaching out.

Indeed, development can be annoying as there's no fail fast testing of node.

I would hope that node level testing could be made in a way that is easier with mock input and output.

[heyong4725]

@jrabary You may use Dora dynamic node to debug individual node, e.g., define the node you want to debug as dynamic in your dataflow.yml file, then "dora up && dora start dataflow.yml", this will bring up all nodes defined in dataflow.yml, except dynamic node.

In separate process (or terminal), assume your node is called abc.py, you can run "python abc.py" in python interpreter mode.

This is still not ideal, but at least provide a way to test individual node against graph infra.

https://github.com/dora-rs/dora/tree/main/examples/translation has example