DepViz V2

[moul]

DepViz offers a range of useful features, but its current configuration can be difficult to customize and modify. To make the most of its existing capabilities and to facilitate future development, we should strive to make it more user-friendly and open to customization.

Suggested changes:

• Split the fetcher/processor so that people can easily use DepViz to fill and maintain a database that they can access manually.
• We should consider switching to a database that is easier to use. Currently, we are using a graph database which is excellent for graph queries but is challenging to manage, limited in some ways, and has few clients. It may be overkill for our needs. We should switch to an RDBS like Sqlite by default and use recursive queries for the graph. A good caching feature can address most of the worst cases, and then we can add arbitrary limits.
  • Alternative: I suggest using SQLite and a graph database rather than switching. SQLite should be used as the primary database for fetching, filling, checking, and updating, which should be enough for many usages. Then, a dedicated script/helper should generate a graph database from the SQLite database, allowing a web interface or other applications to access the graph engine when needed.
• In general, I suggest revamping the CLI to have specific actions by default, i.e.,
  • depviz fetch DRIVER -> subcommand with all fetchers, each fetch has its options; easy to extend.
    • depviz fetch github
    • depviz fetch trello
  • depviz db SUBCOMMAND -> subcommand to manage the database
    • depviz db to-graphdb
    • depviz db query
    • depviz db shell
    • depviz db cleanup
    • depviz db sync-with-airtable
  • depviz gen SUBCOMMAND -> subcommand to use the db to generate outputs, without requiring any fetch
    • depviz gen graphviz
    • depviz gen json
    • depviz gen csv
  • later we'll try to make compositions at the top level for the most famous flows.
• Adding a download status on entities, so that, instead of having to fetch something missing, we can create an empty object with status "TO_DOWNLOAD"; and then wait for a fetcher to process it. it will solve the diamond dependency we had in a previous PR.

┆Issue is synchronized with this Trello card

[moul]

Can be tested on this new usage: gnolang/blog#5