Update Structure of Storage in the WordPress Tables

[dshanske]

This is different than #82 . The issue is about changing to custom tables. This is about redoing the usage of standard WordPress tables.

[dshanske]

To quote @jackjamieson2

Current structure for feed items

Feed items are stored as posts with a post type of yarns_microsub_post. These are hidden from the dashboard and rest of the site

These posts have custom taxonomy to store:
• Channel (yarns_microsub_post_channel): A string representing the channel into which the post was saved
• Type (yarns_microsub_post_type): A string representing post type (article, note, reply, photo, checkin, etc.)

I also registered post statuses to track the read/unread status of each feed item, yarns_unread and yarns_read

And I rely on the following properties stock to wordpress posts:
• Date published
• Date updated (I don't really do anything with this yet, but I'd like to preserve it to have the option to sort by date updated)
• Post ID - a unique identifier for each post
• Post Title: This is stored as {channel UID}|{permalink for the post} -- The rationale for this choice was that it provides a simple way to check if a feed item already exists, i.e. if a given channel already contains a feed pointing to that permalink. This should be reconsidered and a custom database can do a better job
The content of each post is saved as json in the meta field, yarns_microsub_json

^^ That's all described in class-yarns-microsub-posts.php

[dshanske]

I'm working on revising this structure, as I personally think we can do better.

1. First of all, if we are going to store the structure, it should be in the same table. By storing the JSON in the meta field, you are now engaging the wp_postmeta table. Meanwhile, the content field of the post is empty. That seems a better place to store it.
2. Post Status is being used to store read/unread status, but you are also storing it in the JSON blob.
3. The permalink for the post can be stored in the guid field. That is actually what that field is for. You can query this from the database. I've already added the SQL to do this using WP_Query. It's a minor addition that isn't in core.

Also, the latest thinking on Microsub, out of the last session, is that Posts should be attached to a Source. And Channels would be a set of sources. This creates a problem of the current structure where individual posts are attached to a channel taxonomy.

It would also address #61 and #45, which would benefit from this change in structure.

So, this is what I'm going for in a rewrite:

1. Introduce a new post type for sources. It would contain the name of the source. This would replace the use of the option table for sources.
2. The post_parent field in the individual posts would be set to the post ID of the source. That is actually a consistent use of the property and allows proper querying. Imagine using WP_Query to look for all posts of status yarns_unread, where post_parent is the source ID
3. Channel remains a taxonomy, but it's now associated with the new source post type. That means the query for posts in a channel is a query for the sources in a channel, then the posts in a source.
4. There is a problem in the scenario where you might want to move an individual post to another channel. If channels are now filled with sources, then this would be a problem. The only scenario for this I can think of is a 'Read Later' channel. I think it is a specific enough case we could address this with a special scenario.

@jackjamieson2 Does this make sense to you?

[dshanske]

Still thinking. The structure I'm proposing doesn't allow a source to be in multiple channels. The scenario being that you might want a channel for only articles, and have all your articles from all sources in there.

The question being whether we need to cover all cases.

[dshanske]

Another consideration is the idea that a source is not necessarily a traditional feed. What if I decide to teach Yarns to poll an API and output that as a source? It might not have a traditional feed URL.

[dshanske]

I also want to adjust the terminology to match. Entries and Sources, over Posts and Feeds.

[jackjamieson2]

Thanks @dshanske,

This makes sense to me. Very clever to use post_parent - great idea!

Like we discussed on slack, I think there’s value in being able to include a single source in multiple channels. E.g. I might want to file the same source in a ‘friends’ channel and a ‘photos’ channel. Actually, I don’t understand why the structure you propose here wouldn’t allow that — so maybe I’m misunderstanding something

As for polling an API instead of a traditional feed URL, I’m not sure how to best do that. This seems like it might be a special case, in which case maybe an additional field could be used to indicate the source type (defaulting to a regular feed). I don’t know what’s best here, but it seems doable

I don’t have any other comments, except this sounds really good.

[dshanske]

Re the API, that's not something I'm worried about right now. But by not tying the sources to URLs, you open the possibility in the future for different sources. You could reinterpret the new comments on your site as a source, for example.

The structure would allow it. The only annoying complication would be entries from a source moving channels, as opposed to the whole source, but I have an idea for that.

[dshanske]

As I work on this, I'm trimming out some of the pieces to optimize it. The uid for the channel can be stored in the slug field of the term for each channel. With that, the entire channel structure stored in options becomes redundant. If we need to add other fields, taxonomy terms now support their own metadata and have a table. So, to get a list of channels, you just get a list of terms in the taxonomy. To get a list of sources in the channel, you just query which posts, as source is a post type, have that term added to it....

[dshanske]

@jackjamieson2 If you want to follow...I've already made a lot of changes...but it is currently in a broken state. Committing just so you could see.. https://github.com/dshanske/yarns-microsub-server/tree/structure

[dshanske]

After Create Day, have a lot more to do, but got a lot done. Entries, Sources, and Channels are pretty much built and unit tested. Next need to rewrite polling to fit the new organization. Also, with sources being a post type, exploring adding sources using the standard Post screens, rather than custom. Have a lot of paradigms to change.