State of The Mini Index

[aluhrs13]

TL;DR

I've had a handful of 1:1 conversations with people or in Discord and wanted to capture current state of things and direction that I'm thinking of building towards. Feel free to leave comments or thoughts, especially if you think I might've missed something we've talked about. I'll probably be adjusting this as I start tackling things 😀.

The Good!

• Good user engagement and retention recently, both from people searching for Minis and creators adding/tagging their own Minis. Haven't been "advertising" on Facebook or Reddit at all either which is great.
• There's a steady stream of incoming Minis and older Minis with people adding tags.
• The Reddit parser I wrote kinda quickly that sits running on our server is doing a surprising amount of work, scanning Reddit posts and comments for new potential Minis.
• The site has been functioning well with very little maintenance outside of normal approving of Minis.

Problems

• Tagging consistency (and only one moderator) - The UI for tagging isn't great and some people think "Oh the more tags I add, the more people will find my Minis!" leading them to add way too many tags. On the other side, sometimes people only add one or two, and I spend time getting those to the approval level.
• The performance of the UI just isn't great for most people, images are loading quick enough, but the main pages have some frustrating workflows.
• Very Mini-centric, not a ton of terrain/scatter. I'm not quite sure how much of a problem this is, but there is quite a few people searching for terrain and not finding it.

What's Next

So what's top of mind to happen next? This is roughly prioritized, but I always reserve the right to do what sounds most interesting on a given day 😀.

Scanning Creators

Problem - Manually adding Minis kinda sucks, especially when creators upload a bunch at the end of every month.

Solution - A while back I kinda secretly wrote an Azure Function that will scan a creator's page for new items and index them. These have been working for a while, but need to be triggered manually.

Implementation - Tracking in TODO:

• Make sure Azure Functions are checked in
• Add "LastScanned" and "ScanFrequency" field to SourceSite
• Update Azure Functions to read from SQL
• Schedule the Azure Function

Autotagging

Problem - Tagging is tedious and people are bad at it.

Solution - Build out the ability to automatically tag Minis on creation, and start to use heuristics to apply tags.

Implementation - Tracking in TODO:

• Scan the name of the Mini for obvious tag matches (notably CreatureName tags) and apply them + all TagPairs
• Write something to pick N random Minis and show what would be proposed.
• Run that when adding a Mini.
• Decide if that heuristic is "good enough" or if we need more like looking at the affinity of already tagged items.

Beta UI

Problem -

1. The UI for TMI is slow and not well architected, the beta UI is intended to be a complete re-write to fix that.
2. The beta UI does solve that, but doesn't have a ton of people opting into it right now.
3. The beta UI has issues with authenticated pages that'd be pretty hard/time consuming to fix.

Solution - I'm not going to focus on getting the beta UI complete, I'm going to get feature parity for the browsing and viewing experience, while leaving the adding, tagging, and moderating experience in the "old" experience for now.

Implementation - Tracking in TODO:

• Feature - FreeOnly checkbox
• Feature - Add list of common tags
• Bug - Back button should preserve scroll location
• Bug - Get full button to work when going to external site
• Feature - Consider removing pagination from creators list
• TODO - Find lists from notes
• TODO - Minor improvements to the tagging UI

Moderator Clean-up

Problem - I'm the only moderator, partially because the moderator UI is mediocre.

Solution - Find someone else willing to moderate and fix the UI.

Implementation -

• Add "Preapproved" State #222 Add preapproved state
• Clean-up admin pages #203 (Need to read through that and probably break it out better)
• Add description to tags
• Add logging to moderator activities
• Document things like TagPairs and all the pages really.

Code

Problem - The Mini Index is written in code which leads to technical debt ;)

Solution - Endlessly have debt and fix some of it.

Implementation -

• The ADO pipeline broke a while ago and I didn't realize until I was fixing some bugs and didn't feel like dealing with it. Should probably be fixed the next time we make a change that requires EF migrations to be run.
• Add nightly test to validate that parsing from each site works properly #88 - We still don't do any nightly testing, so it's pretty shocking that a parser hasn't been broken for an extended period of time without us noticing.
• Write an Azure Functions job to find 404'd Minis #160 - I've had quite a few people report a Mini being deleted or something, so an automated scanner to run through all the Minis periodically to check for 404s would be nice sooner rather than later.
• Related Minis feature #117 - There's still a couple thousand minis that were submitted before we cached their thumbnails, and a handful of them now have broken thumbnails.
• Give clearer errors when creation of a Mini fails #191 - Mini creation failing just gives a useless error message right now.
• Clean-up GitHub branches and dead code #221 - Clean-up dead code and branches.

Deeper Thoughts

If you are/want to help out on The Mini Index, below here has some deeper thoughts into what I wrote above including links to GitHub issues to read more.

Beta UI

The beta UI is written in React and currently in a different repo (aluhrs13/tmi-react (github.com)). It's fairly barebones but functional and performant for what it does. Most of what needs to be done was captured above, once that's done, to get "full" parity in the long-run, we'll need to figure out the right way to do JWT tokens between the React front-end and ASP.NET back-end in a secure manner.

Viewing / Browsing Minis

There's nothing too crazy in this space to think about, most of the core experiences are continuing to work well, with a couple features on the backlog that are worth consdiering at some point:

• Let users make "collections" based off of tags. #103 - Allowing users to create their own collections and share with friends would be awesome, and help people lean on TMI over Thingiverse or other solutions.
• Use MiniSourceSite.Link instead of Mini.Link everywhere. #119 & Need to support duping between websites #47 - The case where a creator has uploaded the same mini to multiple sites still isn't a major problem, but it has started popping up a bit more with Prusa Printers taking some Thingiverse marketshare.

Adding Minis

Overall nothing major here, scanning of creators is by far the biggest improvement we could do and captured above.

• Master list of parsers to add #185 - Adding a few more parsers is also a good idea, Printable Scenery be on my list soon to tackle the terrain problem.

Tagging

I hit one of the top problems with tagging above, but the two remaining top of mind are also fairly interesting problems.

• Users need it to be VERY CLEAR that they have met the "bare minimum tagging" bar.
• Most TagPairs are D&D 5e focused. Someone with Pathfinder or other systems experience working through making TagPairs would be really nice.

Tag Clean-up

A chunk of moderating is cleaning up other people's "bad" tagging.

• Need a way to tell or imply to people that they should chill out when adding tags.
• Should tags require a description, that'd dissuade people from adding new one-off tags by increasing the barrier to it.

Driving Engagement and Retention

Not at all high priority

• What's new emails
• Follow creators?
• Add "New since your last visit" flag to browse page #153 - "New since your last visit" flag
• Two-Year Anniversary Milestone #219 - Tagging leaderboard

Code

These things just aren't that important.

• Gather traces and look at possible performance improvements. #195 - Performance
• Move a ton of pages to the API layer #199 - Move pages to the API layer
• Pass around a Search ID #209 - Better tracking of search success
• Merging and splitting of creators #114 - Dupe creators aren't a big problem

Other stuff

Other issues that would be nice to do eventually.

• Make a public dashboard for creators to learn about TMI users #175 - Dashboard for creators to learn about use/search patterns
• Next level tag clean-up #206 and Two-Year Anniversary Milestone #219 Next level tag clean-up
• Finish off tag pairing #207 - Finish Tag Pairing

[aluhrs13]

Functions Vs. IHostedService

@anaximander23 pointed out that using IHostedService instead of Azure Functions might be beneficial for a few reasons. The biggest one in my head is that Azure Functions would require a larger REST API surface for some of the things we want to do on a scheduled/periodic basis.

Scenarios and needs

From above, here's the rough list of scenarios/workflows and what REST APIs they'd need.

• Periodic scanning of creator's profiles
  • Get Creator information (ShouldScan and LastScanned)
  • Update Creator information (LastScanned)
  • Add a Mini
• Initial auto-tagging of Minis
  • Get suggested tags based off of given text
  • Add MiniTag
• Auto-tagging of existing Minis (and re-tagging when TagPairs are created/deleted)
  • Get list of "old" Minis and their existing tags
  • Get suggested tags based off of given text
  • Add MiniTag
• Finding 404s
  • Get list of "old" Minis
  • Change a Mini's status
• Parser testing
  • Add a Mini

APIs That Would Be Needed

Taking that list and looking at what we have already...

Already Exists

• Get Creator information - Exists, GET https://theminiindex.com/api/creators/view?id=2
• Add a Mini - Exists, POST https://theminiindex.com/api/Minis/create

Not Yet Implemented

• Get list of "old" Minis - but reasonable to add ordering to search
• Update Creator information - Doesn't exist yet, but shouldn't be hard
• Get Suggested tags - Can add to GET https://theminiindex.com/api/Minis/tagList?id=16473 or a new endpoint

Not Yet Implemented, but really should exist already (#199)

• Add a MiniTag
• Change a Mini's status