Community driven development

[gravemalte]

Concept for a community driven development

I had a conversation with Marlamin in Discord, we briefly written about the architecture and the position of the community in this project. He asked me to write an issue that brings the key aspect to the front.

Please notice, this is just a draft nothing is finished here. So please post anything related in here, no matter if positive or negative (just keep one thing in mind, be nice to each other).

Table of Contents

• Architecture
• Community
• Wiki

Architecture

The problem:

Form my personal perspective, there is no clear structured architecture. It exists no clear definition between the view and the logic level. Some HTML is written hard coded into the logical layer of the application. For maintenance purposes it's a bad practice.

Possible solution:

Adopt a modern MVC framework such as Laravel this will allow us to have a structured MVC architecture.

Positive aspects: The application could divided into standalone modules, the View stands alone, parts like the self written login cloud be dropped (secure authentication is provided), adoption of JWT, good documentation (see more below), ...

Negative aspects: Before restructuring the whole application, there should be a clearly vision and a plan with a detailed description what we want to achieve and how exactly (Time consuming). Most parts need to be rewritten is will results into a very time intensive task.

Community

The community plays a huge part in the project. It should be the number one goal to interact with them and discuss things which are good or not.

The contribution to the project should be easy, a good documentation, coding standards and a development setup should exists. There should also be a structured way in organization issues, tasks and maybe plans.

What about creating an organization like the wowdev group? Tools and library's could also provided there (for example DBC2CSV).

Wiki

A clear structured point with all information which are necessary to contribute back to the community.
This could be a getting started guide to setup a development environment, the coding standards or a description and the purpose of this project.

The Wiki should be the number one information source for the community.

As mention above this is only a draft so please let us discuss.
If I missed something or you want to add something please comment it, I will include it.

[Marlamin]

Currently aiming for WoW.tools to be a bit more portable first so other people can even start thinking of contributing/running local instances. Issues to solve before getting there:

Services
There's a bunch of services (not very micro ones, e.g. BuildBackup, CASCToolHost, DBCDumpHost as well as the minimap stuff) that power certain parts of the site that aren't at all portable. Each of these is required for core site functionality to work as well as requiring a large amount of data to power these which brings me to the next point...

Dataset
WoW.tools is based on data and quite a lot of it. I'll need to look at making the dataset WoW.tools uses public first. Maybe a smaller subset so people can get started (e.g. the latest build, partial DB export and such).

Other than that, there's some other worries as well. I haven't had a lot of contributions in the past, mostly due to the points you make, but also the lack of enthusiasm in contributing to existing projects that plagues the WoW community. I don't want to put too much effort into this until it is apparent that people will contribute because if in the end I'll still be flying this mostly solo I don't see the added value to putting a lot of time into moving to an MVC etc.

[justMaku]

While I do agree with the general idea of "this is hard to maintain Martin, please don't die otherwise noone will ever get this shit running again" I don't think that rewriting it is the right path for the future.

Or at least not rewriting it as a monolith PHP application using yet-another-framework like Laravel.

In my opinion the best course of action would be to start ripping out the old php implementation that does everything bit by bit and replacing it with a clean API that simply handles reading data from the database and optionally does some simple data translation and/or filtering. This is what both CASCToolHost and DBCDumpHost are currently doing and while they're far from perfect, that architecture definitely allowed us to do some really nice features recently (model viewer and new build diffing being the biggest of them).

This approach ties neatly with "THE DATASET" idea that @Marlamin has in mind and would allow us to basically have one API that both external tools and our wow.tools would benefit from.

Regarding the wowdev group: http://github.com/wowdev already exists and is already providing a big chunk of the lower-level infrastructure for wow.tools (with projects such DBCD and DBDefsLib. As far as I understand @Marlamin is not yet ready to move the full wow.tools` codebase under that umbrella org, and I fully support his decision to keep it under his name as long as he is the main contributor.

There's also always the Discord server Martin is maintaining for his community which I belive is the right place to start in case we decide to make this a bigger effort for community-driven development.

[Marlamin]

In regards to the API thing, I've started work on that but am still looking for help as well as possible usecases for it that isn't wow.tools. Throw me a line if anyone has any ideas.

[justMaku]

I think the right approach for that would be: "What APIs would I need to have if I were to implement all of wow.tools functionality only in front-end/as-a-native-app".

I know that majority of the content is already coming from such APIs but I feel like they could use some cleaning up and standardization.

I'd vote for creating a separate issue for API implementation and starting from there.

[justMaku]

After we have API secured in, we could start working on the modernization of the front-end part. Be it with PHP using server-side rendering with a framework like Laravel or go fully client-side with a library like Vue or React.

I'm purposely omitting the whole content-pipeline part for now though. I believe it's would be very hard to start refactoring this right now and while I believe there's plenty of improvements to be made, I believe most of the community requests right now are front-end focused and with an ~somehow stable~ API it would at least allow community developers to work on those without having to have the whole stack running locally.

Not to mention the benefit of being able to write native tools (think model viewers, export tools etc) that just consume the data more easily.

[justMaku]

Another thing that we should tackle in my opinion is a set of JavaScript libraries to handle World of Warcraft's file formats. Right now we have plenty of either one-time-use or even worse: inline JavaScript that handles stuff like rendering BLPs, parsing DBCs and World of Warcraft attributed string syntax that ideally should be kept in one place.

This obviously begs the question: perhaps the API should be also written in JavaScript (or TypeScript) so we could not only do that processing on the client side but also on the backend? Right now we have plenty of implementations of the same code in PHP, C# and JS that can be bothersome to maintain and debug, but I'm not sure what's the proper approach to this yet.

[justMaku]

Either way, this is a major undertaking and yes, I fully agree that it absolutely can't start without a proper plan and design documentation.

Sorry for comments span, having some sort of a brain dump moment over here.

[gravemalte]

Thank you @Marlamin and @justMaku for the detailed discussion. Sorry for responding so late, had a very stressful month at the university, thankfully the exam period is now over. Anyways, you both mention good and very important points. We are all agreeing to have a detailed plan in which all the necessary steps to happen are documented. A structured plan is the number one if a refactoring is planned. From my perspective, the hard question is where do we begin? Of course, @Marlamin is the project leader and he decides were he or maybe we could start. I know it's a big thing that isn't completed within a day, weeks or months.

It would be awesome to hear from you guys. See ya.

[Marlamin]

With Shadowlands coming up I don't think there's going to be a lot of time the next few months. Will keep this open for when things calm down a bit more.