Convert from Laminas\Db to Doctrine

[demiankatz]

This is work in progress on replacing VuFind's database abstraction layer.

TODO

• Handle minimum PHP version increase separately
• Handle Laminas dependency upgrades separately
• Evaluate whether we need BOTH doctrine/doctrine-orm-module and roave/psr-container-doctrine in composer.json, or if we can migrate fully from the psr-container-doctrine to the doctrine-orm-module.
• Finish testing/developing updates in commit ec5d995
• Consider handling public/index.php refactoring separately
• Fully reflect all existing configuration settings
• Finish designing entity and service interfaces for all tables
  • access_token
    • port code to use Doctrine
    • remove legacy Laminas code
  • auth_hash
    • finish Expand Auth hash entity interface #3712
    • finish Add AuthHashService; refactor code to use it. #3714
    • port code to use Doctrine
    • remove legacy Laminas code
  • change_tracker
    • finish Expand ChangeTracker entity interface; add ChangeTracker service #3646
    • finish Use ChangeTrackerService in OAI server. #3706
    • finish Add markDeleted() to ChangeTrackerService. #3776
    • review Doctrine code and make sure time zone handling is appropriate; simplify if possible
  • comments
    • finish Add/implement CommentsServiceInterface::createEntity() #3666
    • finish reconciling CommentsService with dev version (use CommentsServiceInterface to retrieve services)
  • external_session
    • finish Expand external session entity interface #3715
    • finish Introduce ExternalSessionService. #3732
    • convert expiration command to use database service
    • port code to use Doctrine
    • remove legacy Laminas code
  • feedback
    • finish Expand Feedback entity interface; add Feedback service #3650
    • finish Refactor admin FeedbackController to use FeedbackService. #3709
    • finish Update feedback admin template to use FeedbackEntityInterface. #3731
  • login_token
    • finish Establish entity interface and database service for login tokens. #3667
    • create Doctrine entity class
    • port database service methods to Doctrine; remove Laminas classes
  • oai_resumption
  • ratings
    • finish Expand Ratings Entity Interface #3739
    • finish Deprecate and replace findResource() method #3727
    • finish Add Ratings database service. #3603
  • record
    • finish Expand Record entity interface #3716
    • finish Introduce and use RecordService. #3735
  • resource
    • finish Migrate more methods to ResourceService. #3708
    • finish Deprecate and replace findResource() method #3727
    • finish Refactor Favorites search results to use database service #3783
    • finish backporting service refactoring to dev branch
    • investigate whether getOrderByClause() can/should be revised (having a static method seems suboptimal)
  • resource_tags
    • finish Resource tags entity interface #3741
    • finish Refactor user purging to new UserAccountService. #3774
    • finish developing ResourceTagsService
  • search
    • finish Expand Search entity interface #3747
    • finish Refactor more methods to the SearchService #3775
    • convert expiration command to use database service
    • set up foreign key relationship to user table in search table
    • port database service methods to Doctrine; remove Laminas classes
  • session
    • finish Expand use/scope of SessionService #3653
    • convert expiration command to use database service
    • remove legacy Laminas code
  • shortlinks
    • finish Expand Short links entity interface  #3737
    • finish Introduce and use ShortlinksService #3749
    • backport remaining logic
  • tags
    • finish Refactor tag administration to use database services #3703
    • finish Deprecate RecordDriver getTags method; improve TagService. #3771
    • finish Refactor User and UserList tag methods to FavoritesService #3772
    • finish Refactor BrowseController tag search to TagService. #3788
    • finish Eliminate tag table calls from InstallController #3789
    • should getResourceIDsForTag move into ResourceService, possibly as a protected method?
    • refactor fixDuplicateTags logic from TagService into \VuFind\Tags\TagsService (it belongs at a higher level)
  • user
    • finish Improve AuthEmailNotVerified exception. #3700
    • finish Increase use of UserEntityInterface in MyResearchController. #3702
    • finish Refactor Auth modules to use UserEntityInterface/UserServiceInterface #3625
    • finish Refactor user purging to new UserAccountService. #3774
    • finish refactoring core code to use interfaces
    • port code to use Doctrine
    • remove legacy Laminas code
  • user_card
    • finish Refactor libraryCardsEnabled method to AccountCapabilities class. #3629
    • finish Introduce UserCardService. #3679
    • backport UserCardService::getAllRowsWithUsernames method and SwitchDbHashCommand changes
  • user_list
    • finish Expand UserList entity interface #3660
    • finish Introduce UserList service #3701
    • finish Use database services in ListItems channel provider. #3782
    • finish backporting UserListService to dev
  • user_resource
    • finish Expand User resource entity interface #3743
    • finish UserResource Service Migration demiankatz/vufind#28
    • finish Refactor createOrUpdateLink() to UserResourceService. #3784
    • fix bug: when following the test procedure for record fallback loading, if you save the old and new records to different lists, one of the lists gets deleted in the process. This happens when the old resource entity is deleted, but I'm not sure why; maybe Doctrine is caching a relationship and getting overzealous in its cleanup.
    • investigate whether getOrderByClause() can/should be revised (having a static method seems suboptimal)
• Replacing low-level Laminas\Db functionality (e.g. database upgrader) with Doctrine
• Investigate creating an Entity plugin manager, and test overriding/extending entities
• Test all platforms (MySQL / PostgreSQL / etc.)
  • Be sure that code leveraging createNativeQuery works correctly across platforms (e.g. make sure tag deduplication works in PostgreSQL)
• Make sure Doctrine proxy cache is being generated in appropriate place (i.e. reflecting VUFIND_LOCAL_DIR)
• Resolve VUFIND-1480 when merging.
• Investigate supporting inheritance of annotations using kenny1911/doctrine-inherit-annotations or similar.
• Figure out how to reconcile dependency conflict between php-cs-fixer and Doctrine module (see Can't install due to conflicting requirements doctrine/DoctrineModule#800)
• Remove folder_id from search table entity after merging Remove unused folder_id field from search table #2699
• Clean up the way the case_sensitive property is set up in various service classes; this should be centralized in a cleaner way instead of by sharing the TagServiceFactory.
• Adjust getEntityClass() calls to use interfaces instead of direct class names.
• Upgrade to Doctrine 3
• If appropriate, update code to use Doctrine's UUID generator (see Get unique html element id from record view helper #2999)

[demiankatz]

I spent some more time on proof-of-concept work today. I decided to try revising some of the code in the admin module's tag controller, since that does complex queries and seemed like a potentially interesting starting point. So far, results are promising -- not only is the DQL query language more readable than the Laminas\Db object-oriented query builder, but I also managed to find and fix a bug along the way.

Some notes on today's progress:

1.) We need to have a place to put database-related logic. In the previous code, this was in the row and table classes. In the new code, I propose a set of VuFind\Db\Service classes designed to deal with various types of entities. I've started work on a TagService here, and I've set up the necessary plugin mechanism for interacting with services. This may need more granularity, but we can decide that as we gain more experience working with Doctrine.

2.) For extensibility purposes, we also need a way to abstract entity classes so that they can be overridden. I have created a VuFind\Db\Entity\PluginManager for this purpose, and I defined a marker interface for entities so that the plugin manager can validate what it is creating. Note that overriding entities appears to have one major limitation: annotations are not inherited from parent classes, so any local custom entity needs to fully duplicate its parent's properties and annotations. Perhaps there is a way around this, but given how infrequently I expect people to need to extend/override entities, it may also be a limitation that we can live with.

There is still a massive amount of work that needs to be done here, but I'm encouraged by this start. I will continue to chip away as time permits, and I welcome input or offers of collaboration. Eventually, it may be best to "divide and conquer" on all the conversion work, since there is so much of it.

[demiankatz]

Just a quick update on progress, because I believe this branch is currently in a somewhat broken state:

In order to incorporate Doctrine queries with the Laminas Paginator, I had to add the doctrine/doctrine-orm-module dependency to get an appropriate paginator adapter. In order to install this, I had to upgrade several other dependencies to newer versions, and the upgrade to the latest laminas-cache seems to be causing problems. I worked around this while developing by hacking out the cache logic that was failing, but obviously that needs a proper solution, which I will address in a separate Laminas upgrade PR.

The doctrine/doctrine-orm-module dependency also raises the question of whether we actually need roave/psr-container-doctrine, which may be less desirable if the official Doctrine piece meets our needs. I'll investigate that further when time permits.

Anyway, I've created TODO checkboxes to capture all these balls that are in the air, and I apologize for leaving this in a somewhat broken state, but I didn't want to lose my work in progress, and I'm nearly out of time for today. I'll pick this up again as soon as practical.

[aleksip]

Hi @demiankatz, this change is very exciting because our Admin Interface has been using doctrine/doctrine-orm-module for a while and it also reads the VuFind database.

I wonder whether it would make sense to develop the new Doctrine based VuFind database layer as a separate module/package, in a similar way that has been discussed in #2272 and #2283 (which I hope to continue working on as soon as time allows)?

In any case it would be great to have this developed in a way that would allow for code reuse, even if only by copying the classes over!

[demiankatz]

Thanks, @aleksip, I'm glad that you're happy with this approach. I'm open to organizing the code any way that is helpful; I think it's just a question of what works best. Perhaps one approach could be to create a VuFindDb module to contain the Entity classes, ConnectionFactory, etc. I'm not sure if the actual Service classes would fit there as well, or if they'd be better off in the main VuFind module due to their potential entanglement with other dependencies. I think I'd have to get deeper into the project to have a stronger opinion about that, but I'm open to input!

In any case, right now, my priority is finishing up all the outstanding work on VuFind 9, so this is more or less on hold, but I'm hoping this will be one of the priority changes for VuFind 10, and I expect I'll be spending a lot of time on it in 2023. I'm definitely happy to discuss/plan in the meantime so we have a clear path forward when it comes time to execute the work. I'm also very open to brainstorming strategies for collaboration if you or anyone on your team might be able to help with the updates, as this is a very big project. I'm hoping that, in addition to updating everything, we can make strides toward introducing better test coverage for core code, since if we do this right, it should make things easier to test than the old database library did.

[aleksip]

@demiankatz Sounds good! I am currently experimenting with potential vufind-db, vufind-view and vufind-config packages in our Admin Interface, will report on how it goes. 🙂

[demiankatz]

Thanks, @aleksip, that's great! Certainly if you have classes from your admin code that it would make sense to move over here for use in the core, that might save us some time and trouble. I'm not sure if the existing auto-generated Entity classes are actually the best they could be, as most of them have not yet been tested.

[demiankatz]

@aleksip, thanks for the progress on configuration here -- just leaving a couple of comments/questions related to the latest changes.

[demiankatz]

@aleksip, I see that you deleted this factory, but the configuration is still here. We should clean something up.

[aleksip]

Yes, forgot to remove that line. Should I push the change to your branch or what would be the best way to work on this together?

[demiankatz]

Please feel free to push directly here, as long as the code still works. I'm not going to be working on this much until next year, so I welcome any progress you can make in the meantime, and I don't anticipate that we'll "step on each other," so to speak.

[demiankatz]

Also, if at any point it would be helpful to get a second opinion or anything, or if you would like me to run any tests, just let me know and I'll be happy to help!

[aleksip]

@demiankatz I meant to push the changes to my repository and then point you there. Not sure how multi-user PRs should work... Should I push future suggested changes to my clone so you can then decide whether to merge to your branch from there?

[demiankatz]

@aleksip, it is possible to make a pull request against a pull request -- you can just target a PR against this branch. That might be the easiest workflow going forward, especially when multi-user collaboration picks up... but since things are quiet right now, feel free to work directly in this branch, at least until you get things into a correct/stable state again. (I think it's easier to just move forward and fix things than to roll back changes and make a separate PR, at least under the present circumstances).

[aleksip]

@demiankatz Seems like I can create a PR for the branch in your repository! Should I do that? Edit: missed your above reply.

[demiankatz]

If you want to try it as a proof-of-concept, feel free... but otherwise, as I said, you can just go ahead and push. :-)

[aleksip]

@demiankatz I have accidentally pushed my changes to your branch, sorry about that! I have been refactoring our Admin Interface database implementation to be based on this PR, and those were the changes I needed to make. They were needed to support two entity managers with connections to different databases.

I don't claim to fully understand how Doctrine's configuration works, but the orm_default naming convention seems a bit problematic in integrations. It would seem better to define an application specific name, and if necessary create an orm_default alias for that.

I also wondered about the need for AnnotationDriverFactory, as it seems that the same result can be achieved with just configuration.

[demiankatz]

@aleksip, I am certainly not an expert on Doctrine's configuration either, and I've been learning as I go... so if the annotation driver can be replaced by pure configuration, I'm entirely in favor of that, and if a different naming convention would be more robust, I also have no objection to changes.

[demiankatz]

I spent some time today working on PostgreSQL compatibility. The test suite is now in parity with the dev branch when running against PostgreSQL (there is one known bug causing a single test to fail in both places).