Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restructure transaction processing flow #38

Open
bramhoven opened this issue Oct 5, 2023 · 0 comments
Open

Restructure transaction processing flow #38

bramhoven opened this issue Oct 5, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@bramhoven
Copy link
Owner

The transaction processing flow should be restructured into a more flexible and less error prone.
Transactions can be retrieved from Nordigen in batches and processed parallel to that. This alone will use less memory than before.
Besides that we should have a factory which defines a flow to be able to easily turn flow features on and of and maybe make it modifiable by the user. Also we have to remove importing the requisitions from this flow. That should be seperate and happen before the transaction import. We also import the accounts and not the requisitions itself.

Asset account import flow:

  1. Retrieve requisition
  2. Retrieve accounts
  3. Compare accounts with Firefly III asset account
  4. Import the new ones
  5. Store accounts in DB
    • New accounts should be marked as new / have a recalculate balances flag

Base flow:

  1. Retrieve paginated transactions for period from Nordigen
  2. Retrieve transactions within same range of the current page from Firefly III
  3. Store a list of hashes (or normalized strings) of the Firefly III transactions for checking against
  4. Convert transaction from API model to an object we understand
  5. The next steps should be a per transaction flow:
    1. Check for existing transactions against the existing hashes list
    2. Check for duplicates in current import
      • Maintain a list of hashes (or normalized strings) that we already have imported.
      • Whenever we hit a duplicate check if we can extend the currently imported transaction (and do that)
    3. Check whether the account is one of the asset accounts
    4. Check whether it could be a transfer from an internal savings account
    5. Map it to the correct account
    6. Post it to Firefly III
  6. Trigger an transaction processed event
  7. That event should cause a calculation of balances for accounts marked as new or having a recalculate balances flag

A few requirements

  1. Code should be concise and readable (ie. a factory that defines the import flow and another that defines the transaction processing flow)
  2. Batches that have been imported should not be references any more and not be held in memory
  3. Notifications should be send whenever it is not possible to process a transaction, with clear details about the transaction
  4. Logs should be generated for all the steps. For diagnostics purposes and for showing in the import log
@bramhoven bramhoven added the enhancement New feature or request label Oct 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant