Abstract the parallelism idea into a single entity #178

spiderjako · 2024-01-07T07:45:53Z

Description

Why Task Manager

The great benefit of the TaskManager class is that it deals with concurrent worker work methods, whilst at the same time keeping a constant load on Kafka. As time goes by, a lot of the functionality would be transfered from the worker classes to the TaskManager class, because of the need for the TM to take care of the order, primary keys, etc.
I've decided to split the functionality into two versions, one working as before and one - the "new way". I'd need to think of a way to unify the workers' way of calculating intervals and returning the events in a single fashion, but that I thought would take some time. The XRP and Cardano workers in particular were a bit odd for this unification process, at least at first glance. On top of that, I think it's best if we didn't change all of the exporters' logic and we work on iterations.

That being said, I want to add tests for the separate versions, but that will happen in the next few days (w.26.02)

A lot of the stuff from my previous commits I've simplified to a great extent, no need for a buffer [] property, with that, I've made the "retrieveCompleted" function a bit simpler. A lot of functions were added for encapsulation's sake, it looks better too.
Primary Key logic was moved into TM, because of the worker not being able to keep the order, same as with lastExportedBlock. For now, I just copied the logic from the eth worker, but in the future, I'd like to somehow unify this under the #pushToBuffer function.

lib/task_manager.js

YordanPavlov · 2024-01-09T10:42:46Z

index.js

+        if (interval.success) {
+          delete interval.success;
+          this.taskManager.pushToQueue(() => this.worker.work(interval));
+          this.worker.lastExportedBlock = interval.toBlock;


Is the reason for moving this here is that the work() would execute out of order? Here it's too optimistic, the work is not done yet, just scheduled. Better leave this in the work() logic as it was. You can change the setting to:

if (result.toBlock > this.lastExportedBlock) { this.lastExportedBlock = result.toBlock; }

this would still not be exact (as past intervals may be pending) but should still be a bit more accurate than incrementing it once the interval is scheduled.

lastExportedBlock is only used for nextIntervalCalculator, meaning, it just gets its intervals from this

the work is not done yet, just scheduled

indeed, the tasks are just pushed with the calculated intervals, there's no guarantee, that the last one is finished, but lastExportedBlock doesn't overwrite any ZK state

Then I suggest rename this field. lastExportedBlock is not what it seems.

lastExportedBlock is only used for nextIntervalCalculator

It is meant to be exposed to Prometheus, see updateMetrics()

YordanPavlov · 2024-01-09T10:45:58Z

index.js

@@ -75,19 +84,27 @@ class Main {

  async workLoop() {
    while (this.shouldWork) {
+      if (this.taskManager.queue.size < constantsBase.PQUEUE_MAX_SIZE) {


Better await on the promise returned by:
queue.onSizeLessThan(limit)
instead of returning from this method and then coming back after a timeout

I think current implementation is better, the idea is to continue with kafka pushing, even if the queue is full. If I await the promise, I block the program, at least, as far as I understand
Returns a promise that settles when the queue size is less than the given limit: queue.size < limit.

Just separate the things, and do not do them in a single loop. If you go back from this loop and wait for some magic number of milliseconds you are reducing the performance of your program. Also the Kafka writing, again do not do it in this loop.

At the end of this loop there is a timeout (L 112) what should the timeout be when the above condition is false? You either set it to 0 and in effect busy loop until Pqueue resolves the promises or invent some number which would reduce your performance.

YordanPavlov · 2024-01-09T11:09:17Z

index.js

      this.worker.lastExportTime = Date.now();

      this.updateMetrics();
      this.lastProcessedPosition = this.worker.getLastProcessedPosition();

-      if (events && events.length > 0) {
-        await this.exporter.storeEvents(events);
+      if (this.taskManager.buffer && this.taskManager.buffer.length > 0) {


Currently the consumer of the buffer is part of the main loop. I think it would make sense to separate it. Make the buffer blocking (wrap it into a simple class where you can call await waitOnData()) and then start the consumer and make it block on the buffer.

WonderBeat · 2024-01-15T15:19:07Z

index.js

+  async #initTaskManager(blockNumber) {
+    this.taskManager = new TaskManager(blockNumber, constants.BLOCK_INTERVAL);
+    await this.taskManager.initPQueue(MAX_CONCURRENT_REQUESTS);
+  }

  #isWorkerSet() {


it's quite common for isSomething methods to return bool as an answer. This one returns void and confuses me a bit because it throws on error. which might be not the best strategy for the caller

WonderBeat · 2024-01-15T15:22:42Z

index.js

+        if (interval.success) {
+          delete interval.success;
+          this.taskManager.pushToQueue(() => this.worker.work(interval));
+          this.worker.lastExportedBlock = interval.toBlock;


Then I suggest rename this field. lastExportedBlock is not what it seems.

WonderBeat · 2024-01-15T15:26:01Z

index.js

+      if (this.taskManager.buffer && this.taskManager.buffer.length > 0) {
+        await this.exporter.storeEvents(this.taskManager.buffer);
+        this.lastProcessedPosition = this.taskManager.lastProcessedPosition;
+        this.taskManager.buffer = [];


I'm worried and almost sure that buffer may be modified by Task Manager between storeEvents and clean

lib/task_manager.js

WonderBeat · 2024-01-23T12:10:21Z

lib/task_manager.js

+  }
+
+  clearQueue() {
+    this.taskManager.queue.pause();


pause doesn't pause already in flight tasks. We clean queue next step which won't work if there's a long running task running

WonderBeat · 2024-01-23T12:11:40Z

lib/task_manager.js

+
+  #generateDefensiveCopy() {
+    const bufferCopy = [];
+    for (const data of this.buffer) bufferCopy.push(data);


most probably we can use JS clone which should be faster and readable

lib/task_manager.js

blockchains/eth/eth_worker.js

YordanPavlov · 2024-01-25T16:07:28Z

blockchains/eth/eth_worker.js

@@ -145,11 +146,10 @@ class ETHWorker extends BaseWorker {
    return result;
  }

-  async work() {
+  async work(key) {


You give this 'key' argument to the work() method just to have it back. It does not need to know about this key, it does nothing with it. Wrap the call in a closure if you need to preserve it.

It doesn't look really elegant atm, but I didn't think of another way to somehow set an order of the tasks beforehand

but I think I've thought of a way with queue.on('add'), which would be elegant

Wrap the call in a closure if you need to preserve it.

Wouldn't what I suggested work? I am thinking this:

const key = "some key" this.taskManager.pushToQueue(() => { const result = this.worker.work() return [key, result] } );

Hm, this is simple, yes. Thanks

YordanPavlov · 2024-01-25T16:22:56Z

blockchains/eth/lib/next_interval_calculator.js

+  return {
+    success: true,
+    fromBlock: worker.lastBufferedBlock + 1,
+    toBlock: Math.min(worker.lastBufferedBlock + worker.settings.BLOCK_INTERVAL, worker.lastConfirmedBlock)


This function depends on some side effects to work correctly. The worker must modify worker.lastBufferedBlock before the next call, this is imperative or same intervals would be created. I think the logic of the function can be improved in two ways:

Break the 'isNewBlockAvailable' logic. It has nothing to do with the interval.

Make the arguments which the function depend on more explicit. This would make it more obvious to the caller that it is up to him to provide updated 'from' value. The function needs to know two arguments (lastConfirmedBlock, toBlock).

YordanPavlov · 2024-01-25T16:36:33Z

lib/task_manager.js

+
+  #shiftOriginalBuffer(shiftsCount) {
+    while (shiftsCount >= 0) {
+      this.buffer.shift();


What's that shenanigan with shifting by one. Just erase the whole buffer by setting length to 0.

If you do that, you risk losing data.
Let's say we have a buffer of [1,2,3,4]. You copy this. In the meantime, between the copy and the buffer deletion, another element is added at the back, let's say 5. Then you'll erase the whole buffer
Though, I think this could be improved with Array.slice, now that you say it. Maybe it'd be a bit more optimal. But a shenanigan is indeed needed

In the meantime, between the copy and the buffer deletion, another element is added at the back

What do you mean, NodeJS is single threaded, this is the whole proposition. Also if it wasn't (we were dealing with some other language) this approach to access the same data structure most probably wouldn't have been enough to give us thread safety.

YordanPavlov · 2024-01-25T16:41:14Z

lib/worker_base.js

@@ -43,7 +43,7 @@ class WorkerBase {
   */
  getLastProcessedPosition() {
    return {
-      blockNumber: this.lastExportedBlock,
+      blockNumber: this.lastBufferedBlock,


That's wrong. This is the state we would preserve in ZK from which we would recover on restart. Last buffered block is too optimistic, this is not yet in Kafka.

YordanPavlov · 2024-01-25T16:44:23Z

lib/task_manager.js

+    while (this.taskData[this.lastPushedToBuffer]) {
+      for (const data of this.taskData[this.lastPushedToBuffer]) {
+        this.buffer.push(data);
+        this.lastProcessedPosition = {


This is getting messy, we have 'lastProcessed' and 'lastBuffered'. However here you are mixing them here, the fact that something is eligible for going in the buffer doesn't make it any more 'processed', it still needs to be consumed from the Kafka side and send through the producer.

You don't need this member variable. The user of this class can see what the last value is once it calls retrieveCompleted

YordanPavlov · 2024-01-25T16:57:14Z

blockchains/cardano/cardano_worker.js

@@ -170,7 +170,7 @@ class CardanoWorker extends BaseWorker {
  }

  async work() {
-    const fromBlock = this.lastExportedBlock + 1;
+    const fromBlock = this.lastBufferedBlock + 1;


Not a fan of all those renames. You resorted to renaming the 'last exported block' to 'last buffered block' to make it comply more easily with your change. But:

All those other workers don't know what 'buffered block' is

I would argue that 'last exported block' is cleaner in terms of semantics. The last block which got into Kafka. What is 'last buffered block'?

spiderjako · 2024-01-28T21:39:57Z

lib/task_manager.js

+  pushToQueue(workTask) {
+    this.queue.add(() => {
+      const result = workTask();
+      const currIndex = this.taskIndex;


This I thought would be enough, but will update this. Seems this is not enough for copying.

lib/constants.js

YordanPavlov · 2024-02-19T08:56:42Z

index.js

-      }
-      await this.exporter.savePosition(this.lastProcessedPosition);
-      logger.info(`Progressed to position ${JSON.stringify(this.lastProcessedPosition)}, last confirmed Node block: ${this.worker.lastConfirmedBlock}`);
+      if (this.taskManager.buffer.length > 0) await this.waitOnStoreEvents();


This condition on the size of a member variable of another object breaks encapsulation. Is this a performance optimization to spare us the Promise in some case? I would prefer to just call waitOnStoreEvents every time and leave this check inside. If you want the performance optimization, maybe create a method on TaskManager, called isDataAvailable

I think you can't push [] to Kafka. Initially I forgot about this check, but then got some errors.
https://github.com/santiment/san-chain-exporter/blob/master/index.js#L86
We have it in master as well.
But maybe I'll move it into the function, as you say

YordanPavlov · 2024-02-19T09:03:05Z

lib/task_manager.js

+   * @param {Array} newTransformedData 
+   */
+  handleNewData(interval, newTransformedData) {
+    this.taskData[interval.fromBlock] = { toBlock: interval.toBlock, data: newTransformedData };


I think at this point we need to have some blocking mechanism to block the producers. Currently we have no way to throttle the tasks coming from the PQueue and we would run into an OOM if the consumer of this data (the logic writing into Kafka) is slower. Right?

YordanPavlov · 2024-02-26T15:59:49Z

lib/task_manager.js

+   * pause the task generation for a bit.
+   */
+  #maybePauseQueue() {
+    if (this.consequentTaskIndex >= MAX_TASK_DATA_KEYS) {


You only increase this counter consequentTaskIndex until it exceeds MAX_TASK_DATA_KEYS, then you always pause. This behavior doesn't reflect the actual speed of the Kafka producer. What you should be doing instead is throttle based on the size of buffer.

YordanPavlov · 2024-02-26T16:03:09Z

index.js

@@ -95,6 +128,26 @@ class Main {
    }
  }

+  async workLoopV2() {


I like this idea with the gradual migration.

YordanPavlov · 2024-02-26T16:15:47Z

index.js

+  }
+
+  async waitOnStoreEvents() {
+    const buffer = this.taskManager.retrieveCompleted();


In your current implementation some logic which is not native for the TaskManager has been put inside it (last exported block, primary keys). It is better to have more separation of concern. For example setting primary keys as per the logic of each blockchain is not native to it, ideally the TaskManager shouldn't even know which blockchain it is working on. Decorating the result data is something which you can do here.

YordanPavlov · 2024-02-26T16:19:33Z

lib/task_manager.js

+      };
+      logger.info(`Initialized exporter with initial position ${JSON.stringify(lastProcessedPosition)}`);
+    }
+    this.lastExportedBlock = lastProcessedPosition.blockNumber;


No need for that logic to be encapsulated into the TaskManager

YordanPavlov · 2024-02-26T16:22:44Z

lib/task_manager.js

+   * @param {Array} buffer The array that at the end of the primary function would result in the 
+   * combined data, accordingly updated with the primary keys
+   */
+  #updatePrimaryKeysPushToBuffer(events, buffer) {


No need for this to happen here. The TaskManager can provide the data as it is and it can be decorated on the upper level.

YordanPavlov · 2024-02-28T08:22:50Z

index.js

+   * @param {Array} buffer The array of events that would be accordingly updated with the primary keys.
+   */
+  applyPrimaryKeyLogic(buffer) {
+    if (BLOCKCHAIN === 'eth') {


You can leverage polymorphism and have 'decorateWithPrimaryKeys' method in different workers, instead of moving blockchain specific logic in the index.js.

YordanPavlov · 2024-02-28T08:25:47Z

lib/task_manager.js

+    this.lastPrimaryKey += bufferLength;
+  }
+
+  isChangedLastExportedBlock(lastProcessedBlock) {


A bit of a weird method to compare two numbers. The more natural way would be to have a simple get method here and compare at the caller. Minor, your call.

YordanPavlov · 2024-02-28T08:35:06Z

blockchains/eth/lib/next_interval_calculator.js

+ * @param {BaseWorker} worker A worker instance, inherriting the BaseWorker class.
+ * @returns {object} The interval, derived from the progress of the worker
+ */
+function nextIntervalCalculatorV2(worker) {


Changing the original:

function nextIntervalCalculator(worker) { return { fromBlock: worker.lastExportedBlock + 1, toBlock: Math.min(worker.lastExportedBlock + worker.settings.BLOCK_INTERVAL, worker.lastConfirmedBlock) }; }

to

function nextIntervalCalculator(lastExported, lastConfirmed, blockInterval) { ... }

would allow you to reuse the same code. Also this is considered better design, as it more clearly denotes the intent of the function.

YordanPavlov · 2024-02-28T08:50:17Z

index.js

@@ -29,9 +32,22 @@ class Main {
      .catch((err) => { throw new Error(`${INIT_EXPORTER_ERR_MSG}${err.message}`); });
  }

+  initLastProcessedPosition(lastRecoveredPosition) {


This logic used to be part of the worker, what's the gain in moving it here? It seems like half the logic has remained there and half is here.

YordanPavlov · 2024-02-28T08:52:16Z

index.js

@@ -70,7 +95,41 @@ class Main {
    metrics.currentBlock.set(this.worker.lastConfirmedBlock);
    metrics.requestsCounter.inc(this.worker.getNewRequestsCount());
    metrics.requestsResponseTime.observe(new Date() - this.worker.lastRequestStartTime);
-    metrics.lastExportedBlock.set(this.worker.lastExportedBlock);
+    if (BLOCKCHAIN === 'eth') {


Not sure why this logic can't be unified. The 'eth' case can still have this.worker.lastExportedBlock set to the actual block that is written in Kafka.

YordanPavlov · 2024-02-28T08:53:58Z

lib/task_manager.js

+   * Used when updating the position of the exporter in ZooKeeper.
+   * @returns An object of the last exported block and last used primary key
+   */
+  getLastProcessedPosition() {


Why is this here, you don't need neither values in the TM?

YordanPavlov · 2024-02-28T08:55:37Z

lib/task_manager.js

+  /**
+   * @param {number} bufferLength The length of the buffer of updated events
+   */
+  updateLastPrimaryKey(bufferLength) {


As mentioned before TM doesn't need to know about primary keys.

YordanPavlov · 2024-03-18T08:24:37Z

index.js

@@ -73,6 +86,24 @@ class Main {
    metrics.lastExportedBlock.set(this.worker.lastExportedBlock);
  }

+  async waitOnStoreEvents(buffer) {
+    if (buffer.length > 0) {
+      this.worker.decorateWithPrimaryKeys(buffer);


The decoration can happen before calling this method.

this is a bit strange that we modify function argument. could we use return value to make this code a bit easier to read?

YordanPavlov · 2024-03-18T08:29:10Z

index.js

@@ -73,6 +86,24 @@ class Main {
    metrics.lastExportedBlock.set(this.worker.lastExportedBlock);
  }

+  async waitOnStoreEvents(buffer) {
+    if (buffer.length > 0) {
+      this.worker.decorateWithPrimaryKeys(buffer);


decorateWithPrimaryKeys should be declared in the base class and implemented for all workers.

YordanPavlov · 2024-03-18T08:41:12Z

index.js

+        constantsBase.BLOCK_INTERVAL);
+      this.worker.lastQueuedBlock = interval.toBlock;
+      intervals.push(interval);
+      if (interval.toBlock - interval.fromBlock + 1 < constantsBase.BLOCK_INTERVAL) break; //We've caught up with the head, no new intervals needed


Why the +1 ?

Seems the easier to follow logic is to break when (intrval.fromBlock - interval.toBlock) <= 0

YordanPavlov · 2024-03-18T09:05:57Z

index.js

+
+      const workerContext = await analyzeWorkerContext(this.worker);
+      setWorkerSleepTime(this.worker, workerContext);
+      if (workerContext === NO_WORK_SLEEP) return [];//TODO:


Don't forget to revisit this case, would exit atm.

YordanPavlov · 2024-03-18T09:18:04Z

index.js

+    }
+  }
+
+  async updatePosition(lastPrimaryKey, lastExportedBlock) {


It looks like you do not update the this.lastExportedBlock member variable defined in the base worker. It is used for metrics purposes.

YordanPavlov · 2024-03-18T09:48:51Z

test/eth/worker.spec.js

-    });
-
-
-    it('test primary key assignment', async function () {


Why is this removed?

It was because of my previous version, that delegated the primary key assignment elsewhere. I'll bring it back, though probably reworked, as now it's not done in the work()

YordanPavlov · 2024-03-18T09:53:54Z

test/lib/task_manager.spec.js

+
+describe('TaskManager', () => {
+  it('constructor initializes corresponding variables', () => {
+    const taskManager = new TaskManager();


Better unit test the static method, which the rest of the code is using. In other languages the constructor would have been private but I guess this is difficult to achieve in JS.

YordanPavlov · 2024-03-18T09:56:10Z

test/lib/task_manager.spec.js

+    taskManager.consequentTaskIndex = MAX_TASK_DATA_KEYS;
+    taskManager.queue.pause();
+
+    taskManager.restartQueueIfNeeded();


Where is this defined?

YordanPavlov · 2024-03-18T09:58:39Z

test/lib/task_manager.spec.js

+  it('retrieveCompleted returns only the first part of the keys data when holes are present in `taskData`', async () => {
+    const CONCURRENCY = 1;
+    const taskManager = await TaskManager.create(CONCURRENCY);
+    taskManager.initPosition({ blockNumber: 0, primaryKey: 0 });


Hm.. where is TaskManager::initPosition() defined, can't find it.

YordanPavlov · 2024-03-18T10:03:09Z

test/lib/task_manager.spec.js

+    const emmitedTaskData = [{ fromBlock: 1, toBlock: 10 }, exampleEventsData];
+    taskManager.queue.emit('completed', emmitedTaskData);
+
+    expect(taskManager.queue.isPaused).to.eq(true);


I don't think that test reflects the implementation, I think all the TaskManager tests need revisiting. In the current implementation the TaskManager is a nice standalone implementation which can be black box tested.

WonderBeat · 2024-03-20T10:49:36Z

lib/task_manager.js

@@ -0,0 +1,84 @@
+class TaskManager {
+  constructor() {


could you make it private?

To make it private, I'd need to migrate this to Typescript, but I'd do that in a following PR. JS doesn't allow private classes

WonderBeat · 2024-03-20T10:57:47Z

lib/task_manager.js

+    this.taskData[this.lastTaskIndex] = taskMetadata;
+    const taskIndex = this.lastTaskIndex;
+    this.queue.add(async () => {
+      const result = await taskMetadata.lambda(taskMetadata.interval);


can we pass taskMetadata to labda as is. I don't think this method should be aware of .interval

But you do need them. You'd need to know, which is the last handled lastExportedBlock

YordanPavlov

LGTM, let's do some integration tests before merging.

spiderjako requested review from WonderBeat and YordanPavlov January 7, 2024 07:45

YordanPavlov reviewed Jan 9, 2024

View reviewed changes

WonderBeat reviewed Jan 15, 2024

View reviewed changes

WonderBeat requested changes Jan 15, 2024

View reviewed changes

spiderjako force-pushed the add-par-bc branch 2 times, most recently from 0739740 to d9f47ea Compare January 23, 2024 11:23

WonderBeat reviewed Jan 23, 2024

View reviewed changes

lib/task_manager.js Outdated Show resolved Hide resolved

WonderBeat reviewed Jan 23, 2024

View reviewed changes

lib/task_manager.js Outdated Show resolved Hide resolved

WonderBeat reviewed Jan 23, 2024

View reviewed changes

lib/task_manager.js Outdated Show resolved Hide resolved

spiderjako mentioned this pull request Jan 25, 2024

Improving on the parallelism idea #167

Closed

spiderjako requested review from WonderBeat and YordanPavlov January 25, 2024 09:01

YordanPavlov reviewed Jan 25, 2024

View reviewed changes

spiderjako mentioned this pull request Jan 28, 2024

Updating next interval calculator file #180

Merged

spiderjako force-pushed the add-par-bc branch from 5708174 to 1980390 Compare January 28, 2024 21:37

spiderjako commented Jan 28, 2024

View reviewed changes

spiderjako force-pushed the add-par-bc branch 3 times, most recently from eab81fc to 8c5c1de Compare February 16, 2024 13:05

spiderjako requested a review from YordanPavlov February 16, 2024 13:05

spiderjako force-pushed the add-par-bc branch 2 times, most recently from 9ef36b0 to 0c72c36 Compare February 18, 2024 07:11

YordanPavlov reviewed Feb 19, 2024

View reviewed changes

spiderjako force-pushed the add-par-bc branch 2 times, most recently from 3a2c05c to 53662bf Compare February 20, 2024 07:25

spiderjako force-pushed the add-par-bc branch 2 times, most recently from 29145ae to 59ecbfb Compare February 26, 2024 15:06

spiderjako requested a review from YordanPavlov February 26, 2024 15:39

YordanPavlov reviewed Feb 26, 2024

View reviewed changes

YordanPavlov reviewed Feb 28, 2024

View reviewed changes

spiderjako force-pushed the add-par-bc branch 3 times, most recently from 3a95c28 to 09cfb87 Compare March 18, 2024 07:37

YordanPavlov reviewed Mar 18, 2024

View reviewed changes

WonderBeat reviewed Mar 20, 2024

View reviewed changes

spiderjako requested review from WonderBeat and YordanPavlov March 22, 2024 11:50

YordanPavlov approved these changes Mar 22, 2024

View reviewed changes

Lyudmil Danailov added 10 commits March 25, 2024 11:58

Updating the next interval calculator for multiple intervals

31beaab

Add task manager

4cc0dd1

Reworking task order monitoring

f11fe9f

Rehauling lastExportedBlock logic, adding versioning

4779c6c

Addressing comments

547cc03

Changing NIC functionality

f58aa11

Programming session idea

f49f016

Addressing comments

9ad1a96

Updating docs

57155ea

Updating tests

279b0f6

spiderjako force-pushed the add-par-bc branch from fa2af10 to 279b0f6 Compare March 25, 2024 10:36

Rehauling BSC integration test

c09a06a

spiderjako force-pushed the add-par-bc branch from 61c7ff8 to c09a06a Compare April 2, 2024 10:57

Abstract the parallelism idea into a single entity #178

Are you sure you want to change the base?

Abstract the parallelism idea into a single entity #178

Conversation

spiderjako commented Jan 7, 2024 • edited Loading

Description

Why Task Manager

Choose a reason for hiding this comment

spiderjako Jan 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YordanPavlov Jan 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WonderBeat Jan 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WonderBeat Jan 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spiderjako Jan 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YordanPavlov left a comment

Choose a reason for hiding this comment

spiderjako commented Jan 7, 2024 •

edited

Loading

spiderjako Jan 14, 2024 •

edited

Loading

YordanPavlov Jan 16, 2024 •

edited

Loading

WonderBeat Jan 15, 2024 •

edited

Loading

WonderBeat Jan 23, 2024 •

edited

Loading

spiderjako Jan 25, 2024 •

edited

Loading