Does the number of items on the queue effect performance

[brendan-simon-indt]

If I use file based persistent queue, is performance reduced with the more items that are in the queue?

e.g. if I had large data items (e.g. 10MB), and pushed 1000 items to the queue, would performance degrade?

I imagine that adding new items isn't an issue (presuming the files are opened in "append" mode), but what about when data is read - does the whole file get written back out to disk? or is there some smart filesystem way to truncate the start of the file so that it starts at a different offset and the old data on disk is released?

I imagine database queues do not have any performance issues based on size/number of items on the queue. Correct?

I need to implement storing of lots of data files (some large, some small) on disk and be able to purge them when limits are reached. I think persist-queue can help by pushing the filenames to the queue, then pulling them off to delete the files when required.

I'm trying to determine if "file-based" or "sqlite3" based queues are the most suitable. Target is 32-bit ARM Linux system, and also want to be able to perform unit/integration test using standard desktop systems (amd64 Linux - possibly Windows too)

[peter-wangxu]

persis-queue contains a benchmark under benchmark direcotory.

you can adjust the code and benchmark in your environment.
eg.

python benchmark/run_benchmark.py 1000

[peter-wangxu]

If I use file based persistent queue, is performance reduced with the more items that are in the queue?

e.g. if I had large data items (e.g. 10MB), and pushed 1000 items to the queue, would performance degrade?

I imagine that adding new items isn't an issue (presuming the files are opened in "append" mode), but what about when data is read - does the whole file get written back out to disk? or is there some smart filesystem way to truncate the start of the file so that it starts at a different offset and the old data on disk is released?

I imagine database queues do not have any performance issues based on size/number of items on the queue. Correct?

I need to implement storing of lots of data files (some large, some small) on disk and be able to purge them when limits are reached. I think persist-queue can help by pushing the filenames to the queue, then pulling them off to delete the files when required.

I'm trying to determine if "file-based" or "sqlite3" based queues are the most suitable. Target is 32-bit ARM Linux system, and also want to be able to perform unit/integration test using standard desktop systems (amd64 Linux - possibly Windows too)

For file-based queue, the task_done performs an OS atomic replace which may introduce performance issue when the original data file is large. sql based file enables the WAL feature which can dramatically mitigate the issue under this circumstance.