Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create a BLOB database on top of it #9

Open
olegabr opened this issue Jul 24, 2016 · 2 comments
Open

Create a BLOB database on top of it #9

olegabr opened this issue Jul 24, 2016 · 2 comments
Milestone
wishlist

Comments

@olegabr
Copy link

olegabr commented Jul 24, 2016

If I take it correctly, these set of utilities provides a way to manipulate chunks of memory very efficiently.

I've faced a need to efficiently store a huge amount of small (about 20 kB) images and give a read access to it by the web server, nginx for example.

The problem is that having too many plain files are bad for performance, even if proper multilevel directory structure is used.
On the other hand, nginx provides a sendfile functionality that makes file access very efficient.

So, my idea is to join many small image files into one huge file with gigabyte size and provide a way to address each individual file in it. Then an appropriate nginx module can be extended to use offset in the sendfile OS call.
As a result we would have best of two worlds: sendfile performance and low file handlers count to not hurt filesystem and memory.

I'm writing it there because I think that these utilities is a good base for such a general-purpose BLOB database, and because there are people with a huge database creation background that can estimate the value of this idea.

Thank you!
@kostja
Copy link
Contributor

kostja commented Jul 24, 2016

Sure, you can do it, but why not store this stuff in Tarantool in the first place? You can use https://github.com/tarantool/nginx_upstream_module to serve them as files via nginx.

@olegabr
Copy link
Author

olegabr commented Jul 25, 2016

First, tarantool is in-memory database. I'm thinking about terabytes and petabytes of images.
Second, sendfile promises to be very fast because of kernel-level optimizations, primarily, avoid file data copying. Not sure if it can be achieved with the module you have mentioned.

@kyukhin kyukhin added this to the wishlist milestone Oct 15, 2021
nshy added a commit to nshy/small that referenced this issue Jun 16, 2023
@nshy
quota_lessor: disable memory leak assertion check in quota_lessor_des…
db129fb 
…troy

It currently breaks Tarantool debug runs. For example on running
'box/box/reconfigure.test':

    tarantool#5  0x00007fed360483d6 in __assert_fail (
        assertion=0x556b5a6c883d "lessor->leased == 0",
        file=0x556b5a6c87e8 "./src/lib/small/include/small/quota_lessor.h", line=104,
        function=0x556b5a6c99a0 <__PRETTY_FUNCTION__.15> "quota_lessor_destroy")
        at assert.c:101
    tarantool#6  0x0000556b5a1d46ca in quota_lessor_destroy (
        lessor=0x556b5a84a368 <main_cord+328>)
        at /home/shiny/dev/tarantool/src/lib/small/include/small/quota_lessor.h:104
    tarantool#7  0x0000556b5a1d4aee in slab_cache_destroy (cache=0x556b5a84a368 <main_cord+328>)
        at /home/shiny/dev/tarantool/src/lib/small/include/small/slab_cache_malloc.h:142
    tarantool#8  0x0000556b5a1db974 in cord_destroy (cord=0x556b5a84a220 <main_cord>)
        at /home/shiny/dev/tarantool/src/lib/core/fiber.c:1716
    tarantool#9  0x0000556b5a1dd793 in fiber_free ()
        at /home/shiny/dev/tarantool/src/lib/core/fiber.c:2052
    tarantool#10 0x0000556b59f3b2ac in tarantool_free ()
        at /home/shiny/dev/tarantool/src/main.cc:600
    tarantool#11 0x0000556b59f3c01f in main (argc=3, argv=0x556b5b35b0e8)
        at /home/shiny/dev/tarantool/src/main.cc:875

Looks like Tarantool has leaks of memory allocated thru mempool/small
allocators and misses destruction calls for these objects slab (it use
same quota as slab cache and cleanup quota on destruction).

Memory leak investigation is currently not easy task. We have a wide
range of suppressions that may hide positive cases like in this test.
Disabling suppresion is not possible as test-run stops to work due to
Tarantool breaks on FP leaks.

*TODO* add ticket reference.
nshy added a commit to nshy/small that referenced this issue Jun 21, 2023
@nshy
quota_lessor: disable memory leak assertion check in quota_lessor_des…
248600e 
…troy

It currently breaks Tarantool debug runs. For example on running
'box/box/reconfigure.test':

    tarantool#5  0x00007fed360483d6 in __assert_fail (
        assertion=0x556b5a6c883d "lessor->leased == 0",
        file=0x556b5a6c87e8 "./src/lib/small/include/small/quota_lessor.h", line=104,
        function=0x556b5a6c99a0 <__PRETTY_FUNCTION__.15> "quota_lessor_destroy")
        at assert.c:101
    tarantool#6  0x0000556b5a1d46ca in quota_lessor_destroy (
        lessor=0x556b5a84a368 <main_cord+328>)
        at /home/shiny/dev/tarantool/src/lib/small/include/small/quota_lessor.h:104
    tarantool#7  0x0000556b5a1d4aee in slab_cache_destroy (cache=0x556b5a84a368 <main_cord+328>)
        at /home/shiny/dev/tarantool/src/lib/small/include/small/slab_cache_malloc.h:142
    tarantool#8  0x0000556b5a1db974 in cord_destroy (cord=0x556b5a84a220 <main_cord>)
        at /home/shiny/dev/tarantool/src/lib/core/fiber.c:1716
    tarantool#9  0x0000556b5a1dd793 in fiber_free ()
        at /home/shiny/dev/tarantool/src/lib/core/fiber.c:2052
    tarantool#10 0x0000556b59f3b2ac in tarantool_free ()
        at /home/shiny/dev/tarantool/src/main.cc:600
    tarantool#11 0x0000556b59f3c01f in main (argc=3, argv=0x556b5b35b0e8)
        at /home/shiny/dev/tarantool/src/main.cc:875

Looks like Tarantool has leaks of memory allocated thru mempool/small
allocators and misses destruction calls for these objects slab (it use
same quota as slab cache and cleanup quota on destruction).

Memory leak investigation is currently not easy task. We have a wide
range of suppressions that may hide positive cases like in this test.
Disabling suppresion is not possible as test-run stops to work due to
Tarantool breaks on FP leaks.

*TODO* add ticket reference.
nshy added a commit to nshy/small that referenced this issue Jun 29, 2023
@nshy
quota_lessor: disable memory leak assertion check in quota_lessor_des…
80ac3be 
…troy

It currently breaks Tarantool debug runs. For example on running
'box/box/reconfigure.test':

    tarantool#5  0x00007fed360483d6 in __assert_fail (
        assertion=0x556b5a6c883d "lessor->leased == 0",
        file=0x556b5a6c87e8 "./src/lib/small/include/small/quota_lessor.h", line=104,
        function=0x556b5a6c99a0 <__PRETTY_FUNCTION__.15> "quota_lessor_destroy")
        at assert.c:101
    tarantool#6  0x0000556b5a1d46ca in quota_lessor_destroy (
        lessor=0x556b5a84a368 <main_cord+328>)
        at /home/shiny/dev/tarantool/src/lib/small/include/small/quota_lessor.h:104
    tarantool#7  0x0000556b5a1d4aee in slab_cache_destroy (cache=0x556b5a84a368 <main_cord+328>)
        at /home/shiny/dev/tarantool/src/lib/small/include/small/slab_cache_malloc.h:142
    tarantool#8  0x0000556b5a1db974 in cord_destroy (cord=0x556b5a84a220 <main_cord>)
        at /home/shiny/dev/tarantool/src/lib/core/fiber.c:1716
    tarantool#9  0x0000556b5a1dd793 in fiber_free ()
        at /home/shiny/dev/tarantool/src/lib/core/fiber.c:2052
    tarantool#10 0x0000556b59f3b2ac in tarantool_free ()
        at /home/shiny/dev/tarantool/src/main.cc:600
    tarantool#11 0x0000556b59f3c01f in main (argc=3, argv=0x556b5b35b0e8)
        at /home/shiny/dev/tarantool/src/main.cc:875

Looks like Tarantool has leaks of memory allocated thru mempool/small
allocators and misses destruction calls for these objects slab (it use
same quota as slab cache and cleanup quota on destruction).

Memory leak investigation is currently not easy task. We have a wide
range of suppressions that may hide positive cases like in this test.
Disabling suppresion is not possible as test-run stops to work due to
Tarantool breaks on FP leaks.

*TODO* add ticket reference.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
wishlist
Development

No branches or pull requests
3 participants
@kostja @olegabr @kyukhin