Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Accelerated DHT Client causes OOM kill upon start of IPFS, ResourceMgr.MaxMemory ignored #9990

Open
3 tasks done
dokterbob opened this issue Jun 25, 2023 · 14 comments
Open
3 tasks done

Accelerated DHT Client causes OOM kill upon start of IPFS, ResourceMgr.MaxMemory ignored #9990

dokterbob opened this issue Jun 25, 2023 · 14 comments
Labels
kind/bug A bug in existing code (including security flaws) P1 High: Likely tackled by core team if no one steps up status/blocked Unable to be worked further until needs are met

Comments

@dokterbob
Copy link
Contributor

Checklist

Installation method

third-party binary

Version

Kubo version: 0.20.0-b8c4725
Repo version: 13
System version: amd64/linux
Golang version: go1.19.1

Config

{
  "API": {
    "HTTPHeaders": {
      "Access-Control-Allow-Methods": [
        "PUT",
        "POST"
      ],
      "Access-Control-Allow-Origin": [
        "http://<private>:10125",
        "http://localhost:3000",
        "http://127.0.0.1:5001",
        "https://webui.ipfs.io"
      ]
    }
  },
  "Addresses": {
    "API": "/ip4/0.0.0.0/tcp/5001",
    "Announce": [],
    "AppendAnnounce": [
      "/dns4/<private>/tcp/4001",
      "/dns4/<private>/udp/4001/quic",
      "/dns4/<private>/udp/4001/quic-v1",
      "/dns4/<private>/udp/4001/quic-v1/webtransport",
      "/dns4/<private>/tcp/4001",
      "/dns4/<private>/udp/4001/quic",
      "/dns4/<private>/udp/4001/quic-v1",
      "/dns4/<private>/udp/4001/quic-v1/webtransport"
    ],
    "Gateway": "/ip4/0.0.0.0/tcp/8080",
    "NoAnnounce": [
      "/ip4/10.0.0.0/ipcidr/8",
      "/ip4/100.64.0.0/ipcidr/10",
      "/ip4/169.254.0.0/ipcidr/16",
      "/ip4/172.16.0.0/ipcidr/12",
      "/ip4/192.0.0.0/ipcidr/24",
      "/ip4/192.0.2.0/ipcidr/24",
      "/ip4/192.168.0.0/ipcidr/16",
      "/ip4/198.18.0.0/ipcidr/15",
      "/ip4/198.51.100.0/ipcidr/24",
      "/ip4/203.0.113.0/ipcidr/24",
      "/ip4/240.0.0.0/ipcidr/4",
      "/ip6/100::/ipcidr/64",
      "/ip6/2001:2::/ipcidr/48",
      "/ip6/2001:db8::/ipcidr/32",
      "/ip6/fc00::/ipcidr/7",
      "/ip6/fe80::/ipcidr/10"
    ],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/quic",
      "/ip4/0.0.0.0/udp/4001/quic-v1",
      "/ip4/0.0.0.0/udp/4001/quic-v1/webtransport",
      "/ip6/::/udp/4001/quic",
      "/ip6/::/udp/4001/quic-v1",
      "/ip6/::/udp/4001/quic-v1/webtransport"
    ]
  },
  "AutoNAT": {},
  "Bootstrap": [
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ"
  ],
  "DNS": {
    "Resolvers": {}
  },
  "Datastore": {
    "BloomFilterSize": 1048576,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/3",
            "sync": false,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "100GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": false
    }
  },
  "Experimental": {
    "AcceleratedDHTClient": true,
    "FilestoreEnabled": true,
    "GraphsyncEnabled": false,
    "Libp2pStreamMounting": false,
    "OptimisticProvide": false,
    "OptimisticProvideJobsPoolSize": 0,
    "P2pHttpProxy": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "APICommands": [],
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range",
        "User-Agent"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "NoDNSLink": false,
    "NoFetch": false,
    "PathPrefixes": [],
    "PublicGateways": null,
    "RootRedirect": ""
  },
  "Identity": {
    "PeerID": "<private>",
    "PrivKey": "<definitely private>"
  },
  "Internal": {},
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {
    "DownloadSources": [],
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": null
  },
  "Pinning": {
    "RemoteServices": {}
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {},
  "Routing": {
    "Methods": null,
    "Routers": null
  },
  "Swarm": {
    "AddrFilters": [
      "/ip4/10.0.0.0/ipcidr/8",
      "/ip4/100.64.0.0/ipcidr/10",
      "/ip4/169.254.0.0/ipcidr/16",
      "/ip4/172.16.0.0/ipcidr/12",
      "/ip4/192.0.0.0/ipcidr/24",
      "/ip4/192.0.2.0/ipcidr/24",
      "/ip4/198.18.0.0/ipcidr/15",
      "/ip4/198.51.100.0/ipcidr/24",
      "/ip4/203.0.113.0/ipcidr/24",
      "/ip4/240.0.0.0/ipcidr/4",
      "/ip6/100::/ipcidr/64",
      "/ip6/2001:2::/ipcidr/48",
      "/ip6/2001:db8::/ipcidr/32",
      "/ip6/fc00::/ipcidr/7",
      "/ip6/fe80::/ipcidr/10"
    ],
    "ConnMgr": {},
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": true,
    "RelayClient": {},
    "RelayService": {},
    "ResourceMgr": {
      "Limits": {},
      "MaxMemory": "1GB"
    },
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  }
}

Description

Context

Starting kubo

  • Within a container with memory limits of 2, 3 or 4 GB.
  • With ResourceMgr.MaxMemory set to 1 or 2 GB.
  • With ports forwarded on 100/100 fibre line.
  • With about 10 TB of 1 MB blocks added through filestore (so ~10M blocks).

Consistently results in a OOM kill within minutes of startup.

Observations

When the node is started;

  1. It establishes ~2k peer connections (as, arguably, it should).
  2. 6-9 MB/s of DHT traffic is flowing in, with 200-400 KB/s going out.
  3. Within minutes (less than 15m, more than 1m) it consistently swallows all available memory and gets killed.

With AcceleratedDHTClient set to false, this problem does not occur. Hence, it seems that the Accelerated DHT client does not adhere to the resource manager's memory limit. Perhaps the defaults just need to be made accelerated DHT aware.

Might want to fix this before taking it out of the experimental phase. ;)

Possibly related/relevant
@dokterbob dokterbob added kind/bug A bug in existing code (including security flaws) need/triage Needs initial labeling and prioritization labels Jun 25, 2023
@Jorropo
Copy link
Contributor

Jorropo commented Jun 25, 2023

ResourceMgr.MaxMemory does not limit how much memory Kubo is allowed to use, it limit how much the network buffers are allowed to use.

@Jorropo
Copy link
Contributor

Jorropo commented Jun 25, 2023

Could you please post anipfs diag profile when lots of memory is used ?

@dokterbob
Copy link
Contributor Author

dokterbob commented Jun 26, 2023 •
edited
Loading

ResourceMgr.MaxMemory does not limit how much memory Kubo is allowed to use, it limit how much the network buffers are allowed to use.

Ah. That's a bit misleading from the name of the setting and AFAIK not documented anywhere.
So there is still no way to limit IPFS memory usage?

Could you please post anipfs diag profile when lots of memory is used ?

I can make another test run before it goes harakiri perhaps in about 3 days (once I'm done ingesting the 22 TB ipfs-search.com snapshot archive :p).

@Jorropo
Copy link
Contributor

Jorropo commented Jun 26, 2023 •
edited
Loading

So there is still no way to limit IPFS memory usage?

*Kubo 😉
Golang does not have the logistics for that kinda of controls, it's not worth my time implementing memory limits in a language that have implicit allocations everywhere, and also allocations randomly happen or not depending on complex heap escape rules that are not compatible with features as common as interfaces.

Need to rewrite everything into Zig before attempting that.

You can use GOMEMLIMIT (I need to get #9451 merged) but this is a weak cap, it will spin the CPU harder and harder as the limit is breached.

@dokterbob
Copy link
Contributor Author

I've turned it on again and set GOMEMLIMIT. It does seem that it's more well-behaved now.

Perhaps still notable though that without this, the accelerated DHT client is particularly memory hungry.

Config

{
  "API": {
    "HTTPHeaders": {
      "Access-Control-Allow-Methods": [
        "PUT",
        "POST"
      ],
      "Access-Control-Allow-Origin": [
        "http://<private>:10125",
        "http://localhost:3000",
        "http://127.0.0.1:5001",
        "https://webui.ipfs.io"
      ]
    }
  },
  "Addresses": {
    "API": "/ip4/0.0.0.0/tcp/5001",
    "Announce": [],
    "AppendAnnounce": [
      "/dns4/<private>/tcp/4001",
      "/dns4/<private>/udp/4001/quic",
      "/dns4/<private>/udp/4001/quic-v1",
      "/dns4/<private>/udp/4001/quic-v1/webtransport",
      "/dns4/<private>/tcp/4001",
      "/dns4/<private>/udp/4001/quic",
      "/dns4/<private>/udp/4001/quic-v1",
      "/dns4/<private>/udp/4001/quic-v1/webtransport"
    ],
    "Gateway": "/ip4/0.0.0.0/tcp/8080",
    "NoAnnounce": [
      "/ip4/10.0.0.0/ipcidr/8",
      "/ip4/100.64.0.0/ipcidr/10",
      "/ip4/169.254.0.0/ipcidr/16",
      "/ip4/172.16.0.0/ipcidr/12",
      "/ip4/192.0.0.0/ipcidr/24",
      "/ip4/192.0.2.0/ipcidr/24",
      "/ip4/192.168.0.0/ipcidr/16",
      "/ip4/198.18.0.0/ipcidr/15",
      "/ip4/198.51.100.0/ipcidr/24",
      "/ip4/203.0.113.0/ipcidr/24",
      "/ip4/240.0.0.0/ipcidr/4",
      "/ip6/100::/ipcidr/64",
      "/ip6/2001:2::/ipcidr/48",
      "/ip6/2001:db8::/ipcidr/32",
      "/ip6/fc00::/ipcidr/7",
      "/ip6/fe80::/ipcidr/10"
    ],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/quic",
      "/ip4/0.0.0.0/udp/4001/quic-v1",
      "/ip4/0.0.0.0/udp/4001/quic-v1/webtransport",
      "/ip6/::/udp/4001/quic",
      "/ip6/::/udp/4001/quic-v1",
      "/ip6/::/udp/4001/quic-v1/webtransport"
    ]
  },
  "AutoNAT": {},
  "Bootstrap": [
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ"
  ],
  "DNS": {
    "Resolvers": {}
  },
  "Datastore": {
    "BloomFilterSize": 1048576,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/3",
            "sync": false,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "100GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": false
    }
  },
  "Experimental": {
    "AcceleratedDHTClient": true,
    "FilestoreEnabled": true,
    "GraphsyncEnabled": false,
    "Libp2pStreamMounting": false,
    "OptimisticProvide": false,
    "OptimisticProvideJobsPoolSize": 0,
    "P2pHttpProxy": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "APICommands": [],
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range",
        "User-Agent"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "NoDNSLink": false,
    "NoFetch": false,
    "PathPrefixes": [],
    "PublicGateways": null,
    "RootRedirect": ""
  },
  "Identity": {
    "PeerID": "<nop>",
    "PrivKey": "<no>"
  },
  "Internal": {},
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {
    "DownloadSources": [],
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": null
  },
  "Pinning": {
    "RemoteServices": {}
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {},
  "Routing": {
    "Methods": null,
    "Routers": null
  },
  "Swarm": {
    "AddrFilters": [
      "/ip4/10.0.0.0/ipcidr/8",
      "/ip4/100.64.0.0/ipcidr/10",
      "/ip4/169.254.0.0/ipcidr/16",
      "/ip4/172.16.0.0/ipcidr/12",
      "/ip4/192.0.0.0/ipcidr/24",
      "/ip4/192.0.2.0/ipcidr/24",
      "/ip4/198.18.0.0/ipcidr/15",
      "/ip4/198.51.100.0/ipcidr/24",
      "/ip4/203.0.113.0/ipcidr/24",
      "/ip4/240.0.0.0/ipcidr/4",
      "/ip6/100::/ipcidr/64",
      "/ip6/2001:2::/ipcidr/48",
      "/ip6/2001:db8::/ipcidr/32",
      "/ip6/fc00::/ipcidr/7",
      "/ip6/fe80::/ipcidr/10"
    ],
    "ConnMgr": {},
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": true,
    "RelayClient": {},
    "RelayService": {},
    "ResourceMgr": {
      "Limits": {},
      "MaxMemory": "750MB"
    },
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  }
}

Bunch 'o profiles: https://dweb.link/ipfs/Qmays49dWQUXQzh7NgRFRDvrp5qT4nuCcSruAgGmUr539E

@Jorropo
Copy link
Contributor

Jorropo commented Jun 29, 2023 •
edited
Loading

I can't find the issues but I know about that indeed known.

The accelerated DHT client and batch provider load you 22TB of CIDs in memory (slice of ptr to cid + len) and pass it to the accelerated DHT client.
In theory you can limit it but I accidently deleted the code to do that recently (*it was never wired in Kubo anyway). We could expose this in v0.22. But the ideal fix and not workaround is to make that code streaming so you don't have to load everything.

Note: The Announcement cost per CID goes down as you have more CID since acceleratedDHTclient batch them up.

@Jorropo Jorropo mentioned this issue Jun 29, 2023
Open
@Jorropo
Copy link
Contributor

Jorropo commented Jun 29, 2023

After some back of the envelope computation it should be 1~2 GiB of memory just in CID counting pointer and GC overhead.

@dokterbob
Copy link
Contributor Author

Will this memory be freed after loading or does it always stay in memory?

@dokterbob
Copy link
Contributor Author

IPFS with GOMEMLIMIT still within 1GB by the way. Also not swapping (much).

@Jorropo
Copy link
Contributor

Jorropo commented Jun 29, 2023

Will this memory be freed after loading or does it always stay in memory?

it is freed after the providing process is finished.

@dokterbob
Copy link
Contributor Author

dokterbob commented Jul 23, 2023 •
edited
Loading

Just tried again with latest 0.21 release and accel. DHT. Even with GOMEMLIMIT (768 MiB) and ResourceMgr.MaxMemory (512 MiB), kubo gets OOM killed at 2 GiB.

My questions:

  1. Is this type of memory usage always the case? In which case I feel the min. required memory usage should most certainly be documented, e.g. in config).
  2. If not, is this sort of behaviour dependent on other factors (e.g. bandwith, amount of pinned resources ~290k in my case)? In which case it might be very valuable to investigate scaling behaviour as there might clearly be an issue for larger datasets if this is more than O(1).

@Jorropo
Copy link
Contributor

Jorropo commented Jul 24, 2023

GOMEMLIMIT isn't magic, it's just a target. Experimentally minimum is around ~3GiB, given you don't want CPU to spin constantly I think a minimum of 4GiB is required.

Golang is not a language that gives the required tools for programmers to correctly control memory usage, so trying to set a limit in the config wouldn't be very useful.
There is no simple answer to what memory usage scale with, it's both everything and nothing at the same time.

@geolffreym
Copy link

Hello guys. Any update on this issue?

@aschmahmann
Copy link
Contributor

For the issue that IIUC is being discussed here - memory consumption from loading a huge number of CIDs into memory then There are two options:

  1. Reduce (or enable configurability) of the bulk providing mechanism to tradeoff memory against performance
  2. Unified Provide Interface for Content Routers #10097

At the moment the maintainer's preference is to wait until we can do #10097 as that will end up being better for everyone and unlock more options for further reducing resource consumption.

@aschmahmann aschmahmann added status/blocked Unable to be worked further until needs are met and removed need/triage Needs initial labeling and prioritization labels Jun 18, 2024
@lidel lidel added the P1 High: Likely tackled by core team if no one steps up label Jul 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug A bug in existing code (including security flaws) P1 High: Likely tackled by core team if no one steps up status/blocked Unable to be worked further until needs are met
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@dokterbob @lidel @aschmahmann @geolffreym @Jorropo