Ubuntu 22 crashes #561
Comments
|
after new pull and totally clean compile and install it FAILED again. |
|
|
running as root same thing. |
|
It's expected. There is no Etc. Each of these chains will be your personal testnet. |
|
About build error: Can't reproduce on fresh installed About crash error after launch (how you get it launched if you had build error?), try the following:
|
|
Regarding the crash of the pirate.service it looks like a core file was generated during the crash.
|
Where do the core files end up when launching from systemd? I've looked everywhere and cant seem to find it. I think it was a corrupt komodostate or komodostate.ind. it happened again after reboot this morning. it was running fine for last 6 days through reboots. I've deleted above files and it seems to be running syncing something. |
|
looks like komostate files were the problem failed startup with core dump good restart, cant find any reference to komodostate.ind. Is that created on shutdown. I use komodo-cli stop in my service file. Anything I should confirm in the file that could casue the corruption. |
I think i accidently ran make install in source dir. its doesn't seem to do it if i use build.sh, so my fault, i probably. the problems however start about 1 month ago when i did a few minor upgrades to ubuntu 18 lts, then got severely worse when i upgraded first machine to Ubuntu 22 latest. |
core file may be not be created if your OS env disables it (by default) |
komodostate.ind should be created on startup if it did not exist. Could it be a file/dir access rights? |
ubuntu 22 seems to do different coredumps, see if this helps and i'll check how to do old stype for gdb. gdb doesnt recongize this type. |
You could run p.s. As I didn't see your |
|
here is a hint: |
That's why I asked about set 777 permissions on the data folder, bcz it's very similar to this - zcash#5062 . |
ok good to know. differences i see yesterday these params were bigger than today after fixing. yesterday today after deleting and good restart. Good startup yesterday: I need to learn this if you can point me to the source code to review id appreciate it. I'm review this now #548 this has been an ongoing problem for awhile. So, it must be a shutdown problem somehow or possibly an upgraded lib used to compile. Machines are rebooted once per day. I run 5 pool servers, all with local (127.0.0.1) node wallets, some with multiple PIRATE daemons for backups. most times it'll be just 1 Pirate daemon with the problem. I would run dual daemons on all machines but its a pig at startup. is the current "contrib/komodod.service" file good to use exactly as is? except for Pirate This machine should be factory fresh settings from Ubuntu 22 lts P.S. i am running ZFS on this one. default install now i think, but previous was 18 lts, then 20 lts, with the problem for sure started on 20 lts... that was reason for upgrade to 22 lts. Any tweaks to komodod server startup variables that will help speed things up? Too TCP connections, etc? cache sizes? sysctl parameters best suited for node daemons and stratum servers? Any suggestions? 1 dedicated machine for node wallets? or 2 machines dedicated? How do the big pools do it? I'm also having a pretty long delay on local machines 127.0.0.1:7778(my rpc port) when testing komodo-cli getinfo. I'm woried that the node-stratum is seeing same things. Sometimes response is instant and others (often) 5-12 seconds. which is way too long, few times it never responded after few minutes i killed the process. higher delay here is minor delay: fast response: All machines are at lease 4 threads 3.6Ghz, some 8 threads 3.9Ghz, 16gb ram,all SSD drives, 1gb ethernet on single cisco 3750 & MikroTik 10gb switches, some servers on 10gb fiber (not that should make any difference on simple http requests to daemon. Main internet is Google 2.5GB connection. All in All it should be a good setup but maybe high on rejected/orphaned blocks successful restart after deleting both komodostate file. no komodostate.ind file yet after 1 hour and synced I think this is what you were asking for with "bt" |
|
i also did the chmod. ill be rebooting after while and let you know. let me know if this points to specific problem like shutdown. Ill have to do something to ensure all komodod daemons are shutting down clean in the service script. |
|
|
please also consider another rpc instead of getinfo (not using the wallet), f.e. getblockchaininfo (not sure what data you would like to obtain though) |
If you periodically shutdown / reboot the machines, you'll definitely need to add check if daemon stopped, before shutdown / reboot, to avoid blockchain corruption. If you have crash on |
I think that has been happening. I use Is there a full compile log somewhere that would help understand the error. I cant seem to connect to the P2P no matter what i do. disabled firewall - didn't help, i can connect to other machines on local network that are setup for remote RPC with this machine and NOMP pool server. I've been building and running these for Pirate Chain for 2 -3 years now, so I'm not a total noob. problem is the PIC & PIE compiling at very end. It creates a semi working Komodod that seems to run ok. today after shutting it down manually and waiting for full shutdown, I ran it on console foreground and I'm now getting a another error I've never seen: can komodod be compiled with "-no-pie" if so what would command line be to build. I've tried about 5 different ways and it wouldn't start compiling. Makefile looks like it should take compiler commands at end of the build.sh command line. Any help would be nice. I'll keep working on this till i get it fixed. on Ubuntu 22 both machines seem have more problems than ever rebooting with the same exact setup I've always used. and it very random problems or full crashes. Ill get you the core debug as soon as i t happens again. OS cleared all the old ones out.. Here is the config file for PIRATE.conf: and why wont komodod report the proper PORTS at startup? that is not PIRATE peermagic either lol--- just saw it peermagic is hex reversed. -- so forget that problem, ports should be read from config file right? total of 4-5 servers run exactly the same config files. if I run two daemons I use rpcport=7778, and for P2P - port=45453. What about the bind variable would that help with poolservers not connecting to P2P port if you leave ports out of config komodod assigns RPC: 7771 and P2P: 45452.... sorry for the long post but I was trying to cover a few of your guys questions to get up to date on the status of this. Looks like this time it rescanning the whole blockchain again, from the wallet startup error above. -- soo slow. Ill just clear it out and use a backup wallet.dat and bootstrap. |
Directroy perms... Proper shutdown as it should be on 1 of two daemons on this machine. |
Wow, you're right. I use getinfo to get block info and if synced for the pool server, ill look at changing some of the commands if possible. Does getblockchaininfo respond before it synced so i can monitor reboot right now I hold pm2 off until I get a synced: true from getinfo. pm2.service line: i have problems with pool server starting up before it synced. sometimes it come right up other times it just stuck in a weird loop and doesn't recognize blockchain is now synced. the above doesn't start pool server until its synced and comes right up every time. this is pool server and pirate has been synced for about 3 minutes and it stuck. I restart the server and boom everthing is good. if you have ideas i'd like to hear anything. getblockchaininfo seems have most of what need . |
|
it looks like most of my problems are from improper shut down. Its possible the compiling error is contributing to the problems but I haven't looked at that part of the source code. I'll look it over and see if it is any part of the Ubuntu 22 problems. I have recompiled(for no reason) reinstalled, wiped pirate date directories, and copied a known good copy of the blockchain. Both daemons are synced and running fine for now. After a few reboots i'll know for sure. To hopefully make sure the komodod daemon is fully shutdown i added a ExecPost in the systemd service file for both daemons... also added it to my other machines just to confirm it works. Maybe it will help others. I've also included latest dump before i wiped everything. |
|
Maybe a silly question, but have you tried just running the daemon as a daemon (not as a service) without a custom data dir and without manually bootstrapping or manually customizing the conf file or preexisting wallet.dat? |
I havent, but will try that now. I just shut down one deamon and made sure debug file showed complete proper shutdown. and i get a coredump upon restarting. And even more bazzar sometime if i try to restart again it starts up fine and continues to run. here is the coredump from a perfect shutdown, then restart on command line with debug=0 to see what happens. |
|
everytime it references a shared library and that was one problem compiling on Ubuntu 22. after the last up it compiled ok with no errors. I'll confirm if i installed that version. it had to do with referencing a fPIC in shared library that couldn't be found.. This is just what I've posted here. I think its basically same error everytime. This is only app that i have problems with that accesses disk and memory a lot. Debug log: Coredump debug files: |
Ok first i wanted to restart with same command line as before. I synced in about 45-60 seconds and now is up and running fine. f... ME. and it is the updated daemon from dec 14 compiling. I checked the sha1. in about 10 minutes ill start it up like you asked. |
|
while testing all type of things i get a error on bad zcash-params. how is that even possible. does it read and write to the directory? someone give me some pointers on how to debug this with gdb or what i should do? my only option left is to wipe the hard drive and reinstall everything. I'd like to learn how to debug it tho. Any pointers, software, or processes i should try? debug log coredump debug |
|
it truly trashed the .zcash_param files. i tried a restart 4 times failed every time. it also took 2 tries downloading a good copy, which is strange too. after good download it started backup fine. |
|
here is a new clue. after running about 2-3 hours it tried to flush cache and crashed. 138gb free diskspace no errors. First time i've ever seen anything in db.log |
|
I'm not sure if anyone's ever going to be able to help given your reluctance to just run it as default so that there can be a determination as to where the issue lies. Debugging happens best when it's a step-by-step process and generally works best when started with a baseline of "does x work ever?" |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This issue tracker is only for technical issues related to komodod
General Komodo questions and/or support requests and are best directed to Discord
Describe the issue
Please provide a general summary of the issue you're experiencing
Can you reliably reproduce the issue?
If so, please list the steps to reproduce below:
Actual behaviour + errors
Tell us what happens instead including any noticable error output (any messages displayed on-screen when e.g. a crash occurred)
komodod[4090]: segfault at 18 ip 000055d76b92bd53 sp 00007ffffea7aa70 error 4 in komodod[55d76b8e2000+c68000]
The version of Komodo you were using:
latest master & dev
Machine specs:
OS Ubuntu 22.04.1 LTS
CPU: Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 36 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Vendor ID: GenuineIntel
Model name: Intel(R) Core(TM) i5-3470T CPU @ 2.90GHz
CPU family: 6
Model: 58
Thread(s) per core: 2
Core(s) per socket: 2
Socket(s): 1
Stepping: 9
CPU max MHz: 3600.0000
RAM: 16G
Disk size: 1TB
Disk Type (HD/SDD): SDD
Linux kernel version (uname -a):
Linux 138-backup-2 5.15.0-53-generic update master #59-Ubuntu SMP Mon Oct 17 18:53:30 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Compiler version (gcc -version):
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/11/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 11.3.0-1ubuntu1
22.04' --with-bugurl=file:///usr/share/doc/gcc-11/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-11 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --enable-cet --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-build-config=bootstrap-lto-lean --enable-link-serialization=222.04)Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1
Any extra information that might be useful in the debugging process.
This includes the relevant contents of
~/.komodo/debug.log. You can paste raw text, attach the file directly in the issue or link to the2022-11-26 23:59:40 Zcash version v0.7.2-beta1-d456be35a-dirty (2022-06-13 22:13:47 +0200)
2022-11-26 23:59:40 Using /16 prefix for IP bucketing
2022-11-26 23:59:40 Using the 'standard' SHA256 implementation
2022-11-26 23:59:40
2022-11-26 23:59:40 Komodo version v0.7.2-beta1-d456be35a-dirty (2022-06-13 22:13:47 +0200)
2022-11-26 23:59:40 Using OpenSSL version OpenSSL 1.1.1k 25 Mar 2021
2022-11-26 23:59:40 Using BerkeleyDB version Berkeley DB 6.2.23: (March 28, 2016)
The text was updated successfully, but these errors were encountered: