Assertion error at webserver start

[francescocarzaniga]

Environment

• Python version: 3.8.6
• NetBox version: 2.10.1
• PostgreSQL: 13 (external)
• Redis: 6 (external)
• OS: OmniOS 5.11

Steps to Reproduce

Follow the steps in the guide. Everything works fine, Netbox connects to database and to Redis cache.
Arrive at the point where you start the Werkzeug web server.
Curl the host at the specified port.

Expected Behavior

Netbox should start and begin serving the pages. Curl should return the homepage.

Observed Behavior

Werkzeug crashes with

(venv) root@netbox:/opt/netbox/netbox# python3 manage.py runserver 0.0.0.0:8000 --insecure
Performing system checks...

System check identified no issues (0 silenced).
December 19, 2020 - 09:42:43
Django version 3.1.3, using settings 'netbox.settings'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.
Assertion failed: section != NULL, file prof_tree.c, line 528, function profile_node_iterator

[DanSheps]

This appears to be a bug in your kerberos package.

I am going to convert this to a discussion, perhaps someone else has seen this issue before.

[candlerb]

What does /etc/krb5.conf contain?

To try to debug this, you could run python3 under gdb:

# gdb python3
(gdb) run manage.py runserver 0.0.0.0:8000 --insecure

and when it crashes, type bt to get a backtrace.

Otherwise, maybe the python3 binary supplied by your OS (OmniOS) is broken. You could try installing a different version of python, e.g. using pyenv

[francescocarzaniga]

I don't think the binary is broken, I use the same package in different zones on the same system and I don't have this problem.
The content of /etc/krb5/krb5.conf is default:

#
# CDDL HEADER START
#
# The contents of this file are subject to the terms of the
# Common Development and Distribution License (the "License").
# You may not use this file except in compliance with the License.
#
# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
# or http://www.opensolaris.org/os/licensing.
# See the License for the specific language governing permissions
# and limitations under the License.
#
# When distributing Covered Code, include this CDDL HEADER in each
# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
# If applicable, add the following below this CDDL HEADER, with the
# fields enclosed by brackets "[]" replaced with your own identifying
# information: Portions Copyright [yyyy] [name of copyright owner]
#
# CDDL HEADER END
#
#
# Copyright 2007 Sun Microsystems, Inc.  All rights reserved.
# Use is subject to license terms.
#
# ident "%Z%%M% %I%     %E% SMI"
#

# krb5.conf template
# In order to complete this configuration file
# you will need to replace the __<name>__ placeholders
# with appropriate values for your network and uncomment the
# appropriate entries.
#
[libdefaults]
#        default_realm = ___default_realm___

[realms]
#        ___default_realm___ = {
#                kdc = ___master_kdc___
#                kdc = ___slave_kdc1___
#                kdc = ___slave_kdc2___
#                kdc = ___slave_kdcN___
#                admin_server = ___master_kdc___
#        }

[domain_realm]
#       ___domainname___ = ___default_realm___

[logging]
        default = FILE:/var/krb5/kdc.log
        kdc = FILE:/var/krb5/kdc.log
        kdc_rotate = {

# How often to rotate kdc.log. Logs will get rotated no more
# often than the period, and less often if the KDC is not used
# frequently.

                period = 1d

# how many versions of kdc.log to keep around (kdc.log.0, kdc.log.1, ...)

                versions = 10
        }

[appdefaults]
        kinit = {
                renewable = true
                forwardable= true
        }

gdb does not show a backtrace, and using --traceback does not work either. Here is the output:

(venv) root@netbox:/opt/netbox/netbox# gdb python3
GNU gdb (GDB) 10.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-sun-solaris2.11".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python3...
(No debugging symbols found in python3)
(gdb) run manage.py runserver 0.0.0.0:8000 --insecure
Starting program: /opt/netbox-2.10.1/venv/bin/python3 manage.py runserver 0.0.0.0:8000 --insecure
[Thread debugging using libthread_db enabled]
[New Thread 1 (LWP 1)]
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
Assertion failed: section != NULL, file prof_tree.c, line 528, function profile_node_iterator
[Inferior 1 (process 23598    ) exited with code 0372]
(gdb) bt
No stack.
(gdb)

[candlerb]

You're right, that doesn't really help. The other thing you could try is strace:

strace -f -s128 python3 manage.py runserver 0.0.0.0:8000 --insecure

This will show you the last thing it accesses before it crashes (e.g. did it just read krb5.conf ?)

However, if python is crashing, this is really a problem with your upstream distro. Whatever Netbox does in Python source, cpython should never crash. Therefore it's clearly not an issue with Netbox or any of its dependent Python libraries like Django.

It's possible your distro has built kerberos libraries in an unusual way, e.g. with profiling enabled, that exercises a code path that other distros don't use.

If you just want a working system, I think you should run Netbox under Linux in a VM. Ubuntu and CentOS are the two which are most used and tested by Netbox users.

[candlerb]

Also try mv /etc/krb5.conf /etc/krb5.conf.removed

[francescocarzaniga]

The output of truss when the error appears:

waitid(P_PID, 549, 0xFFFFFC7FFFDFC500, WEXITED|WTRAPPED) (sleeping...)
    Received signal #18, SIGCLD, in waitid() [default]
      siginfo: SIGCLD CLD_DUMPED pid=549 status=0x0006
waitid(P_PID, 549, 0xFFFFFC7FFFDFC500, WEXITED|WTRAPPED) = 0
clock_getres(4, 0xFFFFFC7FFFDFD2A0)             = 0
getrlimit(RLIMIT_NOFILE, 0xFFFFFC7FFFDFD0F0)    = 0
open("/dev/poll", O_RDWR|O_CLOEXEC)             = 3
brk(0x04A92000)                                 = 0
brk(0x04AB2000)                                 = 0
brk(0x04AD2000)                                 = 0
brk(0x04B53000)                                 = 0
so_socket(PF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0, 0x00000000, SOV_DEFAULT) = 4
so_socket(PF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0, 0x00000000, SOV_DEFAULT) = 5
so_socketpair(0xFFFFFC7FFFDFD1B0)               = 0
close(4)                                        = 0
getsockname(6, 0xFFFFFC7FFFDFCAC0, 0xFFFFFC7FFFDFCAAC, SOV_DEFAULT) = 0
getsockname(5, 0xFFFFFC7FFFDFCAC0, 0xFFFFFC7FFFDFCAAC, SOV_DEFAULT) = 0
fcntl(6, F_GETFL)                               = 2
fcntl(6, F_SETFL, FWRITE|FNONBLOCK)             = 0
fcntl(5, F_GETFL)                               = 2
fcntl(5, F_SETFL, FWRITE|FNONBLOCK)             = 0
brk(0x04B63000)                                 = 0
sigaction(SIGINT, 0xFFFFFC7FFFDFF7F0, 0xFFFFFC7FFFDFF890) = 0
sigaction(SIGTERM, 0xFFFFFC7FFFDFF7F0, 0xFFFFFC7FFFDFF890) = 0
brk(0x04B73000)                                 = 0
brk(0x04B83000)                                 = 0
brk(0x04B93000)                                 = 0
brk(0x04BA3000)                                 = 0
brk(0x04BB3000)                                 = 0
brk(0x04BC3000)                                 = 0
brk(0x04BD3000)                                 = 0
brk(0x04BE3000)                                 = 0
brk(0x04BF3000)                                 = 0
close(6)                                        = 0
close(5)                                        = 0
brk(0x04C03000)                                 = 0
close(3)                                        = 0
brk(0x04C13000)                                 = 0
brk(0x04C23000)                                 = 0
brk(0x04C33000)                                 = 0
brk(0x04C43000)                                 = 0
brk(0x04C53000)                                 = 0
brk(0x04C63000)                                 = 0
brk(0x04C73000)                                 = 0
brk(0x04C83000)                                 = 0
brk(0x04C93000)                                 = 0
brk(0x04CA3000)                                 = 0
brk(0x04CB3000)                                 = 0
brk(0x04CC3000)                                 = 0
brk(0x04CD3000)                                 = 0
brk(0x04CE3000)                                 = 0
brk(0x04CF3000)                                 = 0
brk(0x04D03000)                                 = 0
brk(0x04D13000)                                 = 0
brk(0x04D23000)                                 = 0
brk(0x04D33000)                                 = 0
brk(0x04D43000)                                 = 0
brk(0x04D53000)                                 = 0
brk(0x04D63000)                                 = 0
brk(0x04D73000)                                 = 0
brk(0x04D83000)                                 = 0
brk(0x04D93000)                                 = 0
brk(0x04DA3000)                                 = 0
brk(0x04DB3000)                                 = 0
brk(0x04DC3000)                                 = 0
brk(0x04DD3000)                                 = 0
brk(0x04DE3000)                                 = 0
brk(0x04DF3000)                                 = 0
brk(0x04E03000)                                 = 0
brk(0x04E13000)                                 = 0
brk(0x04E23000)                                 = 0
brk(0x04E33000)                                 = 0
brk(0x04E43000)                                 = 0
brk(0x04E53000)                                 = 0
brk(0x04E63000)                                 = 0
brk(0x04E73000)                                 = 0
brk(0x04E83000)                                 = 0
brk(0x04E93000)                                 = 0
sigaltstack(0x00000000, 0xFFFFFC7FFFDFF8F0)     = 0
sigaltstack(0xFFFFFC7FC2F28A50, 0x00000000)     = 0
mmap(0x00000000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, 4294967295, 0) = 0xFFFFFC7FEE820000
open("/dev/dtrace/helper", O_RDWR)              = 3
ioctl(3, DTRACEHIOC_REMOVE, 0x00000000)         = 0
close(3)                                        = 0
_exit(4294967290)

I couldn't get anything useful out of this but I'm not good enough with this stuff.

Anyway the problem turned out to be libpq.so. I recompiled psycopg2 with a different libpq and now Netbox works (why that is is beyond me). If somebody can shed some light it would be great, otherwise the issue is solved for me.