[Cluster.NotALeader] No write operations are allowed directly on this database. Writes must pass through the leader

[kostia-sukernik]

Hi,

I am getting this exception when running a write query against a neo4j causal cluster version 4.3.6, this work fine with causal cluster of version 4.1.1. In both cases the query is sent to a follower server, py2neo is 2021.2.3.

Graph is initialized using:
Graph('neo4j://<dns_address>:7687', **settings)

settings:
{'user': 'neo4j', 'password': password, 'name': <db_name>, 'routing': True, 'routing_refresh_ttl': 300}

Any idea what can cause this behavior?

Thanks

[abronin]

We have the same issue with dbms.routing.default_router=SERVER (which looks to be mandatory for exposing Neo4j Browser using LoadBalancer in Kubernetes).

[technige]

How are you running the query?

[abronin]

Thank for answering, @technige!

This is how we run the query:
self.graph.create(user)

We have a Model

class User(GraphObject):
    __primarykey__ = "email"

    createdAt = Property()
    name = Property()
    deactivatedEmail = Property()

We have a Graph:

self.graph = Graph(
    'neo4j://' + NEO4J_URL + ':7687',
    user='neo4j', password='password', name='neo4j', routing=True, routing_refresh_ttl=300
)

[abronin]

We have this error in both 4.3.9 and 4.4.1 with dbms.routing.default_router=SERVER
Failed to execute update after 3 tries

[abronin]

This is complete example of failing code. This code is failing with the latest py2neo and with the neo4j 4.3.9 or 4.4.1 deployed using community based helm chart with default configuration (including default_router=SERVER).

root@backend-568c7588b-thhpt:/usr/src/app# python
Python 3.7.12 (default, Dec 21 2021, 11:35:10) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from py2neo import Graph
>>> graph = Graph('neo4j://neo4j-host:7687', user='neo4j', password='password', routing=True)
>>> from models import User
>>> user = User('andrey', '[email protected]')
>>> graph.create(user)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/py2neo/database.py", line 591, in create
    self.update(lambda tx: tx.create(subgraph))
  File "/usr/local/lib/python3.7/site-packages/py2neo/database.py", line 445, in update
    self._update(cypher, timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/py2neo/database.py", line 490, in _update
    raise WriteServiceUnavailable("Failed to execute update after %r tries" % n)
py2neo.errors.WriteServiceUnavailable: Failed to execute update after 3 tries

[RazLightlytics]

@kostia-sukernik and I are talking about another issue, the error we see after running write queries is different:

graph = Graph(f"neo4j://neo4j:7687", name=database, password=password)
graph.run("create (x)")

Internal server error: [Cluster.NotALeader] No write operations are allowed directly on this database. Writes must pass through the leader. The role of this server is: FOLLOWER

But with the official neo4j python driver and the same cluster configuration we are not getting this error.

[abronin]

I can only confirm that when I run query your way I have the same issue:

>>> graph = Graph('neo4j://neo4j:7687', user='neo4j', password=password)
>>> graph.run("create (x)")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/py2neo/database.py", line 405, in run
    return self.auto().run(cypher, parameters, **kwparameters)
  File "/usr/local/lib/python3.7/site-packages/py2neo/database.py", line 980, in run
    readonly=self.readonly)
  File "/usr/local/lib/python3.7/site-packages/py2neo/client/__init__.py", line 1331, in auto_run
    cx.pull(result, n=pull)
  File "/usr/local/lib/python3.7/site-packages/py2neo/client/bolt.py", line 950, in pull
    self._audit(self._transaction)
  File "/usr/local/lib/python3.7/site-packages/py2neo/client/bolt.py", line 759, in _audit
    task.audit()
  File "/usr/local/lib/python3.7/site-packages/py2neo/client/bolt.py", line 1101, in audit
    item.audit()
  File "/usr/local/lib/python3.7/site-packages/py2neo/client/bolt.py", line 1101, in audit
    item.audit()
  File "/usr/local/lib/python3.7/site-packages/py2neo/client/bolt.py", line 1264, in audit
    raise self._failure
py2neo.errors.ClientError: [Cluster.NotALeader] No write operations are allowed directly on this database. Writes must pass through the leader. The role of this server is: FOLLOWER

[SimonThordal]

I can confirm that when setting up a cluster through docker using the official image I get the `

My conf file:

# Setting that specifies how much memory Neo4j is allowed to use for the page cache.
dbms.memory.pagecache.size=100M

# Setting that specifies the initial JVM heap size.
dbms.memory.heap.initial_size=100M

# Strategy that the instance will use to determine the addresses of other members.
causal_clustering.discovery_type=DNS

# The network addresses of an initial set of Core cluster members that are available to bootstrap this Core or Read Replica instance.
# If the DNS strategy is used, the addresses are fetch using the DNS A records.
causal_clustering.initial_discovery_members=neo4j-network:5000

# Address (the public hostname/IP address of the machine)
# and port setting that specifies where this instance advertises for discovery protocol messages from other members of the cluster.
causal_clustering.discovery_advertised_address=$(hostname -i)

# Address (the public hostname/IP address of the machine)
# and port setting that specifies where this instance advertises for Raft messages within the Core cluster.
causal_clustering.raft_advertised_address=$(hostname)

 # Address (the public hostname/IP address of the machine)
 # and port setting that specifies where this instance advertises for requests for transactions in the transaction-shipping catchup protocol.
causal_clustering.transaction_advertised_address=$(hostname)

# Enable server side routing
dbms.routing.enabled=true

# Use server side routing for neo4j:// protocol connections.
dbms.routing.default_router=SERVER

# The advertised address for the intra-cluster routing connector.
dbms.routing.advertised_address=0.0.0.0

The Docker compose file:

version: '3.8'

# Custom top-level network
networks:
  neo4j-internal:

services:

  core1:
    # Docker image to be used
    image: ${NEO4J_DOCKER_IMAGE}

    # Hostname
    hostname: core1

    # Service-level network, which specifies the networks, from the list of the top-level networks (in this case only neo4j-internal), that the server will connect to.
    # Adds a network alias (used in neo4j.conf when configuring the discovery members)
    networks:
      neo4j-internal:
        aliases:
          - neo4j-network

    # The ports that will be accessible from outside the container - HTTP (7474) and Bolt (7687).
    ports:
      - "7474:7474"
      - "7687:7687"
      - "7688:7688"

    # Uncomment the volumes to be mounted to make them accessible from outside the container.
    volumes:
      - ./conf/:/conf/
      - ./core1/data:/var/lib/neo4j/data
      - ./core1/logs:/var/lib/neo4j/logs
      - ./core1/conf:/var/lib/neo4j/conf
      - ./core1/import:/var/lib/neo4j/import
      #- ./core1/metrics:/var/lib/neo4j/metrics
      #- ./core1/licenses:/var/lib/neo4j/licenses
      #- ./core1/ssl:/var/lib/neo4j/ssl
      #- ./plugins:/var/lib/neo4j/plugins

    # Passes the following environment variables to the container
    environment:
      - NEO4J_ACCEPT_LICENSE_AGREEMENT
      - NEO4J_AUTH
      - EXTENDED_CONF
      - NEO4J_EDITION
      - NEO4J_dbms_mode=CORE

    # Simple check testing whether the port 7474 is opened.
    # If so, the instance running inside the container is considered as "healthy".
    # This status can be checked using the "docker ps" command.
    healthcheck:
      test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider localhost:7474 || exit 1"]

    # Set up the user
    user: ${USER_ID}:${GROUP_ID}

  core2:
    image: ${NEO4J_DOCKER_IMAGE}
    hostname: core2
    networks:
      neo4j-internal:
        aliases:
          - neo4j-network
    ports:
      - "7574:7474"
      - "7787:7687"
      - "7788:7688"
    volumes:
      - ./conf/:/conf/
      - ./core2/data:/var/lib/neo4j/data
      - ./core2/logs:/var/lib/neo4j/logs
      - ./core2/conf:/var/lib/neo4j/conf
      - ./core2/import:/var/lib/neo4j/import
      #- ./core2/metrics:/var/lib/neo4j/metrics
      #- ./core2/licenses:/var/lib/neo4j/licenses
      #- ./core2/ssl:/var/lib/neo4j/ssl
      #- ./plugins:/var/lib/neo4j/plugins
    environment:
      - NEO4J_ACCEPT_LICENSE_AGREEMENT
      - NEO4J_AUTH
      - EXTENDED_CONF
      - NEO4J_EDITION
      - NEO4J_dbms_mode=CORE
    healthcheck:
      test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider localhost:7474 || exit 1"]
    user: ${USER_ID}:${GROUP_ID}

  core3:
    image: ${NEO4J_DOCKER_IMAGE}
    hostname: core3
    networks:
      neo4j-internal:
        aliases:
          - neo4j-network
    ports:
      - "7674:7474"
      - "7887:7687"
      - "7888:7688"
    volumes:
      - ./conf/:/conf/
      - ./core3/data:/var/lib/neo4j/data
      - ./core3/logs:/var/lib/neo4j/logs
      - ./core3/conf:/var/lib/neo4j/conf
      - ./core3/import:/var/lib/neo4j/import
      #- ./core3/metrics:/var/lib/neo4j/metrics
      #- ./core3/licenses:/var/lib/neo4j/licenses
      #- ./core3/ssl:/var/lib/neo4j/ssl
      #- ./plugins:/var/lib/neo4j/plugins
    environment:
      - NEO4J_ACCEPT_LICENSE_AGREEMENT
      - NEO4J_AUTH
      - EXTENDED_CONF
      - NEO4J_EDITION
      - NEO4J_dbms_mode=CORE
    healthcheck:
      test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider localhost:7474 || exit 1"]
    user: ${USER_ID}:${GROUP_ID}

  readreplica1:
    image: ${NEO4J_DOCKER_IMAGE}
    hostname: replica1
    networks:
      neo4j-internal:
        aliases:
          - neo4j-network
    ports:
      - "7477:7474"
      - "7690:7687"
      - "7988:7688"
    volumes:
      - ./conf/:/conf/
      - ./replica1/data:/var/lib/neo4j/data
      - ./replica1/logs:/var/lib/neo4j/logs
      - ./replica1/conf:/var/lib/neo4j/conf
      - ./replica1/import:/var/lib/neo4j/import
      #- ./replica1/metrics:/var/lib/neo4j/metrics
      #- ./replica1/licenses:/var/lib/neo4j/licenses
      #- ./replica1/ssl:/var/lib/neo4j/ssl
      #- ./plugins:/var/lib/neo4j/plugins
    environment:
      - NEO4J_ACCEPT_LICENSE_AGREEMENT
      - NEO4J_AUTH
      - EXTENDED_CONF
      - NEO4J_EDITION
      - NEO4J_dbms_mode=READ_REPLICA
    healthcheck:
      test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider localhost:7474 || exit 1"]
    user: ${USER_ID}:${GROUP_ID}

[iamyoudi]

@technige technige We are also facing the same issue (py2neo.errors.WriteServiceUnavailable) after switching to cluster and enabling routing, could you please give any insight on this?

repo= Repository(ServiceProfile(profile=neo4j://neo4j:password@localhost:7687, routing=True, protocol='neo4j'))
repo.save(node)

Neo4j's official driver is not giving any error.

Thank you for this wonderful library.

[iamyoudi]

@technige Could you please provide some insight ?

@SimonThordal @abronin @RazLightlytics @kostia-sukernik was wondering if you guys found any solution?