Reindexing Follower Indices fails

[SoniaSanzV]

Elasticsearch Version

8.18.0

Installed Plugins

No response

Java Version

n/a

OS Version

n/a

Problem Description

While testing the Kibana Management plugins in preparation for 9.0.0 readiness, we have encountered some reindexing errors when it comes to follower indices. These are the steps we took:

• Created two deployments in 7.17, let's call them "leader" and "follower". It doesn't matter if it's on cloud or stateful, the failure is the same.
• Created a remote cluster in the "follower", pointing the "leader"
• Also created some follower indices and auto-follow patterns, with and without documents.
• Upgraded the follower deployment to 8.18 without any issues.

The problem comes in the process of upgrading to 9.0.0.

• Upgrade assistant offers the possibility of mark as read-only and reindex the follower indices.
• Reindex always fails, with different errors depending on the index to have documents or not. The error for the ones with documents is quite long as it has one for every document.
• Mark as read-only works as expected, no issues.

If after upgrading the "follower" to v8.18 we upgrade the "leader" to v8.18 before trying to upgrade to v9.0, the outcome is the same.

The expected behavior would be to the reindexing succeed. Or maybe we shouldn't allow the users to reindexing follower indices but mark them as read-only.

Steps to Reproduce

• Go to https://console.qa.cld.elstc.co/home and create two deployments with version 7.17, one Leader and one Follower. You also can do it in stateful but in cloud it's easier.
• In the Follower one, navigate to Kibana > Stack Management > Remote Clusters and create one Remote Cluster pointing the Leader one. In the Elasticsearch endpoint URL use the Proxy Address of the Leader one. It can be found in Cloud > Hosted deployments > [leader deployment name] > Security
• Create two indices in the Leader one, one with documents and one without.
• In the Follower one, navigate to Kibana > Stack Management > Cross-Cluster Replication and create two follower indices, one for each one of the indices you have created in the Leader one.
• In the Follower one, navigate to Kibana > Stack Management > Upgrade Assistant. Address the issues if any and Upgrade Elastic to 8.18. It should succeed.
• Once the upgrade of the Follower is complete, come back to Kibana > Stack Management > Upgrade Assistant > Elasticsearch deprecation issues. Your should find a Critical issue for each of the follower indices you created.
• Click the Issue for the follower index with documents and, in the flyout, follow the steps to reindex it. Verify it fails in the Reindexing documents step.
• Click the Issue for the follower index without documents and, in the flyout, follow the steps to reindex it. Verify it fails in the Create alias step.

Logs (if relevant)

Error in the alias creation step for an index without docs:

ResponseError: status_exception Root causes: 
status_exception: can't modify aliases on indices [reindexed-v8-follower-leader-autofollow-not-upgrade-no-docs]; 
aliases of following indices are self-replicated from their leader indices

Error in the reindexing documents step for an index without docs (this index only contained two docs, but it would have one failure per doc):

  Error: Reindexing failed:  {
      "completed": true,
      "task": {
        "node": "-sFwXKU_TYG_Tn7UZD7XUA",
        "id": 98446,
        "type": "transport",
        "action": "indices:data/write/reindex",
        "status": {
          "total": 2,
          "updated": 0,
          "created": 0,
          "deleted": 0,
          "batches": 1,
          "version_conflicts": 0,
          "noops": 0,
          "retries": {
            "bulk": 0,
            "search": 0
          },
          "throttled_millis": 0,
          "requests_per_second": -1,
          "throttled_until_millis": 0
        },
        "description": "reindex from [follower-not-upgrade-with-docs] to [reindexed-v8-follower-not-upgrade-with-docs]",
        "start_time_in_millis": 1741790222292,
        "running_time_in_nanos": 3391471,
        "cancellable": true,
        "cancelled": false,
        "headers": {
          "X-elastic-product-origin": "kibana",
          "trace.id": "266533c444e040e5e86d5411b7c36833",
          "X-Opaque-Id": "b9634800-ee30-4c46-bfde-0e9188bac1f8;kibana:application:management:"
        }
      },
      "response": {
        "took": 2,
        "timed_out": false,
        "total": 2,
        "updated": 0,
        "created": 0,
        "deleted": 0,
        "batches": 1,
        "version_conflicts": 0,
        "noops": 0,
        "retries": {
          "bulk": 0,
          "search": 0
        },
        "throttled": "0s",
        "throttled_millis": 0,
        "requests_per_second": -1,
        "throttled_until": "0s",
        "throttled_until_millis": 0,
        "failures": [
          {
            "index": "reindexed-v8-follower-not-upgrade-with-docs",
            "id": "1",
            "cause": {
              "type": "status_exception",
              "reason": "a following engine does not accept operations without an assigned sequence number"
            },
            "status": 403
          },
          {
            "index": "reindexed-v8-follower-not-upgrade-with-docs",
            "id": "2",
            "cause": {
              "type": "status_exception",
              "reason": "a following engine does not accept operations without an assigned sequence number"
            },
            "status": 403
          }
        ]
      }
    }

[elasticsearchmachine]

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

[tlrx]

I think that this behavior has always existed since the CCR feature was implemented. At least I reproduces the same exact errors with CCR follower indices created on 6.8 and upgraded to 7.17 and then 8.18.

Because they are supposed to replicate an index from another cluster, some operations are forbidden and cannot be executed against CCR follower indices. This is the case of direct indexing operations (like Reindexing does) and alias changes.

In that case, the bug lies in the Upgrade Assistant (UA) that should probably not offer the option to reindex follower indices. Or if it does, it should be "Unfollow & Reindex" to allow converting the index to a regular index and then reindex the docs. Converting would mean to pause the replication, close the index, unfollow and reopen the index to allow reindexing. The replication could not be reenabled afterwards, so I'm not sure of the benefits compared to the "Mark as Read-only" option.

Note that "Mark as read-only" option does what it means, but replication is not stopped either but only silently fails.