table.sort is slightly broken

[igormunkin]

Tarantool version:

$ tarantool -V
Tarantool 1.10.3-33-gfd30346
Target: Linux-x86_64-Debug
Build options: cmake . -DCMAKE_INSTALL_PREFIX=/usr -DENABLE_BACKTRACE=yes
Compiler: /usr/bin/x86_64-pc-linux-gnu-gcc /usr/bin/x86_64-pc-linux-gnu-g++
C_FLAGS:-O2 -pipe -march=skylake -fexceptions -funwind-tables -fno-omit-frame-pointer -fno-stack-protector -fno-common -fopenmp -msse2 -mavx -std=c11 -Wall -Wextra -Wno-strict-aliasing -Wno-char-subscripts -Wno-format-truncation -fno-gnu89-inline -Wno-cast-function-type -Werror
CXX_FLAGS:-O2 -pipe -march=skylake -fexceptions -funwind-tables -fno-omit-frame-pointer -fno-stack-protector -fno-common -fopenmp -msse2 -mavx -std=c++11 -Wall -Wextra -Wno-strict-aliasing -Wno-char-subscripts -Wno-format-truncation -Wno-invalid-offsetof -Wno-cast-function-type -Werror

OS version:

$ cat /etc/os-release
NAME=Gentoo
ID=gentoo
PRETTY_NAME="Gentoo/Linux"
ANSI_COLOR="1;32"
HOME_URL="https://www.gentoo.org/"
SUPPORT_URL="https://www.gentoo.org/support/"
BUG_REPORT_URL="https://bugs.gentoo.org/"

Bug description:

Table is not sorted properly after table.sort.

Steps to reproduce:

Run the chunk below with tarantool (or another Lua 5.1 implementation: lua, luajit, ujit, raptorjit, luajit2 (openresty))

local samples = {
  desc = {
    origin = { 4981, 5006, 5030, 5090, 5067, 5156, 5210, 4942, 5185, 5114, },
    sorted = { 5210, 5185, 5156, 5114, 5090, 5067, 5030, 5006, 4981, 4942, },
    comp = function(a, b) return a >= b end,
  },
  asc = {
    origin = { 5185, 5156, 5114, 5067, 5090, 5006, 4942, 5210, 4981, 5030, },
    sorted = { 4942, 4981, 5006, 5030, 5067, 5090, 5114, 5156, 5185, 5210, },
    comp = function(a, b) return a <= b end,
  },
}

for direction, sample in pairs(samples) do
  local status, err = pcall(function (direction)
    table.sort(direction.origin, direction.comp)
    for _ = 1, #direction.origin do
      assert(direction.origin[_] == direction.sorted[_],
             string.format("Not sorted since #%d", _))
    end
  end, sample)
  if not status then print(direction, err) end
end

The output is the one similar to the following:

$ tarantool table-sort.lua
desc	/home/imun/table-sort.lua:18: Not sorted since #9
asc	/home/imun/table-sort.lua:18: Not sorted since #9

[igormunkin]

According to the Lua reference manual, the comparator to be used for sorting has to return strict comparison result. Thereby I propose to prepare request to enrich tarantool docs and close the issue with wontfix reason.