Skip to content
This repository has been archived by the owner on Oct 31, 2023. It is now read-only.

Zeroeth template in compare_integer.json can never return true #37

Open
ikb-a opened this issue Nov 16, 2022 · 0 comments
Open

Zeroeth template in compare_integer.json can never return true #37

ikb-a opened this issue Nov 16, 2022 · 0 comments

Comments

@ikb-a
Copy link

ikb-a commented Nov 16, 2022

First and foremost, it goes without saying, but great repo and dataset!

Moving on to the issue, I believe there's a slight error in the zeroeth template of compare_integer.json

Specifically, the zeroeth template (with indenting for legibility) is as follows:

    "constraints": [
      {
        "params": [
          1,
          3
        ],
        "type": "OUT_NEQ"
      }
    ],
    "nodes": [
      {
        "inputs": [],
        "type": "scene"
      },
      {
        "inputs": [
          0
        ],
        "side_inputs": [
          "<Z>",
          "<C>",
          "<M>",
          "<S>"
        ],
        "type": "filter_count"
      },
      {
        "inputs": [],
        "type": "scene"
      },
      {
        "inputs": [
          2
        ],
        "side_inputs": [
          "<Z2>",
          "<C2>",
          "<M2>",
          "<S2>"
        ],
        "type": "filter_count"
      },
      {
        "inputs": [
          1,
          3
        ],
        "type": "equal_integer"
      }
    ],
    "params": [
      {
        "name": "<Z>",
        "type": "Size"
      },
      {
        "name": "<C>",
        "type": "Color"
      },
      {
        "name": "<M>",
        "type": "Material"
      },
      {
        "name": "<S>",
        "type": "Shape"
      },
      {
        "name": "<Z2>",
        "type": "Size"
      },
      {
        "name": "<C2>",
        "type": "Color"
      },
      {
        "name": "<M2>",
        "type": "Material"
      },
      {
        "name": "<S2>",
        "type": "Shape"
      }
    ],
    "text": [
      "Are there an equal number of <Z> <C> <M> <S>s and <Z2> <C2> <M2> <S2>s?",
      "Are there the same number of <Z> <C> <M> <S>s and <Z2> <C2> <M2> <S2>s?",
      "Is the number of <Z> <C> <M> <S>s the same as the number of <Z2> <C2> <M2> <S2>s?"
    ]
  },

Note the constraint is OUT_NEQ between nodes 1 and 3. However, nodes 1 and 3 are filter_count nodes rather than filter nodes, so the constraint is that the counts (rather than the objects found) must differ. Consequently, the answer is always false (there cannot be an equal number of objects, because our constraint forces the object count to differ). I threw together a quick script to try and check the answers in the CLEVR training set, and I believe it supports my conclusions.

It's not a serious issue, but it confused me and took me a while to figure out what was going on -- I hope this spares someone else some debugging time. Cheers!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant