Clarification on criteria for valid and unambiguous

[davidwagner]

I wanted to request clarification on the criteria for an image to be a valid unambiguous image.

On the one hand: The instructions for taskers states that there are three criteria, which must all hold true (1. "Definitely yes, definitely no", 2. "at least 25% of the image", 3. "not truncated, occluded, depiction") for an image to count as valid and unambiguous.

On the other hand: The instructions also say that if taskers answer the first two questions with "Definitely yes" for one class and "Definitely no" for the other class, then questions 3, 4 are not asked, so I'm not clear on how we can know whether all three criteria hold.

I'm struggling to see how to reconcile these two facts.

So, consider an image where all taskers answered "Definitely yes" to one class, "Definitely no" to the other class, but the largest object is less than 25% of the image, or the image is truncated, or the image is occluded, or the image is a depiction. Is this considered to be a valid unambiguous bird-or-bicycle image? Based on the criteria, it sounds like the answer is "no", but based on the questions you will actually ask taskers, it sounds like you'll have no way to know that and will end up treating it as though the answer is "yes".

I suspect I'm misunderstanding something. Can you help me understand the rules?

[carlini]

Okay, so I think the problem is that the wording "If the tasker said that there definitely IS NOT one class, and there MAYBE IS the other class, then move on to the following additional questions." is pretty bad and we should fix. (I think technically it's correct, but it took me a minute to remember why we put it that way. So it should be fixed)

Here's what we wanted to mean. Let's suppose WLOG that we're talking about an image with a bicycle in it and no birds in the image. First, we check

• "that there definitely IS NOT one class" - this is satisfied. There are no birds.
• "there MAYBE IS the other class" - this is satisfied. There's (at least) maybe a bicycle.

This is the minimum test, and we can now go and check the rest (that it's sufficiently big, and not occluded, etc).

If either of these two conditions fail, then either

• Both a bird and a bicycle are contained in the image. This is never okay, and is an immediate disqualification. We want this rule so that we don't have to think about someone putting a stuffed bird (which isn't a bird class by our rules) on a real bicycle.
• There is no bird and no bicycle present. This would be bad, there's no good label then.

The ambiguity in your question (I think) comes down to the fact that we "MAYBE IS" what we mean by that is "AT LEAST MAYBE IS" for the intended class.

Does this clarify, hopefully?

[davidwagner]

Oh! Perfect, now it all makes sense to me. Thank you.