Error when setting num_candidates=1 and in dataset there's only one candidate during validation

[dimeldo]

When setting args.num_candidates to 1, and the actual length of the candidates list of each entry is 1, I get this error during validation:

ERROR:ignite.engine.engine.Engine:Current run is terminating due to exception: For binary cases, y_pred must be comprised of 0's and 1's..
ERROR:ignite.engine.engine.Engine:Engine run is terminating due to exception: For binary cases, y_pred must be comprised of 0's and 1's..
ERROR:ignite.engine.engine.Engine:Engine run is terminating due to exception: For binary cases, y_pred must be comprised of 0's and 1's..
Traceback (most recent call last):
  File "./train-regular.py", line 277, in <module>
    train()
  File "./train-regular.py", line 269, in train
    trainer.run(train_loader, max_epochs=args.n_epochs)
  File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 446, in run
    self._handle_exception(e)
  File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 410, in _handle_exception
    raise e
  File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 437, in run
    self._fire_event(Events.EPOCH_COMPLETED)
  File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 345, in _fire_event
    func(self, *(event_args + args), **kwargs)
  File "./train-regular.py", line 223, in <lambda>
    trainer.add_event_handler(Events.EPOCH_COMPLETED, lambda _: evaluator.run(val_loader))
  File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 446, in run
    self._handle_exception(e)
  File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 410, in _handle_exception
    raise e
  File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 433, in run
    hours, mins, secs = self._run_once_on_dataset()
  File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 399, in _run_once_on_dataset
    self._handle_exception(e)
  File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 410, in _handle_exception
    raise e
  File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 392, in _run_once_on_dataset
    self._fire_event(Events.ITERATION_COMPLETED)
  File "/usr/local/lib/python3.5/dist-packages/ignite/engine/engine.py", line 345, in _fire_event
    func(self, *(event_args + args), **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/ignite/metrics/metric.py", line 65, in iteration_completed
    self.update(output)
  File "/usr/local/lib/python3.5/dist-packages/ignite/metrics/accuracy.py", line 126, in update
    self._check_type((y_pred, y))
  File "/usr/local/lib/python3.5/dist-packages/ignite/metrics/accuracy.py", line 57, in _check_type
    self._check_binary_multilabel_cases((y_pred, y))
  File "/usr/local/lib/python3.5/dist-packages/ignite/metrics/accuracy.py", line 48, in _check_binary_multilabel_cases
    raise ValueError("For binary cases, y_pred must be comprised of 0's and 1's.")
ValueError: For binary cases, y_pred must be comprised of 0's and 1's.

[Riccorl]

I have the same problem, did you solve it?

[sshleifer]

I think num_candidates=1 won't work without significant code modification (removing multiple choice head). I think I've said something different in a different thread, which was wrong.

[Riccorl]

@sshleifer It seems like the problem is not the num_candidates=1. The model sees a binary classification task when there is only one candidate in the utterance sample.

[dimeldo]

@sshleifer Do you have a workaround if I don't have multiple candidates for my dataset?

[sshleifer]

It would require significant code modification.
I'd start by

• finding all mentions of multiple choice candidates (and removing them, fixing downstream if they are required)
• updating train.py to not use the CrossEntropyLoss/ or log it

there will probably be more bugs after that!

[dimeldo]

@sshleifer Yeah, probably not worth changing the code that much. I was talking about a hack, like having the multiple-choice candidates be some random string, say "Something.". Would that work?

[sshleifer]

Sounds like it might to me. Try it!