@arisliang Thank you for this FR !

Could we imagine that the actual EarlyStopping is just a specific case of the more general NoImprovmentHandler you suggested? I mean NoImprovmentHandler + Stop Policy Function = EarlyStopping? And maybe it's worth considering other trending policies than no-improvment like too-fast-improvment or whatever. What do you think ?

@sdesrozis Thanks for the response.

Yes exactly. I'd imagine EarlyStopping would be a specific case of the more general NoImprovmentHandler by passing in the Stop Policy Function. So NoImprovement determines a scenario where no improvement is made, and decouples the action to be taken, be it stop training, logging, or other things.

I haven't thought about other trending policies. It's certainly possible as another enhancement to make things even more general.

Not sure which is more suitable, no-improvement may even be thought of as an scenario-based event, and actions like stop training can then register as an event handler when no-improvement happens.

@arisliang I will try to write a piece of code to illustrate my thoughts.

Ok so the actual EarlyStopping is the following.

class EarlyStopping:
    
    def __init__(self, patience, score_function, trainer):
        self.score_function = score_function
        self.patience = patience
        self.trainer = trainer
        self.counter = 0
        self.best_score = float('-inf')
        
    def __call__(self, engine):
        score = self.score_function(engine)

        if score <= self.best_score:
            self.best_score = score
            self.counter += 1
            print("EarlyStopping: %i / %i" % (self.counter, self.patience))
            if self.counter >= self.patience:
                print("EarlyStopping: Stop training")
                self.trainer.terminate()
        else:
            self.best_score = score
            self.counter = 0

We could think about a more flexible implementation based on a score_function (as today) associated to a stop_function (i.e. what to do whether no improvment) and a pass_function (i.e. what to do whether improvment).

class TriggerHandlerState:

    def __init__(self):
        self.best_score = float('-inf')
        self.counter = 0

class TriggerHandler:

    def __init__(self, score_function, pass_function, stop_function):
        self.score_function = score_function
        self.pass_function= pass_function
        self.stop_function = stop_function
        self.state = TriggerHandlerState()

    def __call__(self, engine):

        score = self.score_function(engine, self.state)

        if self._update_and_check_state(score):
            self.stop_function(self.state)
        else:
            self.pass_function(self.state)

    def _update_and_check_state(self, score):
        if score <= self.state.best_score:
            self.state.best_score = score
            self.state.counter += 1
            return True
        else:
            return False

What do you think ?

Looks great for both current need and flexible. What is stuck_function? Is it just typo of pass_function?

Looks great for both current need and flexible. What is stuck_function? Is it just typo of pass_function?

Yeah, a typo. Fixed. Let's see whether this design works although the naming, concepts and scope need to be carefully considered.

Hi @sdesrozis and @arisliang, I am interested in taking this up.
From what I understand, we should create a Base class called something like NoImprovementHandler and then use it to create the EarlyStopping because it is a special case of that.

I was wondering, should we also keep the decision function of the Base class fixed as
if self.counter >= self.patience
or are there some use-cases where allowing the user to change this would also help?