Possible bug: Stasher can panic on some inputs

[ryanwalls]

We are researching what data is getting passed to the stasher to make it panic, but here is the stack trace we hit:

Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: panic: reflect: call of reflect.Value.Type on zero Value
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: 
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: goroutine 96 [running]:
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: panic(0xe1a700, 0xc8205406a0)
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: #011/usr/local/go/src/runtime/panic.go:481 +0x3e6
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: reflect.Value.Type(0x0, 0x0, 0x0, 0x0, 0x0)
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: #011/usr/local/go/src/reflect/value.go:1671 +0x7b
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: encoding/gob.(*Encoder).EncodeValue(0xc82019b7c0, 0x0, 0x0, 0x0, 0x0, 0x0)
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: #011/usr/local/go/src/encoding/gob/encoder.go:230 +0x229
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: encoding/gob.(*Encoder).Encode(0xc82019b7c0, 0x0, 0x0, 0x0, 0x0)
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: #011/usr/local/go/src/encoding/gob/encoder.go:174 +0x72
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: github.com/3dsim/workflow/vendor/github.com/sclasen/swfsm/fsm.(*Stasher).Stash(0xc82052b470, 0x0, 0x0, 0x6)
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: #011/go/src/github.com/3dsim/workflow/vendor/github.com/sclasen/swfsm/fsm/fsm_models.go:188 +0x9e
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: github.com/3dsim/workflow/vendor/github.com/sclasen/swfsm/fsm.(*FSM).Tick(0xc8205b6a20, 0xc820417450, 0xe4ce40, 0x0, 0x0, 0x0, 0x10, 0x0, 0x0)
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: #011/go/src/github.com/3dsim/workflow/vendor/github.com/sclasen/swfsm/fsm/fsm.go:476 +0x10db
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: github.com/3dsim/workflow/vendor/github.com/sclasen/swfsm/fsm.(*FSM).handleDecisionTask(0xc8205b6a20, 0xc820417450)
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: #011/go/src/github.com/3dsim/workflow/vendor/github.com/sclasen/swfsm/fsm/fsm.go:342 +0x3d
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: github.com/3dsim/workflow/vendor/github.com/sclasen/swfsm/fsm.(*FSM).(github.com/3dsim/workflow/vendor/github.com/sclasen/swfsm/fsm.handleDecisionTask)-fm(0xc820417450)
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: #011/go/src/github.com/3dsim/workflow/vendor/github.com/sclasen/swfsm/fsm/fsm.go:338 +0x2a
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: github.com/3dsim/workflow/vendor/github.com/sclasen/swfsm/fsm.(*CallingGoroutineDispatcher).DispatchTask(0x1727c50, 0xc820417450, 0xc8201f3450)
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: #011/go/src/github.com/3dsim/workflow/vendor/github.com/sclasen/swfsm/fsm/dispatchers.go:19 +0x26
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: github.com/3dsim/workflow/vendor/github.com/sclasen/swfsm/fsm.(*FSM).dispatchTask(0xc8205b6a20, 0xc820417450)
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: #011/go/src/github.com/3dsim/workflow/vendor/github.com/sclasen/swfsm/fsm/fsm.go:338 +0x89
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: github.com/3dsim/workflow/vendor/github.com/sclasen/swfsm/fsm.(*FSM).(github.com/3dsim/workflow/vendor/github.com/sclasen/swfsm/fsm.dispatchTask)-fm(0xc820417450)
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: #011/go/src/github.com/3dsim/workflow/vendor/github.com/sclasen/swfsm/fsm/fsm.go:303 +0x2a
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: github.com/3dsim/workflow/vendor/github.com/sclasen/swfsm/poller.(*DecisionTaskPoller).PollUntilShutdownBy(0xc820370500, 0xc82052b460, 0xc8201a4240, 0x20, 0xc82052b490, 0xc82052b4a0)
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: #011/go/src/github.com/3dsim/workflow/vendor/github.com/sclasen/swfsm/poller/poller.go:129 +0x68d
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: created by github.com/3dsim/workflow/vendor/github.com/sclasen/swfsm/fsm.(*FSM).Start
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: #011/go/src/github.com/3dsim/workflow/vendor/github.com/sclasen/swfsm/fsm/fsm.go:303 +0x395

If it helps, this happened on a signal workflow event that happened after we had panicked in some of our code. So these were the lines just before the stack trace above.

Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: 2016/07/18 19:31:35 component=FSM name=SingleBeadWorkflowDecider type=SingleBeadWorkflow id=712 action=tick at=decided-event state=thermalsolver next-state=failed decisions=1
Jul 18 19:31:35 ip-172-31-42-65 docker/e08c77731efd[2614]: 2016/07/18 19:31:35 component=FSM name=SingleBeadWorkflowDecider type=SingleBeadWorkflow id=712 action=tick at=history id=1442 type=WorkflowExecutionSignaled

Perhaps the framework should handle panics from the gob encoder? Or perhaps it should validate the data input before it calls the encoder?

[sclasen]

Thanks for the report @ryanwalls, can you tell me if the FSM.DataType was set on the FSM in question, and if the data was/was not nil?

[ryanwalls]

Took us a while to figure out what was happening. Yep the data was nil. FSM.DataType was set on the FSM.