Mutex deadlock

[nikolaianohyn]

In the function bellow next situation: we lock mutex seg.Lock() and realize it only after function is done. By the way seg.parent.RemoveSubsegment(seg) try to lock parent segment that already locked.

// CloseAndStream closes a subsegment and sends it.
func (seg *Segment) CloseAndStream(err error) {
	// If SDK is disabled then return
	if SdkDisabled() {
		return
	}

	seg.Lock()
	defer seg.Unlock()

	if seg.parent != nil {
		logger.Debugf("Ending subsegment named: %s", seg.Name)
		seg.EndTime = float64(time.Now().UnixNano()) / float64(time.Second)
		seg.InProgress = false
		seg.Emitted = true
		if seg.parent.RemoveSubsegment(seg) {
			logger.Debugf("Removing subsegment named: %s", seg.Name)
		}
	}

	if err != nil {
		seg.addError(err)
	}

	// If segment is dummy we return
	if seg.Dummy {
		return
	}

	seg.beforeEmitSubsegment(seg.parent)
	seg.emit()
}

Parent segment is locked because in parallel we call send() function for parent segment. send() locks current segment(which is parent) and then call packSegments which try to lock the subsegment which already locked upper. SO DEADLOCK.

Guys, I've been debugging it for a week. Pls write more tests for routine leaks and deadlocks and etc...

The working solution for this:

// CloseAndStream closes a subsegment and sends it.
func (seg *Segment) CloseAndStream(err error) {
	// If SDK is disabled then return
	if SdkDisabled() {
		return
	}
	
	if seg.parent != nil {
		seg.Lock()
		logger.Debugf("Ending subsegment named: %s", seg.Name)
		seg.EndTime = float64(time.Now().UnixNano()) / float64(time.Second)
		seg.InProgress = false
		seg.Emitted = true
		seg.Unlock()
		if seg.parent.RemoveSubsegment(seg) {
			logger.Debugf("Removing subsegment named: %s", seg.Name)
		}
	}

	seg.Lock()
	defer seg.Unlock()
	
	if err != nil {
		seg.addError(err)
	}

	// If segment is dummy we return
	if seg.Dummy {
		return
	}

	seg.beforeEmitSubsegment(seg.parent)
	seg.emit()
}

[willarmiros]

Thanks for reporting this, and contributing a fix in #393. It will be released in our next release.