Allow Concurrent Text Classification

[rootfs]

Is your feature request related to a problem? Please describe.
Currently the text classification is serialized, this blocks the concurrency.

pub extern "C" fn classify_modernbert_text(text: *const c_char) -> ModernBertClassificationResult {
    let default_result = ModernBertClassificationResult {
        class: -1,
        confidence: 0.0,
    };

    let text = unsafe {
        match CStr::from_ptr(text).to_str() {
            Ok(s) => s,
            Err(_) => return default_result,
        }
    };

    let bert_opt = MODERNBERT_CLASSIFIER.lock().unwrap();
    match &*bert_opt {
        Some(classifier) => match classifier.classify_text(text) {
            Ok((class_idx, confidence)) => ModernBertClassificationResult {
                class: class_idx as i32,
                confidence,
            },
            Err(e) => {
                eprintln!("Error classifying text with ModernBERT: {e}");
                default_result
            }
        },
        None => {
            eprintln!("ModernBERT classifier not initialized");
            default_result
        }
    }
}

Describe the solution you'd like
Support concurrent text classification

[rootfs]

@cryo-zd do you like to give this a try? thanks!

[cryo-zd]

Sorry for the delay.

After reading the processing logic of handleBatchClassification and the discussions in #24 / comment , one thing I want to check further is that whether our target is to fix the TODO in the following comment to make candle-binding support batch processing ?

semantic-router/src/semantic-router/pkg/api/server.go

Lines 590 to 602 in 7128765

	// Worker goroutine loops to process tasks from the channel
	for taskIndex := range taskChan {
	// TODO: Refactor candle-binding to support batch mode for better performance
	// This would allow processing multiple texts in a single model inference call
	// instead of individual calls, significantly improving throughput
	result, err := s.classifySingleText(texts[taskIndex], options)
	if err != nil {
	errors[taskIndex] = err
	metrics.RecordBatchClassificationError(processingType, "classification_failed")
	continue
	}
	results[taskIndex] = result
	}

If so, may be the function listed below is something we need to implement further?

#[no_mangle]
pub extern "C" fn classify_modernbert_texts_batch(
    texts: *const *const c_char,
    len: c_int,
) -> *mut ModernBertClassificationResult {

[rootfs]

@cryo-zd these two issues are related. The batch classification function classify_modernbert_texts_batch is missing in rust, so the golang program has to call the binding multiple times to batch the requests.

The other (aka this issue) is about supporting concurrent classification calls, helping both batch and multiple discrete classification calls.

[cryo-zd]

Understood. Honestly speaking, I’m not very familiar with the Rust candle-binding internals currently, after learning the code I think a good first step is to optimize how we handle

let bert_opt = MODERNBERT_CLASSIFIER.lock().unwrap();

Currently the lock is held during the entire inference, which serializes all calls. If we store the classifier inside an Arc, acquire the lock only to fetch the Arc, then release it before running inference. Will this way enable multiple calls to proceed concurrently(&safely) while keeping initialization safe ?

static ref MODERNBERT_CLASSIFIER: Arc<Mutex<Option<Arc<ModernBertClassifier>>>> = Arc::new(Mutex::new(None));

// 
let classifier_arc_opt = MODERNBERT_CLASSIFIER.lock().unwrap();
    match &*classifier_arc_opt {
        Some(classifier_arc) => {
            let classifier = classifier_arc.clone();
            drop(classifier_arc_opt);

            match classifier.classify_text(text) {

[rootfs]

@cryo-zd Currently, the classification calls are serialized because the mutex lock is held during the entire inference process, allowing only one request at a time.

You can experiment with changing Arc<Mutex<Option<Arc<ModernBertClassifier>>>> pattern where the lock is acquired to clone the Arc reference, then released immediately before running inference, enabling multiple concurrent calls.

This approach requires validating that the underlying candle components (ModernBert, Tokenizer, Device) are thread-safe for concurrent read operations, but would significantly improve throughput by eliminating lock contention during inference.

[Xunzhuo]

/area core
/priority P1
/milestone v0.1