Roberta ONNX as classifier

[lordofthejars]

I am trying to use a Roberta model as a classifier. The model is https://huggingface.co/guishe/nuner-v1_orgs (I use as tokenizer), and the same version in ONNX (I use for classification) is https://huggingface.co/protectai/guishe-nuner-v1_orgs-onnx

Then the code I run is:

HuggingFaceTokenizer huggingFaceTokenizer = HuggingFaceTokenizer.newInstance("guishe/nuner-v1_orgs");

        String modelLoction = Thread.currentThread()
                .getContextClassLoader()
                .getResource("model.onnx").toExternalForm();

        Criteria<String, String> criteria = Criteria.builder()
                .setTypes(String.class, String.class)
                .optModelUrls(modelLoction)
                .optTranslator(new NoBatchifyTranslator<String, String>() {
                    @Override
                    public String processOutput(TranslatorContext translatorContext, NDList ndList) throws Exception {
                        return ndList.getAsString();
                    }

                    @Override
                    public NDList processInput(TranslatorContext ctx, String s) throws Exception {
                        Encoding input =  huggingFaceTokenizer.encode(s);
                        ctx.setAttachment("encoding", input);
                        return input.toNDList(ctx.getNDManager(), false);
                    }
                })
                .optEngine("OnnxRuntime") // use OnnxRuntime engine by default
                .build();

        try (ZooModel<String, String> model = criteria.loadModel()) {
            System.out.println(model.newPredictor().predict("The CNN tv"));
        }

So when I run it I got Unexpected input data type. Actual: (tensor(int32)) , expected: (tensor(int64))

Encoding object returns longs, so I don't know where they are cast to 32 bits.

Any help is appreciated

[frankfliu]

@lordofthejars

There is a bug introduced by this PR: #3468

For the time being, you need to use DJL 0.29.0 to workaround your issue. Will create a PR to fix this bug.

[lordofthejars]

Thanks it worked but now there is another problem : ai.djl.translate.TranslateException: ai.djl.engine.EngineException: ai.onnxruntime.OrtException: Error code - ORT_INVALID_ARGUMENT - message: Invalid rank for input: input_ids Got: 1 Expected: 2

[frankfliu]

@lordofthejars

I tried 0.29.0, it works for me:

1. I exported the model using djl-convert:

djl-convert -f OnnxRuntime -m "guishe/nuner-v1_orgs" 

2. And I use the following code in java:

        String text = "Foreign governments may be spying on your smartphone notifications, senator says. Washington (CNN) — Foreign governments have reportedly attempted to spy on iPhone and Android users through the mobile app notifications they receive on their smartphones - and the US government has forced Apple and Google to keep quiet about it, according to a top US senator. Through legal demands sent to the tech giants, governments have allegedly tried to force Apple and Google to turn over sensitive information that could include the contents of a notification - such as previews of a text message displayed on a lock screen, or an update about app activity, Oregon Democratic Sen. Ron Wyden said in a new report. Wyden's report reflects the latest example of long-running tensions between tech companies and governments over law enforcement demands, which have stretched on for more than a decade. Governments around the world have particularly battled with tech companies over encryption, which provides critical protections to users and businesses while in some cases preventing law enforcement from pursuing investigations into messages sent over the internet.";
        Path path = Paths.get("models/nuner-v1_orgs");
        Criteria<String, NamedEntity[]> criteria =
                Criteria.builder()
                        .setTypes(String.class, NamedEntity[].class)
                        .optModelPath(path)
                        .optEngine("OnnxRuntime")
                        .optTranslatorFactory(new TokenClassificationTranslatorFactory())
                        .optProgress(new ProgressBar())
                        .build();

        try (ZooModel<String, NamedEntity[]> model = criteria.loadModel();
                Predictor<String, NamedEntity[]> predictor = model.newPredictor()) {
            NamedEntity[] res = predictor.predict(text);
            System.out.println(JsonUtils.GSON_PRETTY.toJson(res));
        }

[lordofthejars]

oh ok let me try thank you very much fir your help

[lordofthejars]

Hi sorry to disturb again, What imports are you using, I am getting:

Caused by: java.lang.UnsupportedOperationException: Not implemented
	at ai.djl.engine.rust.RustLibrary.softmax(RustLibrary.java:326)
	at ai.djl.engine.rust.RsNDArray.softmax(RsNDArray.java:1214)
	at ai.djl.engine.rust.RsNDArray.softmax(RsNDArray.java:35)
	at ai.djl.huggingface.translator.TokenClassificationTranslator.toNamedEntities(TokenClassificationTranslator.java:142)
	at ai.djl.huggingface.translator.TokenClassificationTranslator.batchProcessOutput(TokenClassificationTranslator.java:107)
	at ai.djl.inference.Predictor.batchPredict(Predictor.java:190)

With:

        <dependency>
            <groupId>ai.djl.onnxruntime</groupId>
            <artifactId>onnxruntime-engine</artifactId>
            <version>0.29.0</version>
        </dependency>
        <dependency>
            <groupId>ai.djl.huggingface</groupId>
            <artifactId>tokenizers</artifactId>
            <version>0.29.0</version>
        </dependency>

[frankfliu]

You need include pytorch engine:

    runtimeOnly("ai.djl.pytorch:pytorch-engine")

[frankfliu]

Byte the way, I recommend you use BOM