Translator with two NDLists for processInput

[h4RkhW8t53e]

Hi,
I am wondering how to write a Translator for something like torch.nn.Bilinear
https://nn.readthedocs.io/en/readthedocs/simple/index.html#nn.Bilinear
where processInput () has to return two NDLists?
I will appreciate any help.
Thank you.

[zachgk]

Bilinear looks like it require two NDArrays, not NDLists. The NDArrays are the N-dimensional arrays that would be of shapes N x inputDimension1 and N x inputDimension1.

Then, the NDList is used to store a tuple of NDArrays or a list of NDArrays. In this case, you need a tuple of two NDArrays so you would just give new NDList(inputDim1Array, inputDim2Array).

[h4RkhW8t53e]

Hi,
Thank you for your quick reply and explanation!
I was using Bilinear as an example while trying to make sense of a different model that looks a bit more complicated.
https://github.com/MarcusOlivecrona/REINVENT/blob/master/model.py

I followed your suggestions for Bilinear for this model (please see below).
First, starting from a trained untraced model, I generated traced model.
Next, I tested the traced model in Python.
And finally I tried to use the traced model in Java. Unfortunately, I cannot get it to work.
If you have time, I will be very grateful for any more help and suggestions.
Many thanks!

Python: Generate traced model

voc = Vocabulary(init_from_file="Voc")
model = RNN(voc).rnn.load_state_dict(torch.load(‘model.ckpt'))
model.eval()
batch_size = 1
start_token = torch.zeros(batch_size).long()
start_token[:] = voc.vocab['GO']
x = start_token # tensor([49])
h = torch.zeros(3, batch_size, 512) # torch.zeros(3, 1, 512)
example = x,h
traced_script_module = torch.jit.trace(model, example)
traced_script_module.save(“model-traced.zip")

Restore traced model

model = torch.jit.load(‘./model-traced.zip')
model.eval()
batch_size = 1
start_token = torch.zeros(batch_size).long()
start_token[:] = voc.vocab['GO']
x = start_token # tensor([49])
h = torch.zeros(3, batch_size, 512) # torch.zeros(3, 1, 512)
x_out, h_out = model(x, h)

Java: Restore traced model

Path modelDir = Paths.get("./");
Model model = Model.newInstance(“model-traced.zip);

NDManager manager = NDManager.newBaseManager();
NDArray x = manager.create(new float[] {49.f});
NDArray h = manager.zeros(new Shape(3,1,512));
List input = new ArrayList ();
input.add(x);
input.add(h);

Translator< List , List> translator = new Translator< List , List>() {
@OverRide
public NDList processInput(TranslatorContext ctx, List input) {
return new NDList(input);
}
@OverRide
public List processOutput(TranslatorContext ctx, NDList output) {
return new ArrayList (output);
}
@OverRide
public Batchifier getBatchifier() {
return Batchifier.STACK;
}
};
var predictor = model.newPredictor(translator);
var output = predictor.predict(input);

[h4RkhW8t53e]

It seems the Lists in the Translator definition have disappeared, they are List type.
I also changed NDArray x = manager.create(new float[] {49.f}); to
NDArray x = manager.create(new long[] {long)49});

[h4RkhW8t53e]

Java: Restore traced model

Path modelDir = Paths.get("./");
Model model = Model.newInstance(“model-traced.zip);

NDManager manager = NDManager.newBaseManager();
NDArray x = manager.create(new float[] {49.f});
NDArray h = manager.zeros(new Shape(3,1,512));
List <NDArray> input = new ArrayList <NDArray> ();    
input.add(x);
input.add(h);
        
Translator< List<NDArray> , List<NDArray>> translator = new Translator< List<NDArray> , List<NDArray>>() {

@Override
public NDList processInput(TranslatorContext ctx, List<NDArray> input) {
   return new NDList(input);
}

@Override
public List<NDArray> processOutput(TranslatorContext ctx, NDList output) {
   return new ArrayList <NDArray> (output);
}
        
@Override
public Batchifier getBatchifier() {
   return Batchifier.STACK;
}
};

var predictor = model.newPredictor(translator);
var output = predictor.predict(input);

[zachgk]

I think you are running into issues with batches. The translator by default does not translate in batches, but single records. Then, you use the batchifier to combine multiple preprocessed inputs into a batch for prediction.

One workaround you can take is to use null for your batchifier. Then, DJL won't try to batch and unbatch your data for you. It is also possible to then use your translator to translate a batch instead of a record like it seems like you are doing.

P.S. One small note. One part of the translator is that you would construct your NDArrays inside of them. You can use translator.getNDManager for your manager and it would be automatically closed after post-processing. If you want to ignore this as well, you could just use an NDList for input directly and the NoopTranslator.

[h4RkhW8t53e]

Thank you for your help and suggestions! The Translator works with null for the batchifier.
I'll have to look at the python model in more detail. It seems that there MultiGRU.forward is passed batches rather than records; I may change it. The important thing is that now I have a better understanding of what options I have.
Again, many thanks!