[dlinfer] only compile the language model in vl models

[dlinfer] only compile the language model in vl models #3199