[dlinfer] only compile language_model in vl models (#2893)

[dlinfer] only compile language_model in vl models (#2893) #6241