Enable Pipeline Parallelism on Jax models

[Chenyaaang]

Description

The implementation for Pipeline Parallelism are splitted into the following small PRs.

• Util functions for PP on Jax (Create Utils functions for PP #1042)
• worker changes (Enable Pipeline Parallelism on jax worker #1043)
• runner changes (Enable Pipeline Parallelism on Jax runner #1053)
• platform changes (Enable Pipeline Parallelism to use mp as distributed backend on Jax TPU platform #1054)
• torchax changes (Enable Pipeline Parallelism on torchax path #1055)
• Jax changes (The current PR)
• Single host changes (this will be in vLLM)
• Multi host changes (Ray)

This PR is to enable PP on Jax models.

• model_loader.py: modify get_flax_model to accept model forward args.
• pp_utils.py: util function for jax models.
• llama3.py: an example of Jax models under tpu_inference/models/jax, assign embedding layer to the first rank, lm head to the last rank, split hidden layers equally on each rank
• weight_utils.py: only load the weights for corresponding layers on each worker.

Tests

E2E test has verified the whole PP implementation works properly.

Checklist

Before submitting this PR, please make sure:

• I have performed a self-review of my code.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have made or will make corresponding changes to any relevant documentation.

[github-actions]

Description

Start with a short description of what the PR does and how this is a change from
the past.

The rest of the description includes relevant details and context, examples:

• why is this change being made,
• the problem being solved and any relevant context,
• why this is a good solution,
• some information about the specific implementation,
• shortcomings of the solution and possible future improvements.

If the change fixes a bug or a Github issue, please include a link, e.g.,:
FIXES: b/123456
FIXES: #123456

Tests

Please describe how you tested this change, and include any instructions and/or
commands to reproduce.

Checklist

Before submitting this PR, please make sure:

• I have performed a self-review of my code.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have made or will make corresponding changes to any relevant documentation.