Add possibility to GraphPPO for multi inputs with Dict spaces (including graphs) #446
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- we reuse our stable_baselines3 wrapper
- the policy is extracting features from the graph with a GNN
- the GNN is using pytorch-geometric
- We subclass
- ActorCriticPolicy:
- feature extractor = gnn
- custom conversion of observation to torch to convert into
torch_geometric.data.Data
- PPO to handle properly
- observation conversion
- rollout buffer
- Current limitations:
- we extract a fixed number of features (independent of edge/node
numbers) for now as we end with a feature reduction layer connected
to a classic mlp (not knowning anything about the current graph structure)
- User input: the user can define (and default choices are made else)
- the gnn (default to a 2 layers GCN), taking as inputs w.r.t torch_geometric conventions:
- x: nodes features
- edge_index: edge indices or sparse transposed adjency matrix
- edge_attr (optional): edges features
- edge_weight (optional): edge weights (taken from first dimension
of edge_attr)
- the feature reduction layer from the gnn output to the fixed number of features
(default to global_max_pool + linear layer + relu)
- We update the signature of sample_action(), get_next_action_distribution() and get_next_action() with **kwargs to allow some solver to use extra argument like action masks. - We add `kwargs_sample_action_fn` arg to derive generic kwargs to pass to `sample_action()` during rollout - We add `use_action_masking` flag as a shortcut to define appropriate `kwargs_sample_action_fn` using - domain.action_masks() is existing - `domain.is_applicable_action()` provided that `domain.get_action_space()` is a `skdecide.core.EnumerableSpace`
nhuet
force-pushed
the
multiinput-graph-sb3
branch
from
225a677 to
2c3d1ff
Compare
nhuet
force-pushed
the
multiinput-graph-sb3
branch
from
c46ea2a to
ec4b30d
Compare
|
now included in #441. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.