-
Notifications
You must be signed in to change notification settings - Fork 17
Shared coprocessor framework design (HLD)
Sharing coprocessor implementation consists of following major logical parts:
- Generic coprocessor support code
- Generic coprocessor scheduling code
- Platform common code
- Userspace tools
Listed above parts implement generic initialization operations and provides generic operation logic and interfaces for the platform specific implementations.
In order to let XEN provide sharing of some particular coprocessor with desired scheduling logic following platform/algorithm specific parts should be implemented by solution provider:
- Coprocessor platform operations
- Specific scheduling algorithm
Following are described design details and interfaces of each major part:
- Generic coprocessor support code
Generic coprocessor support code is placed in files:
xen/arch/arm/coproc/coproc.c
xen/arch/arm/coproc/coproc.h
This code does implements different initialization/deinitialization functions and generic context switching logic.
The code does exposes following interface:
struct coproc_device { ... }
This structure represents a particular real coprocessor in the system. Each coprocessor IP shared within the system using the framework will have such unique structure associated. The structure holds resources (irqs, mmio ranges), operations, links to different lists, this coprocessor is assigned to, etc.
struct vcoproc_ops { ... }
This structure describes coprocessor specific operations the framework will issue during its operation. These operations are platform specific and should be implemented by solution provider.
enum vcoproc_state { ... }
This enum describes states of the virtual coprocessor. These states are changed during the framework operation.
struct vcoproc_instance { ... }
This structure describes a virtual coprocessor which is an instantiation of a some coprocessor for a particular domain. Virtual coprocessor is per domain and per coproc unique. It is assigned to a per domain list as a domain resource. Also it is assigned to a per coprocessor list of the virtual coprocessors scheduled on a physical coprosessor.
void coproc_init(void);
This is a shared coprocessor framework initialization function. This function parse device tree and register in a framework any coprocessor marked as xen,coproc, initializes those coprocessors platform code and required schedulers.
There is no deinitialization function due to the fact that currently the framework is designed to be one time initialized without hot plug/unplug support.
int coproc_register(struct coproc_device *);
This is a function to register some particular coprocessor into the framework. This function should be called by platform specific code for registration. There is no unregister function due to the same reason as for initialization function above.
int vcoproc_domain_init(struct domain *);
This function initializes a per-domain vcoproc infrastructure. Due to the fact that list of coprocessors which need to be shared with domain is already present for dom0 this function also creates vcoprocs and attaches them to it.
void vcoproc_domain_free(struct domain *);
This function deinitializes a per-domain vcoproc infrastructure. All created and attached to this domain vcoprocs should be detached and destroyed here.
bool_t coproc_is_attached_to_domain(struct domain *, const char *path);
This function checks if the vcoproc has been already created from the coprocessor pointed by device tree path and attached to this domain.
int coproc_do_domctl(struct xen_domctl *, struct domain *,
XEN_GUEST_HANDLE_PARAM(xen_domctl_t));
This function is a handler for a XEN_DOMCTL_attach_coproc. It does create vcoproc and attach it to the new DomU guest being created.
s_time_t vcoproc_context_switch(struct vcoproc_instance *,
struct vcoproc_instance *);
This function is a generic context switch handler called by a generic scheduling code in order to switch context running on the coprocessor. Return value is a time to wait if the coprocessor can not unload currently running context. If 0 is returned, the context is switched successfully.
int coproc_release_vcoprocs(struct domain *);
This function does the same work as vcoproc_domain_free function, but it is intended to be called when domain tries to relinquish its resources unlike the former that is called when domain destroys resources.
- Generic coprocessor scheduling code
Generic coprocessor scheduling code is placed in files: xen/arch/arm/coproc/schedule.c xen/arch/arm/coproc/schedule.h
This code does implements generic scheduler initialization and provides scheduling operations both for the generic support and platform specific code.
The code does exposes following interface:
struct vcoproc_task_slice { ... }
This structure presents the decision taken by the scheduling algorithm in terms of the next vcoproc to be scheduled in and time slice to reschedule by timer.
struct vcoproc_schedule_data { ... }
This structure presents the generic scheduling data per each physical coprocessor.
struct vcoproc_scheduler { ... }
This structure presents the particular scheduling algorithm with its operations and data. Generic scheduling code uses those operations in order to initialize/deinitialize algorithm, acquire a scheduling decision from it or provide it an information about a virtual coprocessor(s) state changes.
struct vcoproc_scheduler *vcoproc_scheduler_init(struct coproc_device *);
This function initializes a specified scheduler and associates it to a physical coprocessor.
int vcoproc_scheduler_vcoproc_init(struct vcoproc_scheduler *,
struct vcoproc_instance *);
This function initializes a virtual coprocessor scheduler specific data.
int vcoproc_scheduler_vcoproc_destroy(struct vcoproc_scheduler *,
struct vcoproc_instance *);
This function deinitializes a virtual coprocessor scheduler specific data.
void vcoproc_schedule(struct vcoproc_scheduler *);
This function is a generic scheduling interface which is used in case someone needs to reschedule virtual coprocessors queue on some physical coprocessor.
void vcoproc_sheduler_vcoproc_wake(struct vcoproc_scheduler *,
struct vcoproc_instance *);
This function takes a virtual coprocessor to the ready to be scheduled or running state.
void vcoproc_sheduler_vcoproc_sleep(struct vcoproc_scheduler *,
struct vcoproc_instance *);
This function takes a virtual coprosessor to the state it is not running or ready to be scheduled.
void vcoproc_sheduler_vcoproc_yield(struct vcoproc_scheduler *,
struct vcoproc_instance *);
This function takes a virtual coprocessor from the running state to the ready to be scheduled.
- Platform common code
Platform common code is placed in files:
xen/arch/arm/coproc/plat/common.c
xen/arch/arm/coproc/plat/common.h
The common platform code provides variety of helpers for the platform code.
- Userspace tools
TBD
Shared coprocessor framework is initialized in a start_xen() function right after IOMMU setup.
struct arch_domain {} is extended with a list of virtual coprocessors assigned to the domain.
Domain specific framework configuration is done in a arch_domain_create() function.
It is introduced a XEN_DOMCTL_attach_coproc domain management operation which is used on guest domain creation to share a coprocessor to the domain.
XEN hypervisor shared coprocessor framework support compilation is enabled with CONFIG_HAS_COPROC option.
XEN tools shared coprocessor framework support compilation is enabled with TBD option.
Any device with the property "xen,coproc" set will be taken by the coprocessor sharing framework in case there is a correspondent platform support implementation.
This device will be exposed to any domain which is configured to share it.
Dom0 is configured to share the coprocessor via the Xen command line option dom0_coprocs. Coprocessor could be specified using alias or the full node path, f.e.:
bootargs = "dom0_mem=64M console=dtuart dom0_coprocs=coproc1,/soc/dsp@0x32004000,coproc3"
DomU is configured to share a coprocessor using domain configuration file. It is done in a way close to the ARM passthrough configuration - with a partial device tree and additional configuration file options:
device_tree = "/xen/images/domu.dtb"
coproc=[ "/soc/gpu1@0x35000000", "/soc/dsp2@0x32004000", "/soc/dsp1@0x32000000" ]
irqs=[48, 69, 71, 73, 75]
Each shared coprocessor should be assigned a default scheduler on the system boot using device tree in TBD way.
During system operation each shared coprocessor scheduler should be configurable using TDB tool.