-
Notifications
You must be signed in to change notification settings - Fork 17
Shared coprocessor framework design
The Shared coprocessor framework will be implemented as a part of the XEN hypervisor. The framework design and development will be based on master branch not earlier than 4.8.0-rc2 targeting upstreaming before XEN 4.9 release.
Following are described design overview addressing requirements:
The coprocessor resources sharing will be done in a way of switching context (per domain instance of firmware running) on the coprocessor issued by the main processor. It is done by a hypervisor running on the main cpu and aware about domains, their capabilities and needs.
The hypervisor is aware about coprocessors to be shared. For each shared coprocessor there are: an assigned timer, scheduler and per-domain vcoproc instances list.
It will be implemented a vcoproc scheduling mechanism independent of XEN VCPU scheduling mechanism. It is caused by the difference in vcoproc and vcpu nature:
- it is not considered pool of equal vcoproc running on a pool of equal coprocessors (SMP like behavior)
- due to fact that scheduler is not running on the coprocessor itself:
- a vcoproc scheduler does not preempt the coprocessor context
- coprocessor could deny context switching at the moment scheduler decide so, scheduling mechanism should adopt that
- there could be number of different coprocessors instances with scheduler and vcoproc queue associated
Due to the fact that some domain assigned a vcoproc could access coproc when it is running another domain context, framework will implement iomem access emulation for domains which are not provided coproc at the moment of access.
Interrupts from coprocessor will be effectively delivered to the domain which vcoproc context is being running at the moment of the interrupt.
While the context switching is coprocessor and SoC specific procedure, the framework would not encapsulate such procedure, but provides interface to register platform implementation of context switching procedure.
It is assumed that the coprocessor has an iommu associated or corresponding system level iommu (i.e. arm smmu) owned by hypervisor. The iommu setup is also platform specific code and part of context switching.
In case of only having coprocessor associated iommu, platform code is also responsible to handle that iommu setup (i.e. in a way of iomem access emulation).
The coprocessor framework will have a list of coprocessors available for sharing during system runtime. Any running domain could be using none, one or several shared coprocessors.
The compilation time configuration will provide separate config for the shared coprocessor framework support, configs enabling scheduling algorithms, configs enabling support of the coprocessor platform support.
The system startup configuration will be described in the system device tree in the xen "choosen" node. The configuration should at least describe which coprocessors within the system will be shared to the domains, a scheduling algorithm assigned to each shared coprocessor, and an identificator of this particular coprocessor instance. Hypervisor will read corresponding nodes and configure the coprocessor framework with platform implementations on startup.
In a domain configuration file it will be specified which particular instance of the coprocessor should be shared to this domain.
The shared coprocessor domain configuration should be read by tools/xen and on domain creation a vcoproc should be created and assigned to vcoproc instances list of the particular coprocessor. Starting from here it is assumed that vcoproc is ready to be scheduled as soon as domain starts using it.
In case of failure of vcoproc creation, the domain creation should be aborted.
There will be a initial domain tool implemented which will provide a CLI for runtime monitoring of shared coprocessors instances available, per-domain/per-coproc instance list of vcoproc and runtime setup of the scheduler per each coprocessor instance.
The coprocessor sharing framework is a platform independent entity which implements generic flows and actions. It does need platform specific hooks which implement exact coprocessor management operations. The coprocessor platform specific implementations will provide:
- domain create/destroy hooks
- context switch hooks (coprocessor start, stop, context load/save)
- iomem handler(s)
- interrupt handler(s)
There are number of possible coprocessor sharing scenarios with different needs of coprocessor resources allocation for different domains. Within current design the coprocessor resource allocation to domains is a matter of scheduling algorithm and its configuration. The framework will use the scheduling algorithm to determine if context switching should be done and what is the next vcoproc to be ran on coprocessor.
The algorithm impelementation will provide:
- initialization/deinitialization hooks
- scheduling operations hooks
(Optional) The coprocessor firmware and drivers should not be changed in order to support the sharing.
This requirement could be compromised for implementations which support coprocessors not designed for externally issued context switching. Such context switch function support would be introduced into the firmware and the driver of the coprocessor.