Skip to content

Commit

Permalink
[backport] Hotplug thread infrastructure
Browse files Browse the repository at this point in the history 
smp: Add generic smpboot facility

Start a new file, which will hold SMP and CPU hotplug related generic
infrastructure.

Signed-off-by: Thomas Gleixner <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Rusty Russell <[email protected]>
Cc: Paul E. McKenney <[email protected]>
Cc: Srivatsa S. Bhat <[email protected]>
Cc: Matt Turner <[email protected]>
Cc: Russell King <[email protected]>
Cc: Mike Frysinger <[email protected]>
Cc: Jesper Nilsson <[email protected]>
Cc: Richard Kuo <[email protected]>
Cc: Tony Luck <[email protected]>
Cc: Hirokazu Takata <[email protected]>
Cc: Ralf Baechle <[email protected]>
Cc: David Howells <[email protected]>
Cc: James E.J. Bottomley <[email protected]>
Cc: Benjamin Herrenschmidt <[email protected]>
Cc: Martin Schwidefsky <[email protected]>
Cc: Paul Mundt <[email protected]>
Cc: David S. Miller <[email protected]>
Cc: Chris Metcalf <[email protected]>
Cc: Richard Weinberger <[email protected]>
Cc: [email protected]
Link: http://lkml.kernel.org/r/[email protected]

Change-Id: Ia1ad435435aa12c47ac0d381ae031ebf6edcff1f

smp: Provide generic idle thread allocation

All SMP architectures have magic to fork the idle task and to store it
for reusage when cpu hotplug is enabled. Provide a generic
infrastructure for it.

Create/reinit the idle thread for the cpu which is brought up in the
generic code and hand the thread pointer to the architecture code via
__cpu_up().

Note, that fork_idle() is called via a workqueue, because this
guarantees that the idle thread does not get a reference to a user
space VM. This can happen when the boot process did not bring up all
possible cpus and a later cpu_up() is initiated via the sysfs
interface. In that case fork_idle() would be called in the context of
the user space task and take a reference on the user space VM.

Signed-off-by: Thomas Gleixner <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Rusty Russell <[email protected]>
Cc: Paul E. McKenney <[email protected]>
Cc: Srivatsa S. Bhat <[email protected]>
Cc: Matt Turner <[email protected]>
Cc: Russell King <[email protected]>
Cc: Mike Frysinger <[email protected]>
Cc: Jesper Nilsson <[email protected]>
Cc: Richard Kuo <[email protected]>
Cc: Tony Luck <[email protected]>
Cc: Hirokazu Takata <[email protected]>
Cc: Ralf Baechle <[email protected]>
Cc: David Howells <[email protected]>
Cc: James E.J. Bottomley <[email protected]>
Cc: Benjamin Herrenschmidt <[email protected]>
Cc: Martin Schwidefsky <[email protected]>
Cc: Paul Mundt <[email protected]>
Cc: David S. Miller <[email protected]>
Cc: Chris Metcalf <[email protected]>
Cc: Richard Weinberger <[email protected]>
Cc: [email protected]
Acked-by: Venkatesh Pallipadi <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]

Change-Id: Ie2d32789f3a69ee15f38ba704aaa84d6be85bcd4

smp, idle: Allocate idle thread for each possible cpu during boot

percpu areas are already allocated during boot for each possible cpu.
percpu idle threads can be considered as an extension of the percpu areas,
and allocate them for each possible cpu during boot.

This will eliminate the need for workqueue based idle thread allocation.
In future we can move the idle thread area into the percpu area too.

[ tglx: Moved the loop into smpboot.c and added an error check when
  the init code failed to allocate an idle thread for a cpu which
  should be onlined ]

Signed-off-by: Suresh Siddha <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Rusty Russell <[email protected]>
Cc: Paul E. McKenney <[email protected]>
Cc: Srivatsa S. Bhat <[email protected]>
Cc: Tejun Heo <[email protected]>
Cc: David Rientjes <[email protected]>
Cc: [email protected]
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Thomas Gleixner <[email protected]>

Change-Id: I36828165fc08b7c0a8a0fe6a2aa24d358e623dd2

smpboot, idle: Optimize calls to smp_processor_id() in idle_threads_init()

While trying to initialize idle threads for all cpus, idle_threads_init()
calls smp_processor_id() in a loop, which is unnecessary. The intent
is to initialize idle threads for all non-boot cpus. So just use a variable
to note the boot cpu and use it in the loop.

Signed-off-by: Srivatsa S. Bhat <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Thomas Gleixner <[email protected]>

Change-Id: Ib65df4c31e93e1622c26f2c2a4946ffd28c1839d

smpboot, idle: Fix comment mismatch over idle_threads_init()

The comment over idle_threads_init() really talks about the functionality
of idle_init(). Move that comment to idle_init(), and add a suitable
comment over idle_threads_init().

Signed-off-by: Srivatsa S. Bhat <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Thomas Gleixner <[email protected]>

Change-Id: Ib0cd6d6e19e0c64868a42a77101b080a5f3b04f8

kthread_worker: reorganize to prepare for flush_kthread_work() reimplementation

Make the following two non-functional changes.

* Separate out insert_kthread_work() from queue_kthread_work().

* Relocate struct kthread_flush_work and kthread_flush_work_fn()
  definitions above flush_kthread_work().

v2: Added lockdep_assert_held() in insert_kthread_work() as suggested
    by Andy Walls.

Signed-off-by: Tejun Heo <[email protected]>
Acked-by: Andy Walls <[email protected]>

Change-Id: Ie1eef2c000c328ec16f32db011377415237da93d

kthread_worker: reimplement flush_kthread_work() to allow freeing the work item being executed

kthread_worker provides minimalistic workqueue-like interface for
users which need a dedicated worker thread (e.g. for realtime
priority).  It has basic queue, flush_work, flush_worker operations
which mostly match the workqueue counterparts; however, due to the way
flush_work() is implemented, it has a noticeable difference of not
allowing work items to be freed while being executed.

While the current users of kthread_worker are okay with the current
behavior, the restriction does impede some valid use cases.  Also,
removing this difference isn't difficult and actually makes the code
easier to understand.

This patch reimplements flush_kthread_work() such that it uses a
flush_work item instead of queue/done sequence numbers.

Signed-off-by: Tejun Heo <[email protected]>

Change-Id: I06e2ab5ef8ea3caa8e40257da0a636ab9eb5ae55

kthread: Implement park/unpark facility

To avoid the full teardown/setup of per cpu kthreads in the case of
cpu hot(un)plug, provide a facility which allows to put the kthread
into a park position and unpark it when the cpu comes online again.

Signed-off-by: Thomas Gleixner <[email protected]>
Reviewed-by: Namhyung Kim <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Reviewed-by: Srivatsa S. Bhat <[email protected]>
Cc: Rusty Russell <[email protected]>
Reviewed-by: Paul E. McKenney <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Thomas Gleixner <[email protected]>

Change-Id: I05d28788540b666349bafecf6cb3fdc873b6cdde

smpboot: Provide infrastructure for percpu hotplug threads

Provide a generic interface for setting up and tearing down percpu
threads.

On registration the threads for already online cpus are created and
started. On deregistration (modules) the threads are stoppped.

During hotplug operations the threads are created, started, parked and
unparked. The datastructure for registration provides a pointer to
percpu storage space and optional setup, cleanup, park, unpark
functions. These functions are called when the thread state changes.

Each implementation has to provide a function which is queried and
returns whether the thread should run and the thread function itself.

The core code handles all state transitions and avoids duplicated code
in the call sites.

[ paulmck: Preemption leak fix ]

Signed-off-by: Thomas Gleixner <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Reviewed-by: Srivatsa S. Bhat <[email protected]>
Cc: Rusty Russell <[email protected]>
Reviewed-by: Paul E. McKenney <[email protected]>
Cc: Namhyung Kim <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Thomas Gleixner <[email protected]>

Change-Id: Ib2ac667cd13cf26a042d65c1b3f20fe7e4b02423

hotplug: Fix UP bug in smpboot hotplug code

Because kernel subsystems need their per-CPU kthreads on UP systems as
well as on SMP systems, the smpboot hotplug kthread functions must be
provided in UP builds as well as in SMP builds.  This commit therefore
adds smpboot.c to UP builds and excludes irrelevant code via #ifdef.

Signed-off-by: Paul E. McKenney <[email protected]>
Signed-off-by: Paul E. McKenney <[email protected]>
Signed-off-by: Thomas Gleixner <[email protected]>

Change-Id: I7b570d6c241c513227c3fdc1d843bf369bed036c

smpboot: Allow selfparking per cpu threads

The stop machine threads are still killed when a cpu goes offline. The
reason is that the thread is used to bring the cpu down, so it can't
be parked along with the other per cpu threads.

Allow a per cpu thread to be excluded from automatic parking, so it
can park itself once it's done

Add a create callback function as well.

Signed-off-by: Thomas Gleixner <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Rusty Russell <[email protected]>
Cc: Paul McKenney <[email protected]>
Cc: Srivatsa S. Bhat <[email protected]>
Cc: Arjan van de Veen <[email protected]>
Cc: Paul Turner <[email protected]>
Cc: Richard Weinberger <[email protected]>
Cc: Magnus Damm <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Thomas Gleixner <[email protected]>

Change-Id: I864f39336a2cb648c518526459929c081f831216

kthread: Prevent unpark race which puts threads on the wrong cpu

The smpboot threads rely on the park/unpark mechanism which binds per
cpu threads on a particular core. Though the functionality is racy:

CPU0	       	 	CPU1  	     	    CPU2
unpark(T)				    wake_up_process(T)
  clear(SHOULD_PARK)	T runs
			leave parkme() due to !SHOULD_PARK
  bind_to(CPU2)		BUG_ON(wrong CPU)

We cannot let the tasks move themself to the target CPU as one of
those tasks is actually the migration thread itself, which requires
that it starts running on the target cpu right away.

The solution to this problem is to prevent wakeups in park mode which
are not from unpark(). That way we can guarantee that the association
of the task to the target cpu is working correctly.

Add a new task state (TASK_PARKED) which prevents other wakeups and
use this state explicitly for the unpark wakeup.

Peter noticed: Also, since the task state is visible to userspace and
all the parked tasks are still in the PID space, its a good hint in ps
and friends that these tasks aren't really there for the moment.

The migration thread has another related issue.

CPU0	      	     	 CPU1
Bring up CPU2
create_thread(T)
park(T)
 wait_for_completion()
			 parkme()
			 complete()
sched_set_stop_task()
			 schedule(TASK_PARKED)

The sched_set_stop_task() call is issued while the task is on the
runqueue of CPU1 and that confuses the hell out of the stop_task class
on that cpu. So we need the same synchronizaion before
sched_set_stop_task().

Reported-by: Dave Jones <[email protected]>
Reported-and-tested-by: Dave Hansen <[email protected]>
Reported-and-tested-by: Borislav Petkov <[email protected]>
Acked-by: Peter Ziljstra <[email protected]>
Cc: Srivatsa S. Bhat <[email protected]>
Cc: [email protected]
Cc: Ingo Molnar <[email protected]>
Cc: [email protected]
Link: http://lkml.kernel.org/r/alpine.LFD.2.02.1304091635430.21884@ionos
Signed-off-by: Thomas Gleixner <[email protected]>

Change-Id: If1e9993951c4ad1f6f35ad0698f6ccd05a67e81f

stop_machine: Store task reference in a separate per cpu variable

To allow the stopper thread being managed by the smpboot thread
infrastructure separate out the task storage from the stopper data
structure.

Signed-off-by: Thomas Gleixner <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Rusty Russell <[email protected]>
Cc: Paul McKenney <[email protected]>
Cc: Srivatsa S. Bhat <[email protected]>
Cc: Arjan van de Veen <[email protected]>
Cc: Paul Turner <[email protected]>
Cc: Richard Weinberger <[email protected]>
Cc: Magnus Damm <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Thomas Gleixner <[email protected]>

Change-Id: Ibfe2389e42fcf2e236940bbc223a36da571ed6e9

stop_machine: Use smpboot threads

Use the smpboot thread infrastructure. Mark the stopper thread
selfparking and park it after it has finished the take_cpu_down()
work.

Signed-off-by: Thomas Gleixner <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Rusty Russell <[email protected]>
Cc: Paul McKenney <[email protected]>
Cc: Srivatsa S. Bhat <[email protected]>
Cc: Arjan van de Veen <[email protected]>
Cc: Paul Turner <[email protected]>
Cc: Richard Weinberger <[email protected]>
Cc: Magnus Damm <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Thomas Gleixner <[email protected]>

Change-Id: I30771810f2cbb2a64ca090864156edc79d338dfd

stop_machine: Mark per cpu stopper enabled early

commit 14e568e78 (stop_machine: Use smpboot threads) introduced the
following regression:

Before this commit the stopper enabled bit was set in the online
notifier.

CPU0				CPU1
cpu_up
				cpu online
hotplug_notifier(ONLINE)
  stopper(CPU1)->enabled = true;
...
stop_machine()

The conversion to smpboot threads moved the enablement to the wakeup
path of the parked thread. The majority of users seem to have the
following working order:

CPU0				CPU1
cpu_up
				cpu online
unpark_threads()
  wakeup(stopper[CPU1])
....
				stopper thread runs
				  stopper(CPU1)->enabled = true;
stop_machine()

But Konrad and Sander have observed:

CPU0				CPU1
cpu_up
				cpu online
unpark_threads()
  wakeup(stopper[CPU1])
....
stop_machine()
				stopper thread runs
				  stopper(CPU1)->enabled = true;

Now the stop machinery kicks CPU0 into the stop loop, where it gets
stuck forever because the queue code saw stopper(CPU1)->enabled ==
false, so CPU0 waits for CPU1 to enter stomp_machine, but the CPU1
stopper work got discarded due to enabled == false.

Add a pre_unpark function to the smpboot thread descriptor and call it
before waking the thread.

This fixes the problem at hand, but the stop_machine code should be
more robust. The stopper->enabled flag smells fishy at best.

Thanks to Konrad for going through a loop of debug patches and
providing the information to decode this issue.

Reported-and-tested-by: Konrad Rzeszutek Wilk <[email protected]>
Reported-and-tested-by: Sander Eikelenboom <[email protected]>
Cc: Srivatsa S. Bhat <[email protected]>
Cc: Rusty Russell <[email protected]>
Link: http://lkml.kernel.org/r/alpine.LFD.2.02.1302261843240.22263@ionos
Signed-off-by: Thomas Gleixner <[email protected]>

Change-Id: Iaff8824879eb21552fc9e46e259b604dfce113bc
  • Loading branch information
@KAGA-KOKO @bb-qq
KAGA-KOKO authored and bb-qq committed May 16, 2017
1 parent 5afe583 commit 2b36104
Show file tree
Hide file tree
Showing 15 changed files with 723 additions and 167 deletions.
3 changes: 3 additions & 0 deletions arch/Kconfig
View file
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,9 @@ config HAVE_DMA_CONTIGUOUS
config USE_GENERIC_SMP_HELPERS
bool

config GENERIC_SMP_IDLE_THREAD
bool

config HAVE_REGS_AND_STACK_ACCESS_API
bool
help
Expand Down
1 change: 1 addition & 0 deletions arch/arm/Kconfig
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ config ARM
select CPU_PM if (SUSPEND || CPU_IDLE)
select GENERIC_PCI_IOMAP
select HAVE_BPF_JIT if NET
select GENERIC_SMP_IDLE_THREAD
help
The ARM series is a line of low-power-consumption RISC chip designs
licensed by ARM Ltd and targeted at embedded applications and
Expand Down
1 change: 1 addition & 0 deletions fs/proc/array.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,7 @@ static const char * const task_state_array[] = {
"x (dead)", /* 64 */
"K (wakekill)", /* 128 */
"W (waking)", /* 256 */
"P (parked)", /* 512 */
};

static inline const char *get_task_state(struct task_struct *tsk)
Expand Down
19 changes: 12 additions & 7 deletions include/linux/kthread.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,11 @@ struct task_struct *kthread_create_on_node(int (*threadfn)(void *data),
kthread_create_on_node(threadfn, data, -1, namefmt, ##arg)


struct task_struct *kthread_create_on_cpu(int (*threadfn)(void *data),
void *data,
unsigned int cpu,
const char *namefmt);

/**
* kthread_run - create and wake a thread.
* @threadfn: the function to run until signal_pending(current).
Expand All @@ -34,9 +39,13 @@ struct task_struct *kthread_create_on_node(int (*threadfn)(void *data),

void kthread_bind(struct task_struct *k, unsigned int cpu);
int kthread_stop(struct task_struct *k);
int kthread_should_stop(void);
bool kthread_should_stop(void);
bool kthread_should_park(void);
bool kthread_freezable_should_stop(bool *was_frozen);
void *kthread_data(struct task_struct *k);
int kthread_park(struct task_struct *k);
void kthread_unpark(struct task_struct *k);
void kthread_parkme(void);

int kthreadd(void *unused);
extern struct task_struct *kthreadd_task;
Expand All @@ -49,8 +58,6 @@ extern int tsk_fork_get_node(struct task_struct *tsk);
* can be queued and flushed using queue/flush_kthread_work()
* respectively. Queued kthread_works are processed by a kthread
* running kthread_worker_fn().
*
* A kthread_work can't be freed while it is executing.
*/
struct kthread_work;
typedef void (*kthread_work_func_t)(struct kthread_work *work);
Expand All @@ -59,15 +66,14 @@ struct kthread_worker {
spinlock_t lock;
struct list_head work_list;
struct task_struct *task;
struct kthread_work *current_work;
};

struct kthread_work {
struct list_head node;
kthread_work_func_t func;
wait_queue_head_t done;
atomic_t flushing;
int queue_seq;
int done_seq;
struct kthread_worker *worker;
};

#define KTHREAD_WORKER_INIT(worker) { \
Expand All @@ -79,7 +85,6 @@ struct kthread_work {
.node = LIST_HEAD_INIT((work).node), \
.func = (fn), \
.done = __WAIT_QUEUE_HEAD_INITIALIZER((work).done), \
.flushing = ATOMIC_INIT(0), \
}

#define DEFINE_KTHREAD_WORKER(worker) \
Expand Down
5 changes: 3 additions & 2 deletions include/linux/sched.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -193,9 +193,10 @@ print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)
#define TASK_DEAD 64
#define TASK_WAKEKILL 128
#define TASK_WAKING 256
#define TASK_STATE_MAX 512
#define TASK_PARKED 512
#define TASK_STATE_MAX 1024

#define TASK_STATE_TO_CHAR_STR "RSDTtZXxKW"
#define TASK_STATE_TO_CHAR_STR "RSDTtZXxKWP"

extern char ___assert_task_state[1 - 2*!!(
sizeof(TASK_STATE_TO_CHAR_STR)-1 != ilog2(TASK_STATE_MAX)+1)];
Expand Down
52 changes: 52 additions & 0 deletions include/linux/smpboot.h
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
#ifndef _LINUX_SMPBOOT_H
#define _LINUX_SMPBOOT_H

#include <linux/types.h>

struct task_struct;
/* Cookie handed to the thread_fn*/
struct smpboot_thread_data;

/**
* struct smp_hotplug_thread - CPU hotplug related thread descriptor
* @store: Pointer to per cpu storage for the task pointers
* @list: List head for core management
* @thread_should_run: Check whether the thread should run or not. Called with
* preemption disabled.
* @thread_fn: The associated thread function
* @create: Optional setup function, called when the thread gets
* created (Not called from the thread context)
* @setup: Optional setup function, called when the thread gets
* operational the first time
* @cleanup: Optional cleanup function, called when the thread
* should stop (module exit)
* @park: Optional park function, called when the thread is
* parked (cpu offline)
* @unpark: Optional unpark function, called when the thread is
* unparked (cpu online)
* @pre_unpark: Optional unpark function, called before the thread is
* unparked (cpu online). This is not guaranteed to be
* called on the target cpu of the thread. Careful!
* @selfparking: Thread is not parked by the park function.
* @thread_comm: The base name of the thread
*/
struct smp_hotplug_thread {
struct task_struct __percpu **store;
struct list_head list;
int (*thread_should_run)(unsigned int cpu);
void (*thread_fn)(unsigned int cpu);
void (*create)(unsigned int cpu);
void (*setup)(unsigned int cpu);
void (*cleanup)(unsigned int cpu, bool online);
void (*park)(unsigned int cpu);
void (*unpark)(unsigned int cpu);
void (*pre_unpark)(unsigned int cpu);
bool selfparking;
const char *thread_comm;
};

int smpboot_register_percpu_thread(struct smp_hotplug_thread *plug_thread);
void smpboot_unregister_percpu_thread(struct smp_hotplug_thread *plug_thread);
int smpboot_thread_schedule(void);

#endif
2 changes: 1 addition & 1 deletion include/trace/events/sched.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,7 @@ TRACE_EVENT(sched_switch,
__print_flags(__entry->prev_state & (TASK_STATE_MAX-1), "|",
{ 1, "S"} , { 2, "D" }, { 4, "T" }, { 8, "t" },
{ 16, "Z" }, { 32, "X" }, { 64, "x" },
{ 128, "W" }) : "R",
{ 128, "K" }, { 256, "W" }, { 512, "P" }) : "R",
__entry->prev_state & TASK_STATE_MAX ? "+" : "",
__entry->next_comm, __entry->next_pid, __entry->next_prio)
);
Expand Down
2 changes: 1 addition & 1 deletion kernel/Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ obj-y = fork.o exec_domain.o panic.o printk.o \
kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o mutex.o \
hrtimer.o rwsem.o nsproxy.o srcu.o semaphore.o \
notifier.o ksysfs.o cred.o \
async.o range.o groups.o
async.o range.o groups.o smpboot.o

ifdef CONFIG_FUNCTION_TRACER
# Do not trace debug files and internal ftrace files
Expand Down
23 changes: 22 additions & 1 deletion kernel/cpu.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@

#include <trace/events/sched.h>

#include "smpboot.h"

#ifdef CONFIG_SMP
/* Serializes the updates to cpu_online_mask, cpu_present_mask */
static DEFINE_MUTEX(cpu_add_remove_lock);
Expand Down Expand Up @@ -210,6 +212,8 @@ static int __ref take_cpu_down(void *_param)
return err;

cpu_notify(CPU_DYING | param->mod, param->hcpu);
/* Park the stopper thread */
kthread_park(current);
return 0;
}

Expand Down Expand Up @@ -240,12 +244,13 @@ static int __ref _cpu_down(unsigned int cpu, int tasks_frozen)
__func__, cpu);
goto out_release;
}
smpboot_park_threads(cpu);

err = __stop_machine(take_cpu_down, &tcd_param, cpumask_of(cpu));
if (err) {
/* CPU didn't die: tell everyone. Can't complain. */
smpboot_unpark_threads(cpu);
cpu_notify_nofail(CPU_DOWN_FAILED | mod, hcpu);

goto out_release;
}
BUG_ON(cpu_online(cpu));
Expand Down Expand Up @@ -302,11 +307,23 @@ static int __cpuinit _cpu_up(unsigned int cpu, int tasks_frozen)
int ret, nr_calls = 0;
void *hcpu = (void *)(long)cpu;
unsigned long mod = tasks_frozen ? CPU_TASKS_FROZEN : 0;
struct task_struct *idle;

if (cpu_online(cpu) || !cpu_present(cpu))
return -EINVAL;

cpu_hotplug_begin();

idle = idle_thread_get(cpu);
if (IS_ERR(idle)) {
ret = PTR_ERR(idle);
goto out;
}

ret = smpboot_create_threads(cpu);
if (ret)
goto out;

ret = __cpu_notify(CPU_UP_PREPARE | mod, hcpu, -1, &nr_calls);
if (ret) {
nr_calls--;
Expand All @@ -321,12 +338,16 @@ static int __cpuinit _cpu_up(unsigned int cpu, int tasks_frozen)
goto out_notify;
BUG_ON(!cpu_online(cpu));

/* Wake the per cpu threads */
smpboot_unpark_threads(cpu);

/* Now call notifier in preparation. */
cpu_notify(CPU_ONLINE | mod, hcpu);

out_notify:
if (ret != 0)
__cpu_notify(CPU_UP_CANCELED | mod, hcpu, nr_calls, NULL);
out:
cpu_hotplug_done();
trace_sched_cpu_hotplug(cpu, ret, 1);

Expand Down
Loading

0 comments on commit 2b36104

Please sign in to comment.