How to launch tasks on a custom thread pool?

[topkanoguzhan]

Hi everyone,

Is there an example or a guide on how to schedule a task on a custom thread pool. I followed the resource partitioner guide in the documentation to add a bunch of threads from a NUMA node to a thread pool called "my_thread_pool" exactly like in the documentation, and testing this with hpx:print-bind shows that it works as intendend. From the main function executed on the default pool I would now like to launch some of the tasks on the custom thread pool. I was not able to find something in the documentation, but it states that this should be possible.

I am trying to know specifically how this could be done with dataflows like these:
V[0] = dataflow( hpx::launch::async, operation, arg1, arg2, ..., argn);

Thank you very much in advance for your help!

[hkaiser]

Would you mind sharing a small example for what you're doing? Just to make sure I fully understand what you're doing and are trying to achieve...

[topkanoguzhan]

Thank you for your help. I mistakenly wrote an answer to my post instead of your reply. I hope you will take a look. Thank you very much

[topkanoguzhan]

Apologies for the delayed response, my example is based on one of your stencil examples. The link for it is provided below. I am trying to ensure that each NUMA node works on the same block of the overall grid over all iterations. In my example I used sched_getcpu() to see the history of pus that worked on each partition over time and the results are mixed. When initializing the partitions with a parallel_for on 2 numa nodes, each NUMA node has approximately one half written into their memory, but after that the computational tasks are not split among the 2 NUMA nodes cleanly. I know that in your and my example there are essentially new partitions created and written into memory so that if a partition of timestep 0 was on NUMA node 0, the partition for its next timestep may be computed on NUMA node 1 and thus its data would lie on NUMA node 1s memory. This way remote accesses might still be mitigated if there is a streak of timesteps where one partition is computed on the same NUMA node, even if intermittently there are switches.

However, if I wanted to make a clean division of responsibilities among the NUMA nodes that I have, how could I achieve this?
In terms of your example below I want to schedule the tasks for the partitions from 0-np/2 on NUMA node 0 and np/2-np on NUMA node 1, while still enforcing the dependencies. Are there any features in HPX that are intended for such a scenario in a use case similar to your example. I have already a version with parallel algorithms working on HPX's numa aware vectors, but I also wanted to create an example with an explicitly created tree of tasks with constraint based synchronization.

1d_stencil_4.cpp

 hpx::future<space> do_work(
        std::size_t np, std::size_t nx, std::size_t nt, std::uint64_t nd)
    {
        using hpx::dataflow;
        using hpx::unwrapping;

        // U[t][i] is the state of position i at time t.
        std::vector<space> U(2);
        for (space& s : U)
            s.resize(np);

        // Initial conditions: f(0, i) = i
        std::size_t b = 0;
        auto range = boost::irange(b, np);
        using hpx::execution::par;
        hpx::ranges::for_each(par, range, [&U, nx](std::size_t i) {
            U[0][i] = hpx::make_ready_future(partition_data(nx, double(i)));
        });

        // limit depth of dependency tree
        hpx::lcos::local::sliding_semaphore sem(nd);

        auto Op = unwrapping(&stepper::heat_part);

        // Actual time step loop
        for (std::size_t t = 0; t != nt; ++t)
        {
            space const& current = U[t % 2];
            space& next = U[(t + 1) % 2];

            for (std::size_t i = 0; i != np; ++i)
            {
                next[i] =
                    dataflow(hpx::launch::async, Op, current[idx(i, -1, np)],
                        current[i], current[idx(i, +1, np)]);
            }

            // every nd time steps, attach additional continuation which will
            // trigger the semaphore once computation has reached this point
            if ((t % nd) == 0)
            {
                next[0].then([&sem, t](partition&&) {
                    // inform semaphore about new lower limit
                    sem.signal(t);
                });
            }

            // suspend if the tree has become too deep, the continuation above
            // will resume this thread once the computation has caught up
            sem.wait(t);
        }

        // Return the solution at time-step 'nt'.
        return hpx::when_all(U[nt % 2]);
    }
};

[gonidelis]

@topkanoguzhan I am on it. Will come back to you later in the day. Thanks!

[topkanoguzhan]

Thank you so much, Giannis:)

[gonidelis]

@topkanoguzhan apologies for the late response. I am on it.

[topkanoguzhan]

@gonidelis are you still on it? If not, could you maybe just tell me what the challenges and difficulties are for solving my issue? Either way I am really grateful you made an attempt on it:)

[gonidelis]

@topkanoguzhan still on it. I am trying to reproduce your example at this point. Again sorry for stalling but I was occupied with working on our new release.