Re-implement SYCL backend parallel_for to improve bandwidth utilization

[mmichel11]

High Level Description
This PR improves hardware bandwidth utilization of oneDPL's SYCL backend parallel for pattern through two ideas:

• Process multiple input iterations per work-item which involves a switch to a nd_range kernel combined with a sub / work group strided indexing approach.
• To generate wide loads for small types, implement a path that vectorizes loads / stores by processing adjacent indices within a single work item. This is combined with the above approach to maximize hardware bandwidth utilization. Vectorization is only applied to fundamental types of size less than 4 (e.g. uint16_t, uint8_t) under a contiguous container.

Implementation Details

• Parallel for bricks have been reworked in the following manner:
  • Each brick contains a pack of ranges within its template parameters to define tuning parameters.
  • The following static integral members are defined (implemented with inheritance):
    • __can_vectorize
    • __preferred_vector_size (1 if __can_vectorize is false)
    • __preferred_iters_per_item
  • The following public member functions are defined
    • __scalar_path (for small input sizes this member function is explicitly called)
    • __vector_path (optional for algorithms that are not vectorizable e.g. binary_search)
    • An overloaded function call operator which dispatches to the appropriate strategy

To implement this approach, the parallel for kernel rewrite from #1870 was adopted with additional changes to handle vectorization paths. Additionally, generic vectorization and strided loop utilities have been defined with the intention for these to be applicable in other portions of the codebase as well. Tests have been expanded to ensure coverage of vectorization paths.

This PR will supersedes #1870. Initially, the plan was to merge this PR into 1870 but after comparing the diff, I believe the most straightforward approach will be to target this directly to main.

[SergeyKopienko]

1. I think that __raw_ptr isn't very good name because begin() usually linked in mind with iterator. But raw usually is some pointer.
2. Do we really need to have here local variable __raw_ptr ? Can we pass __rng.begin() instead of that variable into __vector_walk call?

[mmichel11]

In the contexts in which we vectorize, begin() does return pointers, but I agree the name is confusing.

I have addressed this in a different way due to a performance issue. With uint8_t types, I found the compiler was not properly vectorizing even when calling begin() on the set of ranges within the kernel leading to performance regressions (about 30% slower than where we should be). Calling begin from the host and passing it to the submitter to use in the kernel resolves the issue and gives us good performance.

Since begin() is called on all ranges and passed through the bricks from the submitter, I have switched from the _Rng naming to _Acc here as the underlying type may not be a range. Additional template types are also needed.

Update
Please see the comment: #1976 (comment). All of the begin() calls in this context have been removed.

[SergeyKopienko]

So now we have 3 entity with defined constexpr static bool __can_vectorize :

1. class walk_vector_or_scalar_base
2. class walk_scalar_base
3. struct __brick_shift_left

Does these constexpr-variables really has different semantic?

And if the semantic of these entities are the same, may be make sense to make some re-design to have only one entity __can_vectorize ?

[SergeyKopienko]

In some moments implementation details remind me tag-dispatching which were designed by @rarutyun.
But with some differences: for example the walk2_vectors_or_scalars has not only information about vectorization or parallelization should be executed, but also two variant of functional staff and operator() with compile-time condition check to run one code or another code.

But what if we instead of two different functions

    template <typename _IsFull, typename _ItemId>
    void
    __vector_path(_IsFull __is_full, const _ItemId __idx, _Range __rng) const
    {
        // This is needed to enable vectorization
        auto __raw_ptr = __rng.begin();
        oneapi::dpl::__par_backend_hetero::__vector_walk<__base_t::__preferred_vector_size>{__n}(__is_full, __idx, __f,
                                                                                                 __raw_ptr);
    }

    // _IsFull is ignored here. We assume that boundary checking has been already performed for this index.
    template <typename _IsFull, typename _ItemId>
    void
    __scalar_path(_IsFull, const _ItemId __idx, _Range __rng) const
    {

        __f(__rng[__idx]);
    }

we will have some two functions with the same name and the format excepting the first parameter type which will be used as some tag ?

Please take a look at __parallel_policy_tag_selector_t for details.

[SergeyKopienko]

One more point: __vector_path and __scalar_path tell me about some path but not imp.
May be better to rename them to ..._impl ?

[danhoeflinger]

First round of review. I've not gotten to all the details yet, but this is enough to be interesting.

[danhoeflinger]

Is this a general utility which might have utility for other commutative operations beyond just parallel_for or is there a reason you believe this to be specific to this algorithm / kernel?

If we think it might be useful, we could lift this to a general utility level. Obviously we don't need to incorporate it elsewhere in this PR. An alternative is to add an issue to explore this and only lift it if we find utility.

[mmichel11]

@SergeyKopienko and I had brief discussion regarding this in the first PR: #1870 (comment).

I think in the initial PR making it a static member was the best choice. However, in this new PR I am pushing to expose more general utilities for future use (e.g. strided loops, vectorization paths) to enforce good memory access patterns, and I think stride recommender can be a good general utility. @SergeyKopienko What are your thoughts here with adding this function as a utility?

[danhoeflinger]

Unless there is a clear and easy consensus to make it a utility, lets just leave it here and add an issue to explore its use elsewhere as a utility at a later time. I don't want to derail the PR for this purpose.

[danhoeflinger]

Lets fix the naming of this while were touching all its instances __custom_brick

[mmichel11]

It seems that the historical convention within the internal/ directory is to not use any leading underscores although it has changed a bit over time.

I do not have a strong preference if we make this change or leave it as is, but maybe it fits in a broader discussion regarding the remaining implementations in this directory.

[danhoeflinger]

I'm not sure if there is compelling reason other than resistance to making purely cosmetic changes in the changelog to have a different convention here. This is why I suggest adjusting it while we are already touching all (or most) instances of it. Perhaps someone with a longer historical knowledge of this code could chime in here if there is a reason to keep this with different conventions.

Not super important to me, so optional nitpick.

[mmichel11]

I will wait a bit longer to see if anyone has objections. If not, then I will add this suggestion.

[danhoeflinger]

I think it should be possible to combine walk*_vectors_or_scalars together with some complicated fold instructions, lambdas, tuples, and std::apply.

Take a look at the first answer of https://stackoverflow.com/questions/7230621/how-can-i-iterate-over-a-packed-variadic-template-argument-list. I think you should do something similar, chaining together instructions by returning tuples and then with std::apply.

Here is an example I was playing with.
https://godbolt.org/z/vc8dK4ed6

In the end, I'm not sure if (1) its actually possible and (2) its worth the complexity to consolidate these structs, but its worth considering...

[mmichel11]

So now we have 3 entity with defined constexpr static bool __can_vectorize :

1. class walk_vector_or_scalar_base
2. class walk_scalar_base
3. struct __brick_shift_left

Does these constexpr-variables really has different semantic?

And if the semantic of these entities are the same, may be make sense to make some re-design to have only one entity __can_vectorize ?

These three cases are all unique when you consider that they define __can_vectorize, __preferred_vector_size, and __preferred_iters_per_item. These three fields are all tightly coupled, so in my opinion it makes sense to define them together for readability. If we were to define a single __can_vectorize, then I think it would need to function more like a trait class dependent on the provided brick as the brick itself plays a roll in whether not vectorization is possible. This design would not get us much in my opinion as we would still need specializations for the different cases.

The three unique cases I mention are the following:

1. struct walk_vector_or_scalar_base - Vectorization is possible so long as the ranges meet the requirements to be vectorizable. This is then used to determine iters per item and the vector size.
2. struct walk_scalar_base - Vectorization is not possible due to some limitation of the brick. Binary search is a good example since its accesses are non-sequential. The iterations per work item is still set based on the size of the provided ranges.
3. struct __brick_shift_left - This brick has a limitation that prevents vectorization and that only one iteration per item can be processed and is a special case.