BackStack issue with <fragment> destinations

[massivemadness]

Describe the bug
There's an issue when navigating from <swipeable> to <fragment> destination, the fragment will not be added to the backstack, which makes impossible to go back using popBackStack() or navigateUp() methods.

To Reproduce
Steps to reproduce the behavior:

1. Make <swipeable> your start destination
2. Navigate to a <fragment> destination
3. Call popBackStack() to return to the previous fragment

Any ideas of how to fix
The issue is not exactly in SwipeBackNavigator, It's the default behavior of androidx.navigation.fragment.FragmentNavigator, which depends on the backstack size to decide whether it should add a transaction to the backstack or not.