Add a .gitignore and make HF deps fully optional

[muellerzr]

Enable Accelerate integration by fully making HF deps optional

What does this add?

This PR adds import guards across unsloth for the various integration libs, making sure that core imports are still possible without triggering external lib imports.

It does so by following an is_x_available workflow, similar to what we use at HF.

This PR also adds in a .gitignore relative to working with a python file, as I found it a bit cumbersome not being able to do git add .. If we want to remove it, that's quite alright 😉

Who is it for?

Users of unsloth who want to try out the cool gradient offloading mechanism, while only having the core parts of unsloth installed.

Why is it needed?

There are areas in the code that do a large deal of patching to transformers and peft. This simply guards said patching so its only done if the lib is available.

What parts of the API does this impact?

User-facing:

None

Internal structure:

Adds new library imports checks for:

• bitsandbytes
• peft
• transformers
• flash_attn

[danielhanchen]

@muellerzr Thanks again!! Sorry was caught up in stuff - will review today!

[muellerzr]

No worries @danielhanchen, no rush :) the integration on our side can still be made, just not merged until this and I probably won’t get to that until later today/tommorow, so plenty of time!

One thing I’ll look at eventually is Accelerate is device-agnostic, so the torch.cuda() specifics may need to eventually be agnostic in some form.

That may mean eventually choosing this implementation for GPUs, and accelerate mimicking it but with appropriate autograd decorators. We shall see, and find a good way to share credit and combine efforts still :)

I’m not wondering if it can just be torch.amp.custom_fwd also but haven’t dug into it yet

[danielhanchen]

@muellerzr Sorry just got to this - I ran T4 and L4 on some examples and nothing seems broken + had a look through the code!

In terms of autocast - I can add an update to make the autocast work on all devices - I need to check again how (forgot lol)

For moving to GPU and back, I think there can be a way forward - let me investigate first