Pydantic v1 and v2 compatibility

[JasonWeill]

Fixes #390. Allows Pydantic v1 or v2 to be used with Jupyter AI. Internally, Jupyter AI continues to use Pydantic v1, but using the langchain.pydantic_v1 package, we can import the pydantic.v1 package if the user has Pydantic v2 installed.

Updates the user docs to refer to the Pydantic compatibility guide in LangChain.

[3coins]

@JasonWeill
This was rejected in #375. What version of pydantic would work for example, if a user has a package that creates a new provider for Jupyter AI?

[JasonWeill]

@JasonWeill This was rejected in #375. What version of pydantic would work for example, if a user has a package that creates a new provider for Jupyter AI?

We talked about this earlier today: depending on LangChain's pydantic_v1 class will make us rely on LangChain's pass-through implementation of either pydantic (if the user has Pydantic v. 1.x) or pydantic.v1 (if v. 2.x). That concern about LangChain blocked #375 in late August, but at this point, I think that (a) we are wedded to LangChain until we do our next major version bump, at least; and (b) LangChain's Pydantic implementation does not appear likely to change anytime soon.

Following the guidance from LangChain, users who are building on top of Jupyter AI will be expected to use Pydantic v1 classes, to prevent conflicts. This is in LangChain's docs about Pydantic v1, which I link to from our docs as revised in this PR.

[JasonWeill]

@meeseeksdev please backport to 1.x

[lumberbot-app]

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

git checkout 1.x
git pull

2. Cherry pick the first parent branch of the this PR on top of the older branch:

git cherry-pick -x -m1 2c89d139a8205f1fcd318cfd922e1771df7564ef

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

git commit -am 'Backport PR #466: Pydantic v1 and v2 compatibility'

4. Push to a named branch:

git push YOURFORK 1.x:auto-backport-of-pr-466-on-1.x

5. Create a PR against branch 1.x, I would have named this PR:

"Backport PR #466 on branch 1.x (Pydantic v1 and v2 compatibility)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.