help: pastebin provider should be configurable

[dgw]

The help module used to use GitHub's Gist function, but anonymous Gists were deprecated (see #1254).

We switched to ptpb.pw as a pastebin provider (see #1257) for Sopel 6.5.2, but for a short while yesterday (November 30, 2018) its domain was in an expired state. While the domain has been restored, the notice @kwaaak sent me on IRC about it being gone gave me a good scare.

I think it's prudent to start thinking about how to make help.py more configurable so a more permanent outage of ptpb.pw won't break Sopel's help functionality until the next release. Our release cadence is slow enough that it's a valid concern. I want to keep trying to minimize the number of things that can break in a way that gives the user no recourse but to wait (like etymology.py—see #1248). New versions should come out when they're done, and not be rushed because something is broken in the latest release.

In #1212, it was suggested that the use of a pastebin should be configurable, with the option to use the old behavior of just sending the command list to IRC. Also in that issue, I brought up the idea of dumping to a local file on Sopel's disk and linking to it, kind of like how meetbot.py's meeting notes work.

On top of those features, I propose a new ChoiceAttribute with several different supported pastebin providers. ptpb would be one, of course.

Suggestions for additional pastebin services are welcome in the comments! Any service like pastebin, hastebin, etc. is fine as long as it seems unlikely to vanish in the foreseeable future.

---

Here is the current list of possibilities:

• ptpb.pw shut down (ptpb.pw permanent shutdown ptpb/pb#246)
• pastebin.com requires API key
• hastebin.com
• 0x0.st
• termbin.com
• clbin.com — default as of v6.6.5

[uint]

Hi!

From my understanding you're not working on this yet? Mind if I give it a go, along with related tickets? I'm looking to gain a bit of experience in Python. This looks like something I should be able to tackle.

Regardless of who does it, I spent a little time checking out the pastebin services you suggest. This might help.

pastebin.com

This one, according to the API documentation, requires a dev key. We'd probably need to give the user a config option to provide a dev key. Seems like unnecessary hassle I think?

hastebin.com

They don't seem to document their API which worries me (no claims of it being maintained the way it works now), but I figured out how stuff is processed there nonetheless. This seems to work:

>>> r = requests.post('https://hastebin.com/documents', data = 'stuff')
>>> r.text
'{"key":"nepiwelixa"}'

0x0.st

Official API, no devkeys needed.

>>> r = requests.post('http://0x0.st', files = {'file':'stuff\n'})
>>> r.text
'http://0x0.st/s7t0.txt\n'

[dgw]

From my understanding you're not working on this yet? Mind if I give it a go, along with related tickets?

Certainly! Feel free to throw out ideas for other services, too; I just listed a few off the top of my head when writing this particular issue. I recently discovered termbin.com, for example, which is absolutely bare-bones and expires pastes after a month—no need to worry about stale help output getting left behind. I'll add it to the list of ideas in the OP.

As far as pastebin.com itself goes, you're right that the API key requirement is basically an immediate disqualifier for inclusion in Sopel's core. (Again, I made that list without researching APIs first. 😅) Therefore, I'll remove it from the OP's list of ideas.

[kwaaak]

This is overkill, but if someone wants to flex their Python muscles, this is an opportunity: Use a PrivateBin pastebin service, https://github.com/PrivateBin/PrivateBin/wiki/PrivateBin-Directory

[dgw]

Definitely overkill. That said, I'd at least consider a PR that added it as an option, but preferably it would come separately after the basic configuration setting is done.

[uint]

I think termbin should be easy enough to add via a bare-bones TCP socket. Will look into it.

I've been toying with the idea of using software like wgetpaste (popular on Gentoo) or pastebinit (it's in Debian/Ubuntu repos), but the former is a bit niche and the latter seems outdated, at least on my system. These applications are CLI utilities that let you submit stuff to various pastebin services. Integrating one with Sopel would relieve us from having to maintain a list of working pastebins and the methods to use them. But alas.

[dgw]

These applications are CLI utilities that let you submit stuff to various pastebin services. Integrating one with Sopel would relieve us from having to maintain a list of working pastebins and the methods to use them. But alas.

Alas, indeed. It would probably also relieve us of ever supporting Windows properly. Maybe even macOS.

It's not that those tools can't be run on non-Linux machines—they're just Bash & Python scripts, respectively. But if they're niche on Linux, imagine how hard it would be to get them installed on Windows! (Well, "hard" is not quite it, but it's probably not as simple as a package install command.) And that's assuming those scripts even support Windows themselves, which I haven't tested. It also seems kind of silly to spawn an external process to do something that Sopel could pretty easily do on its own.

One possible outcome of this feature idea, if someone really wants to get down and dirty with the help code, is a truly generic pastebin system that can support (nearly) any site. Past, present, or future. I imagine it's doable with just a few options:

• pastebin_api — the endpoint URL; APIs that require a key (like pastebin.com) should just have the user include it here instead of adding a separate setting and then having to do string formatting
• pastebin_request_type — POST (the most common, and default if unset) GET
• pastebin_text_field — the request field in which the text should be sent (commonly file or text)
• pastebin_options — might be useful to pass extra options as key=value pairs, like to set paste expiration or something

(Field names are for illustration and not necessarily the best choices, and this list probably doesn't contain all the optional values that some APIs would need. This is the spitballing phase!)

While it might be neat to do that at some point, it would be pretty intimidating to the user. That's why I still think having at least a handful of built-in services that can be set by pastebin_service (for example) in the config is the best immediate option. Maybe later, if the user-defined pastebin idea happens, it could just be enabled by pastebin_service = custom or similar.

[uint]

These applications are CLI utilities that let you submit stuff to various pastebin services. Integrating one with Sopel would relieve us from having to maintain a list of working pastebins and the methods to use them. But alas.

Alas, indeed. It would probably also relieve us of ever supporting Windows properly. Maybe even macOS.

Fair point. Worries like that are the reason I've been looking if it's possible to get something like that as, say, a bundled python library rather than an external process (I agree that would be too much, especially since it adds a dependency), but no luck. I'll just stop making things complicated and add a handful of pastebin services for the user to choose from for now.

[dgw]

It really is the simplest option to just hardcode a handful of services for the redundancy. Some more complicated solution can come later if there's demand. :)

I look forward to reviewing your PR!

[dgw]

Updated possibilities: removed ptpb.pw (it shut down — ptpb/pb#246, #1495) and added clbin.com.

[dgw]

Will be resolved by #1451, assuming it doesn't stall. Removing "Patches Welcome" label.