Document developer experience when creating database taps

[amotl]

Hi there,

while working on GH-1, we've encountered some struggles to make the plugin work. The most probable reason is that we didn't work much with Singer/Meltano before (as a plugin developer), and that we didn't follow the upstream documentation closely enough, effectively boiling down to plain ignorance ;]. Still, we wanted to write down where we have been running into obstacles, to have the chance to improve the documentation.

Please find the report and corresponding thoughts below. Saying all that, after you know how things are connected, I want to say the Singer ecosystem is amazing, and the Meltano layer adds the right convenience abstractions to consume it better. Thanks to all who make it better each day. 💯

With kind regards,
Andreas.

I struggled very hard to discover the corresponding mechanisms around "stream/table selection" as a newcomer/developer.

Specifically, when developing the tap, even when using tap-postgres as a blueprint, I found it hard to connect the dots regarding entity selection rules and the select extra of Meltano fame, because they naturally don't have any representation within the Singer SDK. This may revolve around topics discussed at meltano/sdk#1652, but also naturally, those are topics you are being discussed by the system architects, and I only have been able to discover them much later when starting off not knowing anything about those.

As a newcomer developer, while having been so amazed that the SQLAlchemy-based reflection machinery thoroughly discovered and reflected all the system tables of CrateDB, all of them have been deselected, and I haven't been able to figure out how to select them: First, folowing the Singer way, I discovered that using a catalog would be the right way, and I edited deeply nested JSON structures, trying to get data flowing out of our sys.summits table. Eventually, I discovered the entity/stream selection service added to the baseline Singer implementations on behalf of Meltano, but still it did not work: All the streams have been deselected, and you can imagine my frustration because, well, you see the light at the end of the tunnel, but there seems to be an invisible barrier grid between you and the tunnel exit.

In a tinge of despair, I already started to work around it somehow by mimicking the select: attribute into our tap implementation, hooking it into settings.select_beta, only to finally discover that I needed to add the capabilities: section to the Meltano project definition, to make the select: thing work without futher ado. You can imagine my relief at this point to finally have discovered one of the holy grails of Meltano. So, it all boils down to me being too silly or lazy to read the documentation properly. Still, I wanted to share that story with you, and I hope you can appreciate it.

Just revisiting the documentation about Creating a Custom Extractor, also admitting I didn't follow it, I think I also got confused within learning about the difference of config: vs. settings:, and that select: is a completely different thing, not mentioned on that page at all. Maybe it is special for database taps?

[amotl]

Well, it is all right there:

• Plugins
• Plugin extras
• How to use: Plugin extras
• Capabilities

However, I don't know why, but I haven't been able to connect the dots between those concepts sitting on top of Singer. One reason might have been that I studied the documentation wrongly, and that I tried to discover something about "Meltano Services" (apparently the underlying machinery for those plugins/extras), which I had picked up from the code base.

So, I think everything is excellently documented from a user's perspective, and it's only me who took the wrong "bottom up" approach. 🤦

[amotl]

We converged this report into Learning Singer » Creating a database tap.