Discussion: What does it take to defragment an open source community? #14
Comments
|
My favorite success story has to be NumPy, but I wasn't close enough to that to know what made it work (perhaps @njsmith has some ideas?). @aquilesC mentioned the PyLab project via this discussion, which is a joint effort between the developers of lantz, pymeasure, and instrumentkit (maybe others). That seems to have had some success, but again I can't really tell you that story (I welcome comments from @hgrecco, @bilderbuchi, @scasagrande). From my own experience working on vispy, I can say that project was not easy, but ultimately successful in some of its goals (importantly, we reduced the number of active projects and created a self-sustaining community project). The existence of large, prior codebases and strong, competing views about architecture were a major challenge. I and others have worked to avoid that by keeping the scope of this project minimal as a first step toward establishing a cooperative community. I think one of the most important things we'll need is a core set of developers who are able to commit significant time to the project. Typically a project like this is driven by necessity--either someone is developing code because they need it for their own research, or because someone is paying them to do it. Many of the people in this group have already invested in their own acquisition systems and will not have the resources or incentive necessary to push the project through. So perhaps an early question to resolve is whether anyone in this group has the necessary resources and incentive, or what organizations would be willing to support the project. |
|
Thank you for the invitation to contribute. I'll try to be brief to keep focus (a lot can be said about this topic), so if my comments appear curt that's probably because of that. So, from my perspective the effort to unify you mentioned has not really worked. I invite you to look through pymeasure/pymeasure#53 (warning: long thread), which is what the effort to catalogue you mention above came out of. It shows basically the course of the effort to unify several hardware interaction projects, with a lot of discussion and reasoning from a variety of contributors. |
|
One thing that made the unification effort hard is that there are so many ways to segment the space of interest (see the online sheet in the OP of LabPy/labpy-discussion#23 for a try to make structured sense of the landscape), so it's very easy for multiple groups to have different focus (and projects), instead of collaborating on one package. Only having quickly looked from the outside, this also seems to have happened here - you seem to want to start a package focused around microscopy, but instead of using and contributing to an existing project (e.g. https://itom.bitbucket.io/ or https://github.com/python-microscopy/python-microscopy), you initiate a separate thing. Instead of trying to contribute to another effort to unify (e.g. https://github.com/LabPy/ which you might not have found before), you start a fresh community. This will probably lead to more fragmentation in the space, even though you cite the imo extremely relevant https://xkcd.com/927/. |
|
Sadly, this exacerbates the one primary problem that I see OSS project having - lack of developer resources. Therefore, we all end up with N projects with slightly different scope and each with ~ 0.M time-starved maintainers, instead of one project with a more sustainable pool of N*0.M maintainers. This is also how I read the situation from LabPy/labpy-discussion#23. Personally, from a candidate pool of InstrumentKit, Lantz and Pymeasure, I have drifted towards using (and trying to contribute to) Pymeasure mainly because there is at least some activity there, so there is a chance of getting your bugs fixed. Nowadays I have come to regard developer activity as one of the primary rubrics when evaluating if I should use a project or package. |
|
So, what is the solution? I would love to see a Numpy for data acquisition :-) I think, considering the tendency of potential contributors to have different foci, it could be worthwhile to evaluate what is out there, and
IMO, one of these kernels is device drivers for data acquisition, i.e. something on top of VISA, IVI, etc. that implements and tests(!) lots of ready-made interfaces for the many instruments out there! GUI and procedure automation would be different things and out of scope - there, people already have wildly differing needs/wants. Everybody likes pre-made instrument drivers e.g. for the oscilloscope in their lab, though. This is still a hard problem, and lots of different way to "skin the cat" (e.g. what a common API should be)! I think InstrumentKit is a good example for that, especially the way to test against instrument protocols, but sadly there's very little activity. Lantz is basically in the same situation. |
|
Oh man, this still grew too long, sorry for the wall of text. >.< I guess my tl,dr could be:
|
|
Many thanks for your comments, @bilderbuchi ! One note I want to throw in: although we have already started some work on a new project, I consider this just research in that it is intended to help us decide whether we can agree on the kind of API we need, and from there whether any of the existing systems are adequate, or can be adapted to our needs, or would be willing to deprecate their own code in favor of another. |
|
Sure, I hope it helps! I think in the frame of device drivers, my personally ideal package would have
Ideally also then the
from all three to get nearer to critical mass. :dreams: |
|
I think what is important for attracting a sizeable crowd is that what you are rallying people around is generic and not focused on a specific use case. For example, if the direction goes into "control for microscopy", or Quantum computing something, this puts me on a sceptical footing concerning the usefulness for my usecase (I am mostly controlling vacuum chambers and high-voltage power supplies), and I think other people will react similarly. If I think about the larger, well-established projects in this space (numpy, PyVISA, pyserial, matplotlib/pyqtgraph,...), they offer building blocks, not specialist tools. |
|
In any case, I recommend sitting down in a comfy chair, making a glass of your favourite concoction, and read through the long but hopefully illuminating pymeasure/pymeasure#53, I feel we have covered similar ground over there. |
|
@bilderbuchi, I think you summarized the situation very well, and definitely captured my feeling on this. That issue is a great display of multiple failed attempts to consolidate the space. :) |
berl
changed the title
|
@bilderbuchi very interesting read on pymeasure#53, especially considering the range of timestamps traversed by that discussion! I think that there is a natural tension between a specialist group (could be single molecule microscopists, or slice electrophysiologists or whatever) trying to do this for their use cases and the desire you expressed to
And then the impedance mismatch between project scope and contributor hours is probably easiest adjusted by starting small and specialized and building up, and that path of least resistance has led to the hordes (?) of dead or zombie projects in this space staggering around github. I still think that an organic approach based on specialist projects wanting to remove driver dependencies from their projects might have the right combination of risk-reward to gain community buy-in incrementally. It sounds like I'm not the first person to naively hope for this... One more thing Iearned from heading down the amazing array of python hardware control rabbit holes (some from your comments I had no idea about!): There are wildly disparate use-cases represented here and wildly diverse sets of hardware represented in these labs. I think our initial ideas and discussions implicitly included only a tiny fraction of this diversity. Given the resource problems you spelled out above, it seems like carving out the scope of this project (like a list of hardware devices that we're considering and not considering) is critical to making it manageable to even have a discussion. |
|
@berl thank you - basically, 👍 on what you said! I think the complex thing here would be to define a generic but extensible API and usage patterns fitting as many device types as possible, so that everybody wanting to contribute a device driver can find their slot. IVI instrument classes are one way to approach this/learn from, but I am not sure how well this translates to Python. |
|
Hey everyone, I'm actually still working on InstrumentKit, but I no longer work at a job that involves interfacing with hardware. I work on IK as I have time, and some of my bigger ideas can take long enough that people might not notice progress is being made. Such as moving to Pint, implementing a pluggy based plug-in structure, etc. It also doesn't help that I don't get Github notifications unless I manually check. As to why I personally dropped off any discussion on combining efforts, its pretty simple; I just don't have the capacity to carry these sorts of conversations. My day job is to lead a test automation effort (in Python even) and so nearly all of my development energy is focused on that. When I program for fun, I'd rather work on InstrumentKit at my own pace and deal with the few ideas people bring me, instead of having to manage yet more developers. Many people have requested an interface layer for IK to use pymeasure's drivers, but no one has submitted anything at all. And without any tests on the drivers, well, what am I to do 🤷♂️ And to be honest, I didn't see the point me moving to another project that is missing core features that originally spurred the creation of InstrumentKit. |
|
Also relevant I think, if you are not aware: https://github.com/QCoDeS/Qcodes |
|
@scasagrande thank you for your thoughts! I am not sure if I read between the lines too much, but I want you to know that my remarks regarding IK were not made out of a sense of entitlement to your development/discussion effort. I understand the situation, and it's of course fine to work on this when you want. For me the thinking was mainly "if I run into problems, will I be able to reach somebody to help me fix it?", and for that very little repo activity was a bit of a warning flag. |
|
From my point of view, it is also important to keep the code pythonic, readable and as short as possible. (Which was already noted by LabPy/labpy-discussion#23 (comment)) I am one of the experimenters who are OK with reading driver source code to get things working, but expect it to be clear like The reality is truly frustrating. Many of existing projects need some fixing (which is usually not documented), and more than once I wasted a whole day analyzing some interface's logic, only to find out the project does not address my needs. |
|
Further, for mass adoption, the project should keep mandatory dependencies at absolute minimum. This means module detection should be dynamic; if e.g. Also this stipulates that the graphical frontend should not require
But a simple TkInter GUI would be very useful, if it would
|
|
It's been a long time! The pandemic certainly rearranged a lot of priorities for me.. |
|
@campagnola, thanks for the link to another relevant project, I added it to the table: https://ethercalc.net/1anmq248ktu6 It is amazing how much duplicated human effort could have been saved if there was one central repository for instrumentation... |
|
Added https://github.com/AlexShkarin/pyLabLib to theh table: https://ethercalc.net/1anmq248ktu6 This seems like one of the more elaborated projects here. |
|
@roesel I noticed your issue over at the pyLabLib and wanted to draw your attention to this thread- some of the projects linked above may be interesting to you |
|
@berl Thanks for the tag! Apart from the list of projects, there is definitely an interesting discussion going on here (and even with some familiar names, hi @FilipDominec). Although judging from the list, it looks like pyLabLib is still closest to what I am looking for right now. From briefly scanning through this thread and a few linked ones, I would say there should also be a clear separation between libraries (that you can import when writing your own python code) and programs (which you can immediately use). It's fairly straightforward to make libraries modular and reusable, but when it's a finished program with a singular purpose, it is not only harder to extend and reuse, but also requires making more decisions about dependencies such as a GUI framework. |
|
I think we as a community still have a visibility problem. By now, every time someone starts a new, greenfield project (like pylablib, which is not yet a year old) instead of contributing/pooling resources/adapting an existing one, I get a heavy sense of https://xkcd.com/927/. 😞 Over at PyMeasure, it's more of a maintainer manpower problem -- we don't manage to work down all the incoming PRs in a ~reasonable amount of time. It's getting better, though, slowly. |
|
@bilderbuchi I totally get what you are saying, but I think there is a bit more going on than someone wanting to make "yet another standard". Speaking as an employee tasked with automating a setup in a lab, which contains everything from servo motors to photon-counting cameras with their own proprietary software, I would like to be able to confidently say "I'll do it all in Python" rather than grudgingly choose to do it in LabView, which is how it's usually done around me. The LabView approach also doesn't come with a "one-for-all" solution from one source, but somehow all the manufacturers provide a VI that I can glue together with the rest and get something working fast. Maybe what I would actually like is not one swiss-knife library that does it all, but simply an up-to-date list, where I can find each one of my instruments and see which libraries have implemented their control. That seems like a much more useful list (for my use case at least!) than a list of projects with descriptions of which 1-5% of areas each project is able to cover. That's not to say it wouldn't be great if all this code eventually merged into one library that would support a wide range of devices! I don't have experience with PyMeasure (I wonder if it existed already when I was writing my own code to control a Keithley 237 six years ago?), but despite it's age, pyLabLib already looks to cover a very wide span of device types, while letting you use code for any single device without having to even know about the existence of the rest. That looks like a very promising approach to me! If I need to move a Thorlabs stage, it's great that there are multiple other "open source microscope control" packages that have the ability, but if I cannot find the |
|
Previously, I was in a similar boat as you are, although with a different (simpler?) set of instruments (mostly SCPI-compatible, few modbus-based, some dll-based). I pulled it off, so don't lose hope! :-)
Yeah, I fear the network effect is strong here. That definitely puts any Python-only (or rather "non-Labview"/"non-institutional") effort at a disadvantage.
Possibly, but only barely; first commit was in April 2014.
Maybe I don't understand, are there libraries that necessitate you to "know" about other devices when using one specific model? Or what are you trying to say?
I agree, this would be quite useful (if kept up-to-date automatically), and would definitely be an improvement! Also, I don't necessarily mean or want to unify everything in one package, just to pool scarce developer resources, and (ideally) not add new ones where existing ones could be adapted. ☮️ Maybe the take-away has to be that any abstraction beyond PyVISA/pyserial/nidaqmx/minimalmodbus is ultimately doomed because people's needs and interests are too diverse? It's already visible e.g. in the distinction between miscroscopy folks and others, and in the difference in character/scope between something like EPICS/Taurus and simpler packages like ours. Code heritage/legacy/back-compatibility is also an issue, of course. Most packages are quite different in architecture, so are hard/impossible to merge even if interest/drive were there. Decisions from the deep past sometimes restrict what you can currently do, and how generic your package can become. I surely see this in pymeasure, too. In that vein, younger projects have an advantage -- they can study the available solutions, learn from their mistakes, and build it up all over again, better (ofc, at a large time investment).
We're getting there (slowly) -- we just merged a contribution with our first (I think) stepper motor driver. 🥳 We have few Thorlabs instruments, and that is not gonna change for quite some time I fear -- by necessity we have to restrict our scope to non-dll drivers for now. |
I don't have specific links, but I remember coming across several different projects last year that tried to implement a program for controlling a microscope, which had support for (among other things) various stages internally, but unless you were comfortable with diving into the source code and extracting the implementation yourself (which can cost a lot of time in some cases), there was no easy way to just move a stage without spinning up their full GUI or setting up objects that would never be used. Maybe it was even possible, but since the docs did not talk about it, I imagine a person looking for a solution would simply move on rather than invest their time towards an unknown outcome.
Please don't misunderstand, I was trying to say that PyMeasure looks like the opposite of what I am describing above! It seems to be an exemplary library that you can use for your device without the need to commit to any particular model or ecosystem. |
Not at all! I took this to refer to "moving a Thorlabs stage", which we can't help with at this point, due to lack of drivers. All good 😃 |
|
Just piling up in here, if I understand correctly @roesel is looking for some simple implementations on which to build (like getting a simple UI for a stage, or a simple UI for a camera, etc.) This allows you to extract and glue things together. Pretty much in the same way you can use VI's that companies provide for LabView. I think this is perhaps something missing, where very large projects don't allow you to extract just the bits you need, and many projects focus on being very generalistic (from developing drivers all the way to visualizing data in real time). How can we build and, more importantly, share those building blocks is another type of discussion, but that can be very valuable to have at some point. In between the challenges, the most pressing one is the lack of interest from companies in releasing boilerplate code that just works. The other, is that the use cases of hardware vary so much: the same camera used for machine vision in production lines of a pharmaceutical is the one I use for a low cost microscope. It makes it hard to settle on the major guidelines. Not all communities favor Python over C#, or LabView, for example. And here is where the interest of companies can't be steered unless the use case is large enough (I've even seen a company that sells analysis software for microscope images forcing users into TCL because if they needed to learn to script, choosing one or another language does not provide any value). |
|
@aquilesC That's exactly what I was trying to express! The other thing buried in my thoughts above is that these building blocks should be GUI agnostic, or even better provide a GUI which is decoupled from the business logic and easily usable on its own. Where one person has a Qt interface, another uses GTK, a third one needs to write a plugin to augment an existing C# program, and a fourth person simply wants to run a one-time script. That's another point where Python loses to LabView, since all the VIs from all manufacturers already share both the language and the interface, and are easy to combine. To comment on your second point, one thing we can do is to ask the manufacturer for a Python implementation when setting up new equipment. While I have myself rarely actually received one, I hope it has at least raised the awareness that this is something people are interested in. Do you think there is another way? |
|
@roesel the original repo here was made as initial research into some of the key issues you have brought up. The name
important context/history here is the micromanager project, which through pymmcore or pycromanager can provide access to hardware that can be otherwise difficult to control because of proprietary drivers, NDAs, etc. The C drivers in micromanager represent a huge value that has been built up over years of working with individual contributors and manufacturers, and to some extent |
|
I think IVI could be something of a labview Alternative, with device vendors sometimes providing drivers, and with a generic driver structure in place, but I don't know it's current status. |
|
@bilderbuchi You are right we should now collect a universal list of all supported instruments. The structure of all cca. 38 projects in the overview table seems different, so they cannot be scraped automatically. The list is going to contain hundreds of items, so a directory of YAML files may initially be easier to make than a huge online table. It should be noted that it's not all about supported hardware - some projects implement clever hardware autodetection, measurement control & visualisation etc. So there is also stuff that cannot be easily put into a table. @roesel David, please don't forget about TkInter - it is distributed by default with Cpython (all versions, on all platforms), so it takes no extra space and loads immediately. It integrates well with Matplotlib, and also with direct numpy.array drawing. Also Tkinter+TTK is not as In an ideal world authors of all the competing projects would recognize that it would be beneficial to humankind to share one wisely selected repository of instrument drivers, and would gradually port their labs to it. World is not ideal⁽ᶜⁱᵗᵃᵗⁱᵒⁿ ⁿᵉᵉᵈᵉᵈ⁾, there is nothing like a selected repository, but still we could ask them kindly for:
It is going to be human-resource sensitive, as it challenges the appreciable effort people put in their own laboratory control software. I know the feeling from my own experience (e.g.). But as soon as the list of instruments is sketched up, we should openly communicate across the projects. We are facing a tough problem here, an it would grow if it stayed neglected. |
🤣 congratulations, you are where I was when I started assembling that table years ago 😁
I suspect what should be feasible would be to create a couple of lookup rules that scrape instrument names from the repositories of the projects (or maybe any html docs). For example, for pymeasure, we collect all instruments in one folder, with one subfolder per manufacturer, with one python module per device model. I assume most projects will have a structure that is similarly simple to express, and this has to be done only once. I would have started something long ago, but I'm very much not a web person, and pymeasure maintenance eats all my spare time budget for this already. |
@FilipDominec I have nothing against TkInter, but I still believe all projects should strive for the library/program separation I tried to describe above. It's great if the author writes a GUI, and TkInter is definitely the most universal approach, but having business logic decoupled from the interface makes adoption much easier no matter the circumstances.
This might just be my personal opinion, but I believe most scientists want to get the job done, hence the widespread use of LabView despite its shortcomings. The start a new project step only comes when no easy-to-use implementation is to be found, in most cases anyway. It seems there really is much more fragmentation than there is competition, so perhaps once the existing implementations area simpler to find, all this could get easier and easier? |
|
A table update: Added https://github.com/caizikun/Instruments-1 to the big list https://ethercalc.net/1anmq248ktu6, reaching total 43 projects of mostly overlapping scope. |
|
Hm, did you notice the big comment in that sheet that says
I added that a while ago when I had completed the first iteration of https://github.com/pymeasure/python-lab-automation-landscape, so that we have a saner (and citeable) way of tracking these than a file at some file sharing service that might go away at any time. I guess it needs to be even more prominent? I'll see if I can add a yellow background. 😁 |
|
Dear Christoph,
I did notice it, but had no clue where else to contribute.
I hope your project the table refers to can somehow process the new entries. But I did not have time to take a closer look at it.
Filip
------- Original Message -------
…On Thursday, January 26th, 2023 at 3:07 PM, Christoph Buchner ***@***.***> wrote:
Hm, did you notice the big comment in that sheet that says
> THIS LIST IS DEPRECATED/NOT UP-TO-DATE, PLEASE SEE https://github.com/pymeasure/python-lab-automation-landscape INSTEAD
I added that a while ago when I had completed the first iteration of https://github.com/pymeasure/python-lab-automation-landscape, so that we have a saner way of tracking these than a file at some file sharing service that might go away at any time.
I guess it needs to be even more prominent? I'll see if I can achieve that.
—
Reply to this email directly, [view it on GitHub](#14 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AAD47TTS647V6VCAZNVGN5DWUKAKDANCNFSM4MXPUEOQ).
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
Did you check out the link? There is a readme, but I guess I can make the part about how to contribute more prominent. (you basically have to add one very short yaml file). |
No, I migrated all the previous ones by hand/via some editor search/replace. I'm not set up to process new entries, but I did not want to delete the contents of the Ethercalc sheet totally. Maybe i should, to avoid this going out of sync. |
Current situation: several python projects already exist, partially competing and partially complementary. For the most part we agree this is wasteful, that we could potentially do much more by working together, but actually accomplishing that can be very difficult.
In this thread I'd like to collect thoughts about this process:
The text was updated successfully, but these errors were encountered: