Skip to content

Latest commit

 

History

History
998 lines (609 loc) · 15.5 KB

slideshow.md

File metadata and controls

998 lines (609 loc) · 15.5 KB
Raw

name: empty layout layout: true

name: title class: center, middle

PyTango Workshop

Tiago Coutinho - Vincent Michel

ICALEPCS 2017 - Barcelona

GitHub: vxgmichel/icalepcs-workshop

Slides: tinyurl.com/icalepcs-rp

name: presentation layout: true class: middle

What is PyTango?

  • Python library

  • Binding over the C++ tango libray

  • ... using boost-python

  • relies on numpy

  • Multi OS: Linux, Windows, Mac

  • Works on python 2.7 .. 3.6

... plus some extras:

  • Pythonic API

  • asyncio and gevent event loop

  • ITango (now a separate project)

  • alternative TANGO Database server (sqlite, redis backends)

name: menu class: middle layout: true

What's on the menu?

  • A fresh python3 tango install using conda

  • ITango, a powerful client interface

  • Writing tango servers with 15 lines of python

  • Testing our servers without a database

  • A dive into Jupyter notebooks

class: middle layout: true

Playing with

.center[conda]

Conda is both:

  • an open source package management system

  • an environment management system

  • it runs on Windows, macOS and Linux

class: middle layout: true

Playing with conda

Requirements for this workshop:

  • A 64 bits linux machine

  • An internet connection

  • A Tango database accessible (optional)

  • No sudo access is required

Installing miniconda:

# Download the latest miniconda
$ wget http://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
[...]

# Extract to a local directory (~/miniconda3)
$ bash Miniconda3-latest-Linux-x86_64.sh -b # No manual intervention
[...]

# Activate the conda environment
$ source ~/miniconda3/bin/activate

# Test python
(root) $ python
Python 3.6.2 |Anaconda, Inc.| (default, Sep 30 2017, 18:42:57)
[...]

(root) indicates we the main conda environment activated

Creating a tango environment

# Conda information
(root) $ conda info
[...]

# Create a python3 + tango envrionment
(root) $ conda create --name tango3 --channel tango-controls itango python=3
[...]

# Activate the tango3 environment
(root) $ source activate tango3

# Test itango
(tango3) $ itango
ITango 9.2.2 -- An interactive Tango client.
Running on top of Python 3.6.2, IPython 6.1 and PyTango 9.2.2
[...]

Checkout anaconda.org/tango-controls

name: ITango layout: true class: middle

ITango

Features

  • IPython (jupyter) console

  • Direct access to tango classes

  • TANGO class sensitive device name auto-completion

  • Event monitor

  • Qt console

  • Notebook

  • User friendly error handling

Hands on

(tango3) $ conda install jupyter matplotlib
[...]
(tango3) $ jupyter notebook
In [2]: tg_test = TangoTest("sys/tg_test/1")
[...]

Plan B:

name: Server layout: true class: middle

Uau! Writing device servers has never been so easy!

Device servers with pytango >=9.2.1

from time import sleep
from tango.server import Device, attribute, command

class PowerSupply(Device):

    @attribute(dtype=float)
    def voltage(self):
        return 1.23

    @command
    def calibrate(self):
        sleep(0.1)

if __name__ == '__main__':
    PowerSupply.run_server()

layout: true

class: middle

Testing time!

Server:

$ python -m tango.test_context ps0.PowerSupply --host $(hostname)
Ready to accept request
PowerSupply started on port 8888 with properties {}
Device access: tango://yourhostname:8888/test/nodb/powersupply#dbase=no
Server access: tango://yourhostname:8888/dserver/PowerSupply/powersupply#dbase=no

Client:

$ itango
ITango 9.2.2 -- An interactive Tango client.

In [1]: d = Device('tango://yourhostname:8888/test/nodb/powersupply#dbase=no')

In [2]: d.calibrate()

In [3]: d.voltage
Out[3]: 1.23

class: middle

Let's try out events!

Adding a polled attribute:

import random

[...]

    @attribute(
        dtype=float,
        polling_period=500,  # 0.5 seconds
        rel_change=1e-3)     # 0.1% relative change
    def random(self):
        return random.random()

Going back to ITango:

In [4]: cb = tango.utils.EventCallback()

In [5]: eid = d.subscribe_event('random', tango.EventType.CHANGE_EVENT, cb)

2017-10-07 11:35:17.456138 TEST/NODB/POWERSUPPLY RANDOM#DBASE=NO CHANGE
... [ATTR_VALID] 0.9369674083770559

class: middle

A dive into Jupyter

# Install Jupyter
$ conda install jupyter
[...]

# Make sure the itango jupyter kernel is installed
$ itango
[...]

# Enjoy!
$ jupyter notebook
[...]

.center[Jupyter]

class: middle, center

Lunch break!

class: middle, center

Concurrency in Pytango

Tiago Coutinho - Vincent Michel

ICALEPCS 2017 - Barcelona

GitHub: vxgmichel/icalepcs-workshop

Slides: tinyurl.com/icalepcs-rp

class: middle

Concurrency

First approach: threading

  • 1 listener + 1 thread per client

  • Cons: race conditions and thread overhead

  • Pros: parallelization

Second approach: asynchronous programming

  • Single-threaded with a selector

  • Pros: support >10K clients

  • Cons: require specific libraries

Concurrency

Third approach: message passing

Don't communicate by sharing memory; share memory by communicating. (R. Pike)

  • See Erlang and Go

  • Not incompatible with the previous two approaches.

Warning

⚠ Concurrency IS NOT parralellism

class: middle

What about tango?

cppTango

  • threading: a tango server is at least 8 threads

pytango

  • threading since it's a binding to cppTango

  • asynchronous programming is also supported

    • For both client and server interfaces

    • Through Gevent or Asyncio

class: middle

What's wrong with threading?

  • We have a problem!

  • Let's add a thread...

No**•w we ha2ve prob!**lems

How does tango solve it?

  • All requests are serialized using a monitor lock

  • Useful trick: use a polled update command

class: middle

Asynchronous programming in python

Many frameworks:

class: middle

How does it work?

  • a selector monitors the file descriptors

  • a loop manages a callback queue

  • a user interface is provided:

    • Twisted: deferred and inline callbacks

    • Gevent: asynchronous results and implicit coroutines

    • Asyncio: futures and explicit coroutines

    • Curio and Trio: explicit coroutines only

  • concurrency is achieved using execution units (pseudo-threads):

    • greenlet (gevent)

    • task (asyncio, curio, trio)

class: middle

Asynchronous pytango

Also called green modes, checkout the docs:

pytango.readthedocs.io/en/stable/green_modes/green.html

class: middle

Gevent client mode example

# Install gevent
$ conda install gevent
[...]

# Run python
$ python
>>> # Import from tango.gevent
>>> from tango.gevent import DeviceProxy

>>> # Create proxy (uses gevent)
>>> dev = DeviceProxy("sys/tg_test/1")

>>> # Read the state asynchronously
>>> result = dev.state(wait=False)
>>> result
<gevent.event.AsyncResult at 0x1a74050>

>>> # Wait for the result
>>> state = result.get()
>>> print(state)
RUNNING

class: middle

Asyncio client mode example

# Install an asyncio console
$ pip install aioconsole
[...]

# Run apython
$ apython
[...]
>>> # Import from tango.asyncio
>>> from tango.asyncio import DeviceProxy as asyncio_proxy

>>> # Create proxy
>>> device = await asyncio_proxy('sys/tg_test/1')

>>> # Read attribute
>>> result = await device.read_attribute('ampli')
>>> result.value
1.23

class: middle

A simple TCP server for tango attributes

  • Try this simple TCP server for Tango attributes

  • It runs on all interfaces on port 8888:

    $ python tango_tcp_server.py
Serving on 0.0.0.0 port 8888

  • It can be accessed through netcat:

    $ ncat localhost 8888
>>> sys/tg_test/1/ampli
0.0
>>> sys/tg_test/1/state
RUNNING
>>> sys/tg_test/1/nope
DevFailed[
DevError[
     desc = Attribute nope is not supported by device sys/tg_test/1
   origin = AttributeProxy::real_constructor()
   reason = API_UnsupportedAttribute
 severity = ERR]
 ]

class: middle

More resources

Asyncio overview

Previous pytango workshop

name: none class: middle, center layout: true

The next slides are from the previous workshop at SOLARIS

name: PyTango History layout: true

PyTango History

Started at SOLEIL.

2005 - Moved to ALBA. M. Taurel develops server.

2006 - T. Coutinho main contributor.

2012 - A new API for device servers.

2013 - Project moves with Tiago to the ESRF.

2015 - MaxIV joins the game.

2016 - PyTango 9 is realeased.

2017 - Welcome to Solaris.

name: Current Status layout: true

Current Status

  • github.com/tango-controls/pytango

  • 886 commits

  • 11 releases

  • 26 contributors

  • latest release: v9.2.2

name: Contributing to PyTango layout: true

Contributing to PyTango

Git workflow

  • Github issues

  • Pull request

  • PR merged in develop branch (reviewed and approved)

  • develop branch merged into master at each release

  • develop branch as default branch in github

Unit-testing

  • Based on Pytest

  • Continious integration:

    • TravisCI is running tests in a conda environment

    • for python2.7, python3.5, python3.6

  • 606 tests:

    • client tests

    • server tests

    • event tests

Testing

Useful test context, introduced in 9.2.1:

from tango.test_utils import DeviceTestContext

with DeviceTestContext(SomeDevice) as proxy:
    assert proxy.state() == DevState.ON

Or:

$ python -m tango.test_context some_module.SomeDevice --debug=3
Ready to accept request
SomeDevice started on port 8888 with properties {}
Device access: tango://hostname:8888/test/nodb/somemodule#dbase=no
Server access: tango://hostname:8888/dserver/Empty/somemodule#dbase=no

Documentation

  • Documentation is generated from the sources.

  • The documentation is now hosted on readthedocs. (PyTango version >= 9.)

  • Only works with >= python3.5 (because of the _tango module patch)

Coding standards

  • Flake8

    • PEP8

    • PyFlake

  • There are plugins for most IDEs !

Progress

  • Got rid of metaclass definition

  • ITango moved to a different project

  • Rename PyTango module to tango

  • Refactoring (asynchronous layer, etc.)

  • Cleaning repo

Example

Device servers with pytango >=9.2.1

from time import sleep
from tango.server import Device, attribute, command

class PowerSupply(Device):

    @attribute(dtype=float)
    def voltage(self):
        return 1.23

    @command
    def calibrate(self):
        sleep(0.1)

if __name__ == '__main__':
    PowerSupply.run_server()

name: Pending issues and future work. layout: true

Pending issues and future work

Pending issues

  • Pytango server restart segfault

  • Deprecated NumPy API warnings

  • Compilation warnings related to zero message queue.

Tango9 missing features

  • Pipe events (WIP)

  • Pipe write (client & server, WIP)

  • Dynamic commands

  • Forwarded attributes API

  • Device interface change event

  • Fill polling cache from the code ?

Improvements 1/2

  • Unit tests (always!)

  • Continius integration:

    • Official Conda Tango channel

    • Conda package build by travis

    • Windows build

  • Change of binding (hard one --')

  • Server argparse (easy one!)

Improvements 2/2

  • Clean python module (try tango.+TAB in IPython!)

  • Refactor tango objects

  • Documentation:

    • Add documentation about the documentation generation and mock system

    • Documentation need to be reviewed:

      • Make documentation up to date

      • Promote HL API as the default way of programming in Python

      • Document and promote new features

name: The MAX-IV approach to tango events. layout: true

The MAX-IV approach to tango events

  • Problem with archiving

  • Change events as default event stream

  • Facade device approach

  • Archive events as filtered events

name: The facadedevice library. layout: true

The facadedevice library.

Facadedevice library: A reactive event-based approach to high-level tango devices

name: ITango layout: true

ITango

Features

  • IPython (jupyter) console

  • Direct access to tango classes

  • TANGO class sensitive device name auto-completion

  • Event monitor

  • Qt console

  • Notebook

  • User friendly error handling

Hands on

(tango3) $ conda install jupyter matplotlib
[...]
(tango3) $ jupyter notebook
In [2]: tg_test = TangoTest("sys/tg_test/1")
[...]

Plan B:

name: threading with PyTango layout: true

Threading with PyTango

"Adding a thread is adding at least one problem" V.Michel

Monitor lock

Alternative :

  • Tango Polling
  • Polled Update command

name: Green modes layout: true Green modes

tango.GreenMode.Synchronous
tango.GreenMode.Futures
tango.GreenMode.Gevent
tango.GreenMode.Asyncio

name: Asyncio in PyTango layout: true Asyncio in PyTango

Fill free to test it !