Skip to content

Harvester Operation

Jump to bottom
Victor San Kho Lin edited this page Apr 29, 2017 · 4 revisions

How to Screen

  • Ideally Harvester should be a system service process.
  • But before we get into there and for ad-hoc development, we will be using screen. This is norm for most developer practice at alpha/beta phase.
  • So, good to read some tutorial if you are new to screen and get familiar yourself.

Get into Screen

  • Login to 115.146.94.41
  • Check the all screen windows
ubuntu@r-d3i1sr7z-0:~$ screen -ls
There is a screen on:
        16844.pts-0.r-d3i1sr7z-0        (27/04/17 23:35:01)     (Detached)
1 Socket in /var/run/screen/S-ubuntu.
  • You will see a detached screen window with name: 16844.pts-0.r-d3i1sr7z-0. Note that screen name will be vary as a new screen created.
  • Switch/attach into this 16844.pts-0.r-d3i1sr7z-0 screen window
ubuntu@r-d3i1sr7z-0:~$ screen -r 16844.pts-0.r-d3i1sr7z-0
  • Or if you have only one active screen, you can also simply do -r without explicitly passing screen name.
ubuntu@r-d3i1sr7z-0:~$ screen -r

Normal Start/Stop Operation

...
id: 858145954783875073  rev: 1-bf3a867f6f7b8c682354bc8d84c3411b
id: 858145959896690688  rev: 1-fa2834a2779ec78236d1769611bbc971
... [skip]
id: 858145984387231744  rev: 1-9a06f2c306bc80f25cf6c86732c38e63
id: 858145985100341248  rev: 1-a250da55633c36fb5e678c078e01b3ef
...
  • That's because, now you are at Harvester Command Line Interface aka harvester.cli
  • Press Enter, you will briefly see the bash shell prompt then goes back to Rain Code again. Like this.
...
id: 858149521792024576  rev: 1-96da0e8a236377317f122b45d12deafb
id: 858149521741815808  rev: 1-ea83d6d035dbc3bd6619e69f0311fcb3

ubuntu@r-d3i1sr7z-0:~/goal$ 
ubuntu@r-d3i1sr7z-0:~/goal$ id: 858149535251550209  rev: 1-636052d83cc44b57bcb722f40d596e22
id: 858149535092273152  rev: 1-fc3b1446c5928d9213bd62d38f187e14
  • No worries. That's expected. Now trust me, you are at the bash shell and you can invoke shell command as usual. Because harvester.cli is running in background. So, let bring it back to foreground. Just type fg then Enter.
...
id: 858148924657459200  rev: 1-02fb7ff902d29f58d4a6f2d1fe9364b5
id: 858148929912922114  rev: 1-207216ff9fe787fe7c05c5f2a39068b8
fg
python3 -m harvester.cli
  • Now harvester.cli has brought up to foreground of the shell.
  • Pressing Ctrl+C will signal harvester.cli to stop the program
...
id: 858148422976585728  rev: 1-9ca8da467cd06232e349b726cd8eeb28
id: 858149036678938624  rev: 1-d2e07cccbdb2b2e11b3d90e18a390e6d
^CStopping...
ubuntu@r-d3i1sr7z-0:~/goal$
  • To start harvester.cli again in background
ubuntu@r-d3i1sr7z-0:~/goal$ python3 -m harvester.cli &
[1] 19393
ubuntu@r-d3i1sr7z-0:~/goal$ Starting tweet stream harvesting. Press Ctrl+C to stop.
id: 858149172456824832  rev: 1-c17a6c0359eb7141e8e190fb870b3461
id: 858149173173993473  rev: 1-78d847a8ad6eef4aa488f9b2db1a6f74
id: 858149175304769536  rev: 1-85e2a077ef4263ae2f4e17edad0e994a
...

Program Crash/Halt

  • After you attach to the screen, if harvester.cli has crashed or halted, you will see something like this.
id: 857631461553451008  rev: 1-a3c22de9d9bc8b2ec64f4a5f3ecb8132
... [skip]
id: 857631500531105793  rev: 1-c7372944866fe8e142f2c6c4b3a42302
id: 857631584890912768  rev: 1-9fc57b67e5a4b86257fbb5ec5e2e6223
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/ubuntu/goal/harvester/cli.py", line 36, in <module>
    Harvester().start()
  File "/home/ubuntu/goal/harvester/cli.py", line 32, in start
    self.stream.filter(locations=config.locations, track=config.track)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/tweepy/streaming.py", line 445, in filter
    self._start(async)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/tweepy/streaming.py", line 361, in _start
    self._run()
  File "/home/ubuntu/.local/lib/python3.5/site-packages/tweepy/streaming.py", line 294, in _run
    raise exception
  File "/home/ubuntu/.local/lib/python3.5/site-packages/tweepy/streaming.py", line 263, in _run
    self._read_loop(resp)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/tweepy/streaming.py", line 313, in _read_loop
    line = buf.read_line().strip()
AttributeError: 'NoneType' object has no attribute 'strip'
  • Press Enter and you will get back to shell foreground.
[1]+  Exit 1                  python3 -m harvester.cli
ubuntu@r-d3i1sr7z-0:~/goal$
  • Copy the error and create issue ticket, e.g. like this issue.
  • Then start the Harvester again meanwhile fixing the issue.
ubuntu@r-d3i1sr7z-0:~/goal$ python3 -m harvester.cli &
[1] 19405
ubuntu@r-d3i1sr7z-0:~/goal$ Starting tweet stream harvesting. Press Ctrl+C to stop.
id: 858157020909674496  rev: 1-b95e5cacaa2f9d0d0c5e918ae2e8ba50
id: 858157020133728256  rev: 1-cd53c310b47184a5ac943e261b595630
...

Detach from Screen

  • If you are in screen and want to get out of it, this is called detach from screen.
  • Press Ctrl + a then d 
[detached from 16844.pts-0.r-d3i1sr7z-0]
ubuntu@r-d3i1sr7z-0:~$

Already Attached Screen

  • If someone is already attached into screen, you will see like this.
ubuntu@r-d3i1sr7z-0:~$ screen -ls
There is a screen on:
        16844.pts-0.r-d3i1sr7z-0        (27/04/17 23:35:01)     (Attached)
1 Socket in /var/run/screen/S-ubuntu.
  • If you attempt to attach this screen, you can't.
ubuntu@r-d3i1sr7z-0:~$ screen -r
There is a screen on:
        16844.pts-0.r-d3i1sr7z-0        (27/04/17 23:35:01)     (Attached)
There is no screen to be resumed.
Clone this wiki locally