New POW calculation module #1284

Kleshni · 2018-06-23T09:01:18Z

For #1227. It's currently not connected to the PyBitmessage code, I'm going to get it done sooner or later.

It provides the WorkProver thread class which manages tasks and schedules them to solvers. There are 4 possible solvers:

Single-threaded DumbSolver in Python. I optimised it so it now works ~2 times faster. In particular, I replaced hashlib.sha512 with SHA512 from OpenSSL. Since you need OpenSSL to assemble a message anyway, I think it's safe to use it in a fallback solver.
Multiprocess based ForkingSolver in Python. It works faster too, because it relies on DumbSolver.
Multithreaded FastSolver in C. I tried to utilize SIMD, but it gave very little speed up of 7 %, so it still calls OpenSSL.
And GPUSolver in OpenCL.

The library was tested on a 4-core AMD64 Linux with integrated GPU and 45-CPU virtual machines with hyperthreading enabled:

X86 Linux;
AMD64 FreeBSD;
AMD64 OpenBSD with the bsd.mp kernel;
AMD64 NetBSD;
AMD64 Windows 7.

I failed to install Hackintosh on QEMU, so somebody else has to test it there.

PeterSurda · 2018-06-23T09:05:03Z

Have you tried frozen mode on Windows?

Kleshni · 2018-06-23T09:31:28Z

No, but ForkingSolver is disabled in this case complying to the current behaviour. All file paths (to the C library and OpenCL kernel) are relative to the code directory, whose path must be provided by the calling code.

omkar1117 · 2018-06-23T09:17:58Z

src/workprover/Readme.md

@@ -0,0 +1,42 @@
+Please keep this module independent from the outside code, so that it can be reused in other applications.


In that case why can't we make it a separate package and import it where required ?

It's a Python package with the __init__.py file, and it's intended to be imported like import workprover. It could be moved to a separate repository, but I think, it's easier and safer to keep it in the same file tree.

omkar1117 · 2018-06-23T09:19:26Z

src/workprover/__init__.py

+        try:
+            self.availableSolvers["fast"] = fastsolver.FastSolver(codePath)
+        except:
+            pass


pass is the dangerous statement, please log it or please write a print statement atleast...

self.availableSolvers is visible to the outside code, so it can log or print if fast is missing. It should show a message in GUI status bar like the current code does.

omkar1117 · 2018-06-23T09:23:40Z

src/workprover/__init__.py

+        if self.statusUpdated is None:
+            return
+
+        if self.solver is None:


parallelism = self.solver.parallelism if self.solver else 0

It's less readable.

omkar1117 · 2018-06-23T09:26:47Z

src/workprover/__init__.py

+        self.currentTaskID = None
+
+    def notifyStatus(self):
+        if self.statusUpdated is None:


if not self.statusUpdated:
return

It's less acurate.

self.statusUpdated must be either a function or a None. If it's 0 or an empty string, it would be a programming error and further call would rise an exception making the error noticable.

omkar1117 · 2018-06-23T09:27:44Z

src/workprover/__init__.py

+        self.statusUpdated((self.solverName, parallelism, self.speed))
+
+    def setSolver(self, name, parallelism):
+        if name is None and self.solverName is None:


if not name and not self.solverName

omkar1117 · 2018-06-23T09:58:58Z

src/workprover/fastsolver.py

+import ctypes
+
+def loadFastSolver(codePath):
+    if hasattr(sys, "winver"):


doc type required here

omkar1117 · 2018-06-23T09:59:12Z

src/workprover/fastsolver.py

+
+        if platform.architecture()[0] == "64bit":
+            suffix = "-64"
+


pep8 validation

omkar1117 · 2018-06-23T09:59:54Z

src/workprover/fastsolver.py

+        self.parallelism = 0
+
+    def search(self, initialHash, target, seed, timeout):
+        found = self.libfastsolver.fastsolver_search(


documentation required here

omkar1117 · 2018-06-23T10:02:19Z

src/workprover/forkingsolver.py

@@ -0,0 +1,106 @@
+import os


pep8 validation for the file

omkar1117 · 2018-06-23T10:07:47Z

src/workprover/forkingsolver.py

+def setIdle():
+    try:
+        import psutil
+


remove un necessary spaces

g1itch

Seems to work for me: two "singleWorker" threads appear when doing PoW. Oh, I realized it's not connected so far :(

g1itch · 2018-06-23T12:25:37Z

@Kleshni workprover.test also contains tabs

g1itch · 2018-06-23T12:48:01Z

Some tests are failing. It seems to be expected for TestGPUSolver, what about the rest?

Kleshni · 2018-06-23T13:26:38Z

TestSolver is not a test case, it's a base class for other test cases. Now it should be skipped during automatic test discovery.

TestFastSolver fails loading compiled libfastsolver.so for the reason I don't know. A print in the corresponding except could clarify this.

g1itch · 2018-06-23T15:09:41Z

Strange thing:

OSError: /home/travis/build/g1itch/PyBitmessage/src/workprover/fastsolver/libfastsolver.so: undefined symbol: SHA512_Update

It seems to be specific to ubuntu:trusty or Travis-CI.

Nevertheless could you please add pybitmessage.workprover to packages and decorate TestGPUSolver like I did?

Kleshni · 2018-06-25T08:04:14Z

I have registered in Travis-CI and debugged the issue. It was a very stupid mistake, shared libraries like -lpthread and -lcrypto should be listed after object files in the linker command line.

Travis-CI also gives an OS X environment, so I tested the module there.

PeterSurda · 2018-06-27T09:42:20Z

packages/pyinstaller/bitmessagemain.spec needs updating.

g1itch · 2018-06-27T09:47:35Z

src/workprover/gpusolver.py

+
+        try:
+            import numpy
+            import pyopencl


Could you please comment #1283 and maybe change the import order too if you agree?

omkar1117 · 2018-07-25T03:55:47Z

packages/pyinstaller/bitmessagemain.spec

@@ -54,6 +54,8 @@ else:
 a.binaries += [('libeay32.dll', openSSLPath + 'libeay32.dll', 'BINARY'),
 	(os.path.join('bitmsghash', 'bitmsghash%i.dll' % (arch)), os.path.join(srcPath, 'bitmsghash', 'bitmsghash%i.dll' % (arch)), 'BINARY'),
 	(os.path.join('bitmsghash', 'bitmsghash.cl'), os.path.join(srcPath, 'bitmsghash', 'bitmsghash.cl'), 'BINARY'),
+	("workprover/fastsolver/libfastsolver-{}.dll".format(arch), os.path.join(srcPath, "workprover/fastsolver/libfastsolver-{}.dll".format(arch)), "BINARY"),


PEP8 validation required here.

omkar1117 · 2018-07-25T04:02:59Z

src/api.py

+
+# This thread, of which there is only one, runs the API.
+class singleAPI(threading.Thread, helper_threading.StoppableThread):
+    def __init__(self):


please add doc string to the class.

omkar1117 · 2018-07-25T04:03:47Z

src/api.py

+        self.initStop()
+
+    def stopThread(self):
+        super(singleAPI, self).stopThread()


please add doc string to the class

omkar1117 · 2018-07-25T04:04:41Z

src/bitmessagemain.py

@@ -502,10 +384,85 @@ def main():
    mainprogram = Main()
    mainprogram.start()

+# See "workprover/Readme.md"
+
+import os


please write all the imports on the starting of the file.

omkar1117 · 2018-07-25T04:05:06Z

src/bitmessagemain.py


 if __name__ == "__main__":
-    main()
+    import multiprocessing


Please place all the imports in the starting the file...

These imports must be conditional.

omkar1117 · 2018-07-25T04:06:34Z

src/class_sqlThread.py

+
+        # Update status values inherited from old versions
+
+        self.cur.execute("""UPDATE "sent" SET "status" = 'msgsent' WHERE "status" == 'sentmessage';""")


pep8 validation here.

omkar1117 · 2018-07-25T04:07:41Z

src/workprover/__init__.py

@@ -0,0 +1,253 @@
+import Queue


please place all the imports in a alphabetic order.

omkar1117 · 2018-07-25T04:08:09Z

src/workprover/__init__.py

+        self.statusUpdated(Status(self.solverName, status, self.speed, len(self.tasks), self.totalDifficulty))
+
+    def setSolver(self, name, configuration):
+        if name is None and self.solverName is None:


if not name and not self.solverName:
pass

omkar1117 · 2018-07-25T04:08:25Z

src/workprover/__init__.py

+        elif name == self.solverName:
+            self.solver.setConfiguration(configuration)
+        else:
+            if self.solver is not None:


if self.solver:
….

omkar1117 · 2018-07-25T04:08:36Z

src/workprover/__init__.py

+                self.solverName = None
+                self.solver = None
+
+            if name is not None:


if name:
….

Kleshni · 2018-07-29T11:24:00Z

Added API methods for dealing with raw objects in the inventory, solving #1225 and superseding #1226.

`disseminateRawObject`

Argument: a hex-encoded object without packet headers, starting from nonce.

Tries to send the object to the network. POW must be already calculated.

Returns an error or the object's inventory hash.

`getRawObject`

Argument: an inventory hash.

Returns an error or a JSON object with the following fields:

hash - the inventory hash;
expiryTime - a UNIX timestamp;
objectType - an integer;
stream;
tag - hex-encoded object tag;
payload - hex-encoded object without packet headers, starting from nonce.

`listRawObjects`

Parameters: the desired object type or None, stream number or None, hex-encoded tag or None.

Returns a list of hex-encoded hashes.

`queueRawObject`

Arguments: TTL in seconds and a hex-encoded object without packet headers, nonce and expiry time, starting from object type.

Queues the object for POW calculation and sending to the network.

Returns a unique handle to track the objects' state.

`cancelQueuedRawObject`

Argument: a handle returned from the queueRawObject method.

Tries to cancel a previously queued object.

`checkQueuedRawObject`

Argument: a handle returned from the queueRawObject method.

Returns an array with the first element being the current status of the object:

queued;
doingwork;
failed for invalid objects;
sent. In this case the second item of the array is the hex-encoded inventory hash;
canceled;
notfound for wrong handles and for handles that previously returned failed, sent or canceled.

If a queued object enters the notfound state, it can mean that the daemon was restarted, because the queued objects are not saved to the disk.

Kleshni · 2018-07-29T11:47:44Z

I deleted the getMessageDataByDestination{Hash,Tag} methods because they were undocumented and didn't work at all, so I concluded that no-one ever used them. They are now replaced by getRawObject and listRawObjects.

The same for disseminatePreEncryptedMsg and disseminatePubkey.

g1itch · 2018-08-02T13:52:45Z

src/workprover/__init__.py

+                    self.solverName = name
+                    self.solver = self.availableSolvers[name]
+                    self.solver.setConfiguration(configuration)
+        except GPUSolverError:


undefined name 'GPUSolverError'

g1itch · 2018-08-02T14:00:14Z

I got 3 additional threads (workprover.WorkProver instances) which don't stop when I run src/pybitmessage.py -t:
https://travis-ci.org/g1itch/PyBitmessage/builds/411231409

This patch helped:

diff --git a/src/singleworker.py b/src/singleworker.py
index 9fa1391c..e1eeda3f 100644
--- a/src/singleworker.py
+++ b/src/singleworker.py
@@ -270,17 +270,19 @@ def setBestSolver():
 
 setBestSolver()
 
+
 class singleWorker(threading.Thread, helper_threading.StoppableThread):
     name = "singleWorker"
 
     def __init__(self):
-        super(self.__class__, self).__init__()
+        super(self.__class__, self).__init__(name="singleWorker")
 
         self.initStop()
 
     def stopThread(self):
         queues.workerQueue.put(("stopThread", "data"))
 
+        workProver.commandsQueue.put(("shutdown", ))
         super(self.__class__, self).stopThread()
 
     def run(self):

Kleshni · 2018-09-17T11:46:39Z

Any plans on merging?

dimyme · 2018-09-20T11:21:57Z

your POW branch works nicely here ! great job !

PeterSurda · 2018-09-28T09:41:42Z

@Kleshni I haven't looked at it yet, but it first needs to pass all the code quality checks and all commits need to be signed (you can squash commits if it helps).

However, others gave good feedback so chances are good.

PeterSurda · 2018-09-28T09:45:28Z

Also, please split this into multiple patches. The PoW, the API and the other changes should be separate. You'll have better chances of getting it merged that way too.

PeterSurda · 2019-04-08T09:25:07Z

@Kleshni would it be ok if I assigned someone else to the task to help you with cleaning it up? You'd have to give them write access to your repo.

Kleshni · 2019-04-24T05:19:14Z

OK.

PeterSurda · 2019-04-24T06:53:25Z

@Kleshni please give @omkar1117 write access to the POW branch. this way your commits will be preserved and you can still get paid through tip4commit.

Kleshni · 2019-04-25T08:00:00Z

I have sent him an invite yesterday but forgot to notify you 👀

omkar1117 · 2019-05-08T17:10:07Z

@Kleshni could you please resolve the conflicts please.

PeterSurda · 2019-06-08T06:46:13Z

@Kleshni Omkar says he doesn't have write access. He could fork it into his own repo, but since this PR is already open I would prefer to continue this way. Can you check and let me know?

Kleshni · 2019-06-11T08:30:41Z

This is what I see in the repository settings:

New POW calculation module

a181da3

PeterSurda requested review from PeterSurda, g1itch, omkar1117 and coffeedogs June 23, 2018 09:04

PeterSurda added enhancement New feature refactoring labels Jun 23, 2018

omkar1117 suggested changes Jun 23, 2018

View reviewed changes

Kleshni added 3 commits June 23, 2018 10:32

More precise exceptions

079326b

Fixed indentation

46da7e8

Sorted imports

917b55d

g1itch reviewed Jun 23, 2018

View reviewed changes

Fixed indentation

2eeb58a

Improved unit tests loading

43f7e1e

Kleshni added 4 commits June 23, 2018 15:53

Skip OpenCL test if unavailable

30e7df6

Add "workprover" to packages list

721ad8c

More build system integration for WorkProver

4f9274a

OS X compatibility

cac0237

g1itch reviewed Jun 27, 2018

View reviewed changes

Kleshni added 3 commits June 27, 2018 10:16

Merge branch 'v0.6' into POW

4c0b0c7

Downgrade OpenSSL version

7999330

Added WorkProver to executable builders

90ae95d

Added estimated POW time tooltip

2c7d677

omkar1117 suggested changes Jul 25, 2018

View reviewed changes

Kleshni added 3 commits July 29, 2018 04:38

Added POW settings to settings window

373157d

Added API for raw objects

00c4ee8

Added more API for raw objects

02ece2d

Removed broken API methods

c22c5c2

Kleshni added 4 commits July 29, 2018 12:06

Deleted old module

a398354

More old code deleted

8e1259d

Fixed big-endian decoding

0aaad8b

Readability

192b083

g1itch reviewed Aug 2, 2018

View reviewed changes

Applying solution I don't understand

7ad1725

PeterSurda assigned omkar1117 Jun 12, 2019

g1itch mentioned this pull request Jul 30, 2019

UI refactoring, part 1.5: Settings #1495

Merged

PeterSurda unassigned omkar1117 Mar 9, 2020

PeterSurda removed the request for review from coffeedogs March 9, 2020 07:02

		@@ -0,0 +1,42 @@
		Please keep this module independent from the outside code, so that it can be reused in other applications.


		# Update status values inherited from old versions

		self.cur.execute("""UPDATE "sent" SET "status" = 'msgsent' WHERE "status" == 'sentmessage';""")

New POW calculation module #1284

Are you sure you want to change the base?

New POW calculation module #1284

Uh oh!

Conversation

Kleshni commented Jun 23, 2018

Uh oh!

PeterSurda commented Jun 23, 2018

Uh oh!

Kleshni commented Jun 23, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

g1itch left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

g1itch commented Jun 23, 2018

Uh oh!

g1itch commented Jun 23, 2018

Uh oh!

Kleshni commented Jun 23, 2018

Uh oh!

g1itch commented Jun 23, 2018

Uh oh!

Kleshni commented Jun 25, 2018

Uh oh!

PeterSurda commented Jun 27, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Kleshni commented Jul 29, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

g1itch left a comment •

edited

Loading

Kleshni commented Jul 29, 2018 •

edited

Loading

`disseminateRawObject`

`getRawObject`

`listRawObjects`

`queueRawObject`

`cancelQueuedRawObject`

`checkQueuedRawObject`

omkar1117 commented May 8, 2019 •

edited

Loading