Implement C-Go interface for HORNET integration

[marktwtn]

HORNET is a lightweight alternative to IOTA's fullnode software IRI written in Go language.
In order to make it use the PoW acceleration of dcurl, the interface between C and Go language should be implemented.

For the interface implementation, please reference: cgo.

[marktwtn]

A simple Go program calling the dcurl APIs would look like this:

package main

// #cgo LDFLAGS: -L ./build -ldcurl
/*
#include <stdbool.h>
extern bool dcurl_init();
extern bool dcurl_destroy();
*/
import "C"

func main() {
  C.dcurl_init()
  C.dcurl_destroy()
}

To execute the program, we need to specify the LD_LIBRARY_PATH value with the path of libdcurl.so.

[marktwtn]

HORNET is composed with multiple plugins.
To integrate dcurl into HORNET, we do not have to write a new plugin but to modify the existed one.
That is modifying the pow.go file of the webapi plugin.

---

IMPORTANT:

However, the benefit of the integration would be based on the PoW acceleration of FPGA and remote worker mostly.
The reason is the HORNET has multi-thread and SIMD PoW acceleration already.

The PoW function of webapi plugin calls the Go client library iota.go.
webapi plugin uses goroutine(which is like thread) and calls the CPU acceleration PoW function with SIMD in iota.go.
SIMD acceleration

[Fangop]

• The function used for pow is returned by iota.go/api/api.go/ProofOfWorkFunc doc

• The ProofOfWorkFunc returns a proper pow for environment.

  • from iota.go/pow

• 2 ways approach to integrate dcurl with HORNET

  • Integrate with iota.go

    • 

  • Solely integrate with HORNET, as the way @marktwtn provides above.

    • 

How to implement?

• 2 files required:

  • pow_dcurl.go

    • func init()
      • May be more complicated due to remote worker?
    • func syncProofOFfwork
      • SyncProofOfWork is like ProofOfWork() but only runs one ongoing Proof-of-Work task at a time.
      • mutex
    • func ProofOfWork

  • pow_ducrl_test.go

[Fangop]

package main

/*
 #cgo LDFLAGS: -L ./build -ldcurl
#include <string.h>
#include <stdlib.h>
#include <stdbool.h>
#include <stdint.h>
extern bool dcurl_init();
extern bool dcurl_destroy();
extern int8_t *dcurl_entry(int8_t *trytes, int mwm, int threads);

char*dcurl_wrapper(void* trytes, int mwm, int threads ){
	return (char*) dcurl_entry((int8_t*) trytes, mwm, threads);
}
*/
import "C"
import (
	"fmt"
	"sync"
	"unsafe"

	"github.com/iotaledger/iota.go/curl"
	"github.com/iotaledger/iota.go/guards"
	. "github.com/iotaledger/iota.go/trinary"
	"github.com/pkg/errors"

	. "github.com/iotaledger/iota.go/consts"
	. "github.com/onsi/ginkgo"
	. "github.com/onsi/gomega"
)

func init() {
}

func DCURLProofOfWork(trytes Trytes, mwm int, parallelism ...int) (Trytes, error) {
	C.dcurl_init()
	defer C.dcurl_destroy()

	return dcurlProofOfWork(trytes, mwm, parallelism...)
}

var DCURLProofOfWorkMutex = sync.Mutex{}

func SyncDCURLProofOfWork(trytes Trytes, mwm int, parallelism ...int) (Trytes, error) {
	C.dcurl_init()
	defer C.dcurl_destroy()

	DCURLProofOfWorkMutex.Lock()
	defer DCURLProofOfWorkMutex.Unlock()
	nonce, err := dcurlProofOfWork(trytes, mwm, parallelism...)
	if err != nil {
		return "", err
	}
	return nonce, nil
}

func dcurlProofOfWork(trytes Trytes, mwm int, parallelism ...int) (Trytes, error) {
	if trytes == "" {
		return "", errors.New("invalid trytes supplied to Proof-of-Work func")
	}

	tr := unsafe.Pointer(&MustTrytesToTrits(trytes)[0])
	defer C.free(tr)
	var numThread int
	if len(parallelism) != 0 && parallelism[0] > 0 {
		numThread = parallelism[0]
	} else {
		numThread = 1
	}
	result := C.dcurl_wrapper(tr, C.int(mwm), C.int(numThread))

	return C.GoString(result), nil
}

func main() {
	var _ = Describe("PoWC", func() {
		Context("PoWC()", func() {
			rawTx := "999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999A9RGRKVGWMWMKOLVMDFWJUHNUNYWZTJADGGPZGXNLERLXYWJE9WQHWWBMCPZMVVMJUMWWBLZLNMLDCGDJ999999999999999999999999999999999999999999999999999999YGYQIVD99999999999999999999TXEFLKNPJRBYZPORHZU9CEMFIFVVQBUSTDGSJCZMBTZCDTTJVUFPTCCVHHORPMGCURKTH9VGJIXUQJVHK999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999"
			It("computes a correct nonce", func() {
				nonce, err := DCURLProofOfWork(rawTx, 10)
				rawTx = rawTx[:len(rawTx)-NonceTrinarySize/3] + nonce
				Expect(err).ToNot(HaveOccurred())
				hashedTrytes := curl.MustHashTrytes(rawTx)
				Expect(guards.IsTransactionHashWithMWM(hashedTrytes, 10)).To(BeTrue())
			})
		})
	})
}

This is an update to C-Go interface of dcurl.
The usage of main function is to do the unit test provide by iota.go

• Building command, assume this file has name pow_dcurl.go in directory of dcurl.

build your dcurl first
go build pow_dcurl.go
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:./build

It'll be split to 2 part: pow_dcurl.go and pow_dcurl_test.go.

• I need help for creating a robust pow_dcurl_test.go
• I need help for test this can run correctly on fpga.

[marktwtn]

The test is running correctly on the DE10-Nano.

However, there are several things need to be mentioned.

• The environment variable LIBRARY_PATH and LD_LIBRARY_PATH should be set correctly.
  They should be added with the path where the libdcurl.so is located.
• To enable the FPGA acceleration, the root privilege is required. (Ref: Privilege problem of device files on de10-nano board #184)
  But using the testing command $ sudo go test or sudo -s go test would cause the error message

  /usr/local/go/pkg/tool/linux_arm/link: running gcc failed: exit status 1
  /usr/bin/ld: cannot find -ldcurl
  collect2: error: ld returned 1 exit status
  

  However, using the testing command like

  $ sudo -s
  $ go test
  

  would have no problem at all.
  The root behavior should have further investigation.
• To use sudo with go, please reference the solution.

[marktwtn]

In order to run the testing on DE10-Nano with FPGA enabled, please use the instruction:

$ sudo env LIBRARY_PATH=[path_of_libdcurl.so] LD_LIBRARY_PATH=[path_of_libdcurl.so] go test

The reasons are:

1. Using FPGA acceleration requires root privilege.
2. Using sudo would make the required environment variables back to their original value, hence the env command is necessary to assign the value to environment variables.

@JulianaTa please make sure the root privilege problem is recorded in the document.