clp-ffi-go

This module provides Go packages to interface with CLP's core features through CLP's FFI (foreign function interface). For complete technical documentation, see the Go docs: https://pkg.go.dev/github.com/y-scope/clp-ffi-go

Getting started

To add the module to your project run: go get github.com/y-scope/clp-ffi-go

Here's an example showing how to decode each log event containing "ERROR" from a CLP IR byte stream.

import (
  "fmt"
  "time"

  "github.com/klauspost/compress/zstd"
  "github.com/y-scope/clp-ffi-go/ffi"
  "github.com/y-scope/clp-ffi-go/ir"
)

file, _ := os.Open("log-file.clp.zst")
defer file.Close()
zstdReader, _ := zstd.NewReader(file)
defer zstdReader.Close()
irReader, _ := ir.NewReader(zstdReader)
defer irReader.Close()

var err error
for {
  var log *ffi.LogEventView
  // To read every log event replace ReadToContains with Read()
  log, err = irReader.ReadToContains("ERROR")
  if nil != err {
    break
  }
  fmt.Printf("%v %v", time.UnixMilli(int64(log.Timestamp)), log.LogMessageView)
}
if ir.EndOfIr != err {
  fmt.Printf("Reader.Read failed: %v", err)
}

Building

We use the go generate command to build the C++ interface to CLP's FFI code as well as stringify Enum style types.

1. Install requirements:
   1. A C++ compiler that supports C++20
   2. CMake 3.23 or higher
   3. The Stringer tool: https://pkg.go.dev/golang.org/x/tools/cmd/stringer
      • go install golang.org/x/tools/cmd/stringer@latest
2. go generate ./...
   • Run all generate directives (note the 3 dots after '/')

Bazel support

We provide a Bazel module and build files for each Go package in the repository. Additionally, we provide a module extension for the FFI core component of CLP necessary to build the native library.

The following is an example to pull in the ir Go package as a dependency through Bazel. For development and testing it may be useful to use the commented local_path_override snippet instead of the git_override code.

# Add to MODULE.bazel

bazel_dep(name = "com_github_y_scope_clp_ffi_go", version = "<version>")
_com_github_y_scope_clp_ffi_go_commit = "<commit hash>"
archive_override(
    module_name = "com_github_y_scope_clp_ffi_go",
    integrity = "sha256/512-<base 64 sha of commit>",
    urls = [
        "https://github.com/y-scope/clp-ffi-go/archive/{}.zip".format(
            _com_github_y_scope_clp_ffi_go_commit
        ),
    ],
    strip_prefix = "clp-ffi-go-{}".format(_com_github_y_scope_clp_ffi_go_commit),
)
# Use as an alternative to archive_override for local development
# local_path_override(
#     module_name = "com_github_y_scope_clp_ffi_go",
#     path = "/home/user/clp-ffi-go",
# )
clp_ffi_go_ext_deps = use_extension(
    "@com_github_y_scope_clp_ffi_go//cpp:deps.bzl", "clp_ffi_go_ext_deps"
)
use_repo(clp_ffi_go_ext_deps, "com_github_y_scope_clp")

# Add the ffi-go package as a dependency in a BUILD.bazel file

go_binary(
    name = "example",
    srcs = ["example.go"],
    visibility = ["//visibility:public"],
    deps = ["@com_github_y_scope_clp_ffi_go//ir"],
)

Why not build with cgo?

The primary reason we choose to build with CMake rather than directly with cgo, is to ease code maintenance by maximizing the reuse of CLP's code with no modifications. If a platform you use is not supported by the pre-built libraries, please open an issue and we can integrate it into our build process.

Testing

To run all unit tests run: go_test_ir="/path/to/my-ir.clp.zst" go test ./...

• Some of the ir package's tests currently require an existing CLP IR file compressed with zstd. This file's path is taken as an environment variable named go_test_ir. It can be an absolute path or a path relative to the ir directory.

Linting

1. Install golangci-lint:

curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | \
  sh -s -- -b $(go env GOPATH)/bin v1.59.0

2. Run with golangci-lint run

Using an external C++ library

Use the external build tag to link with different CLP FFI library instead of the pre-built ones found in lib. This tag only prevents the linking of the pre-built libraries and does nothing else. It is up to the user to use CGO_LDFLAGS to point to their library. You may also need to update CGO_CFLAGS to update the header include path.

For example, to run the tests using the external you can run:

CGO_LDFLAGS="-L/path/to/external_libs -lclp_ffi_linux_amd64 -Wl,-rpath=/path/to/external_libs" \
  go_test_ir="/path/to/my-ir.clp.zst" \
  go test -tags external ./...