Skip to content

Commit

Permalink
Add support for bulk operations (#42)
Browse files Browse the repository at this point in the history
* Add support for bulk operations

* Fix mktemp

* Fixup tests

* Fix dirty working dir check

* Add missing file (for dirty working dir check)

* Fix dirty working dir check

* Update readme with bulk docs
  • Loading branch information
myshkin5 authored Jul 27, 2022
1 parent d8216fa commit 767ac61
Show file tree
Hide file tree
Showing 38 changed files with 6,848 additions and 299 deletions.
11 changes: 11 additions & 0 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,19 @@ jobs:

command: |
set -ux
find . -name moq_\*.go -delete
go build -o $GOPATH/bin/moqueries github.com/myshkin5/moqueries
export MOQ_BULK_STATE_FILE=$(mktemp --tmpdir= moq-XXXXXX)
moqueries bulk-initialize
go generate ./...
MOQ_DEBUG=true moqueries bulk-finalize
# fail if working directory is dirty
git status --short
if [[ -n $(git status --short) ]]; then
echo "Working directory dirty"
exit 1
fi
- run:
name: Run tests
Expand Down
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -26,4 +26,3 @@ moq_*_dst.txt
# other
.envrc
/.idea
/.go
18 changes: 10 additions & 8 deletions .golangci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,18 +17,22 @@ linters:
- maintidx
- maligned
- nlreturn
- nosnakecase
- paralleltest
- prealloc
- predeclared
- varnamelen
- scopelint
- tagliatelle
- varnamelen
- wsl

# Consider enabling
- gocyclo
- golint
- wrapcheck

# Usually disabled but useful for checking everything has godoc
- golint

linters-settings:
gci:
sections:
Expand All @@ -47,6 +51,7 @@ linters-settings:
- "-ID"

issues:
exclude-use-default: false
exclude-rules:
- path: '(.+)_test.go'
linters:
Expand All @@ -56,9 +61,6 @@ issues:
- linters:
- lll
source: "^//go:generate "
# Inline nolint directive isn't working (CI/CD fails on nolintlint)
- linters:
- thelper
# This is the actual test so not a helper
path: 'generator/testmoqs/testmoqs_test.go'
source: "testOptionalSuccess := func"
include:
# disable excluding of issues about comments from golint.
- EXC0002
44 changes: 38 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -249,6 +249,22 @@ Quite often tests will require several mocks. A `Scene` is a collection of mocks
scene.AssertExpectationsMet()
```

## Bulk generation
Generating mock can be CPU intensive! Additionally, Moqueries only knows the package where to look for a type so the entire package has to be parsed. And to top it off, you will quite often mock several types from the same package. To avoid re-parsing the same package repeatedly, Moqueries has a bulk mode that can best be described by these three steps:
1. Initialize the bulk processing file
2. `go generate ./...`
3. Finalize the bulk processing (and generate all the mocks)

Moqueries [CI/CD pipeline](.circleci/config.yml) accomplishes this with the following few commands:
```shell
export MOQ_BULK_STATE_FILE=$(mktemp --tmpdir= moq-XXXXXX)
moqueries bulk-initialize
go generate ./...
moqueries bulk-finalize
```

The first line creates a new temporary file to hold the state. The second line initializes the file (holds on to some global attributes to ensure consistency). The third line is the standard `go generate` line but because `MOQ_BULK_STATE_FILE` is defined, it only records the intent to generate a new mock. The forth and final line is where the work actually occurs, and it might take some time depending on how many mocks you want to generate. See more details below in the [Command line reference](#command-line-reference).

## More command line options
Below is a loose collection of out-of-the-ordinary command line options for use in out-of-the-ordinary situations.

Expand Down Expand Up @@ -285,8 +301,8 @@ writerMoq.OnCall().Write([]byte("3")).ReturnResults(0, fmt.Errorf("couldn't writ
## Command line reference
The Moqueries command line has the following form:

```bash
$ moqueries [options] [interfaces and/or function types to mock] [options]
```shell
moqueries [options] [interfaces and/or function types to mock] [options]
```

Interfaces and function types are separated by whitespace. Multiple types may be specified.
Expand All @@ -297,6 +313,7 @@ Interfaces and function types are separated by whitespace. Multiple types may be
| `--destination <file>` | `string` | `./moq_<type>.go` when exported or `./moq_<type>_test.go` when not exported | The file path where mocks are generated relative to directory containing generate directive (or relative to the current directory) |
| `--destination-dir <dir>` | `string` | `.` | The file directory where mocks are generated relative to the directory containing the generate directive (or relative to the current directory) |
| `--export` | `bool` | `false` | If true, generated mocks will be exported and accessible from other packages |
| `-h` or `--help` | `bool` | `false` | Display command help |
| `--import <name>` | `string` | `.` (the directory containing generate directive) | The package containing the type (interface or function type) to be mocked |
| `--package <name>` | `string` | The test package of the destination directory when `--export=false` or the package of the destination directory when `--export=true` | The package to generate code into |
| `--test-import` | `bool` | `false` | Indicates that the types are defined in the test package |
Expand All @@ -310,19 +327,34 @@ Options with a value type of `bool` are set (turned on) by specifying the option
### Environment Variables
The Moqueries command line can also be controlled by the following environment variables:

| Name | Usage |
|-------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `MOQ_DEBUG` | If set to a "true" value (see [`strconv.ParseBool`](https://pkg.go.dev/strconv#ParseBool)), debugging output will be logged (also see `--debug` in [Command line reference](#command-line-reference) above) |
| Name | Usage |
|-----------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `MOQ_BULK_STATE_FILE` | If set, defines the bulk state file in which generate requests will be stored for bulk generation. See [Bulk generation](#bulk-generation) above. |
| `MOQ_DEBUG` | If set to a "true" value (see [`strconv.ParseBool`](https://pkg.go.dev/strconv#ParseBool)), debugging output will be logged (also see `--debug` in [Command line reference](#command-line-reference) above) |

### Subcommands

#### Default
The default subcommand generates one or more mocks based on the command specified. As described [above](#generating-mocks), this is typically invoked by a `go:generate` directive. The default subcommand is invoked when no subcommand is specified.

If the `MOQ_BULK_STATE_FILE` environment variable is defined (see [above](#environment-variables)), the default subcommand does not immediately generate the mocks, but instead appends the generate request to the state file. See [Bulk generation](#bulk-generation) above.

#### Bulk initialize
Initializes the bulk state file defined by the `MOQ_BULK_STATE_FILE` environment variable. `MOQ_BULK_STATE_FILE` is required. Note that the bulk state file is overwritten if it exists.
```shell
moqueries bulk-initialize
```

#### Bulk finalize
Finalizes bulk processing by generating multiple mocks at once. The `MOQ_BULK_STATE_FILE` environment variable is required and specifies which mocks to generate.
```shell
moqueries bulk-finalize
```

#### Summarize metrics
The `summarize-metrics` subcommand takes the debug logs from multiple generate runs (using the [default](#default) subcommand), reads metrics from each individual run, and outputs summary metrics. This subcommand takes a single, optional argument specifying the log file to read. If no file is specified or if the file is specified as `-', standard in is read.
The following command generates all mocks specified in `go:generate` directives and summarizes the metrics for all runs:
```shell
$ MOQ_DEBUG=true go generate ./... | moqueries summarize-metrics
MOQ_DEBUG=true go generate ./... | moqueries summarize-metrics
```
3 changes: 2 additions & 1 deletion ast/cache.go
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
// Package ast provides utilities for working with Go's Abstract Syntax Tree
package ast

import (
Expand Down Expand Up @@ -82,7 +83,7 @@ func (c *Cache) Type(id dst.Ident, testImport bool) (*dst.TypeSpec, string, erro
typ, ok := c.typesByIdent[realId]
if !ok {
return nil, "", fmt.Errorf(
"%q (original package %q): %w", realId, id.Path, ErrTypeNotFound)
"%w: %q (original package %q)", ErrTypeNotFound, realId, id.Path)
}

return typ, pkgPath, nil
Expand Down
1 change: 0 additions & 1 deletion ast/testpkg/testpkg.go

This file was deleted.

41 changes: 41 additions & 0 deletions bulk/bulk.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
// Package bulk is used to generate several Moqueries mocks at once
package bulk

import (
"io"
"os"

"github.com/myshkin5/moqueries/bulk/internal"
"github.com/myshkin5/moqueries/generator"
)

// Initialize initializes bulk processing and creates the bulk processing state
// file
func Initialize(stateFile, rootDir string) error {
createFn := func(name string) (io.WriteCloser, error) {
//nolint:gosec // Users can use any file for bulk operations
return os.Create(name)
}

return internal.Initialize(stateFile, rootDir, createFn)
}

// Append appends a mock generate request to the bulk state
func Append(stateFile string, request generator.GenerateRequest) error {
openFileFn := func(name string, flag int, perm os.FileMode) (internal.ReadWriteSeekCloser, error) {
//nolint:gosec // Users can use any file for bulk operations
return os.OpenFile(name, flag, perm)
}

return internal.Append(stateFile, request, openFileFn)
}

// Finalize complete bulk processing by generating all the requested mocks
func Finalize(stateFile, rootDir string) error {
openFn := func(name string) (io.ReadCloser, error) {
//nolint:gosec // Users can use any file for bulk operations
return os.Open(name)
}

return internal.Finalize(stateFile, rootDir, openFn, generator.Generate)
}
135 changes: 135 additions & 0 deletions bulk/internal/appender.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
package internal

import (
"bufio"
"encoding/json"
"errors"
"fmt"
"go/build"
"io"
"os"
"path"
"path/filepath"
"strings"

"github.com/myshkin5/moqueries/generator"
"github.com/myshkin5/moqueries/logs"
)

var (
// ErrBadAppendRequest is returned when a caller passes bad parameters to
// Append
ErrBadAppendRequest = errors.New("bad request")
// ErrBulkState is returned when the bulk state is invalid
ErrBulkState = errors.New("bulk state error")
)

//go:generate moqueries OpenFileFn

// OpenFileFn is the function type of os.OpenFile
type OpenFileFn func(name string, flag int, perm os.FileMode) (ReadWriteSeekCloser, error)

//go:generate moqueries ReadWriteSeekCloser

// ReadWriteSeekCloser is the interface that groups the basic Read, Write,
// Seek and Close methods.
type ReadWriteSeekCloser interface {
io.Reader
io.Writer
io.Seeker
io.Closer
}

// Append appends a mock generate request to the bulk state
func Append(stateFile string, req generator.GenerateRequest, openFileFn OpenFileFn) error {
if !path.IsAbs(req.WorkingDir) {
return fmt.Errorf("%w: the request working directory must be absolute: %s",
ErrBadAppendRequest, req.WorkingDir)
}

f, err := openFileFn(stateFile, os.O_RDWR|os.O_APPEND, 0)
if err != nil {
return fmt.Errorf("error opening state file: %w", err)
}
defer func() {
err := f.Close()
if err != nil {
logs.Error("error closing state file", err)
}
}()

_, err = verifyState(f, stateFile, req.WorkingDir, false)
if err != nil {
return err
}

err = appendRequest(f, stateFile, req)
if err != nil {
return err
}

return nil
}

func verifyState(f io.ReadCloser, stateFile, workingDir string, rootDirOnly bool) (*bufio.Scanner, error) {
scanner := bufio.NewScanner(f)
if !scanner.Scan() {
return nil, fmt.Errorf("%w: state file %s not initialized properly",
ErrBulkState, stateFile)
}

txt := scanner.Text()
err := scanner.Err()
if err != nil {
return nil, fmt.Errorf("error reading state file %s: %w", stateFile, err)
}

var state initialState
err = json.Unmarshal([]byte(txt), &state)
if err != nil {
return nil, fmt.Errorf("error unmarshalling state file %s: %w", stateFile, err)
}

if state.GoPath != build.Default.GOPATH {
return nil, fmt.Errorf("%w: current GOPATH doesn't match GOPATH from state file %s (%s != %s)",
ErrBulkState, stateFile, build.Default.GOPATH, state.GoPath)
}

if rootDirOnly {
if state.RootDir != workingDir {
return nil, fmt.Errorf("%w: finalize root directory %s does"+
" not match root directory %s from state file %s",
ErrBulkState, workingDir, state.RootDir, stateFile)
}
} else {
rel, err := filepath.Rel(state.RootDir, workingDir)
if err != nil {
logs.Panicf("error getting relative path %s from %s: %#v",
state.RootDir, workingDir, err)
}

if strings.HasPrefix(rel, "..") {
return nil, fmt.Errorf("%w: working directory %s is not a"+
" child of root directory %s from state file %s",
ErrBulkState, workingDir, state.RootDir, stateFile)
}
}

return scanner, nil
}

func appendRequest(f ReadWriteSeekCloser, stateFile string, req generator.GenerateRequest) error {
_, err := f.Seek(0, io.SeekEnd)
if err != nil {
return fmt.Errorf("error seeking end of state file %s: %w", stateFile, err)
}
_, err = f.Write(compact(req))
if err != nil {
return fmt.Errorf("error writing state file %s: %w", stateFile, err)
}
_, err = f.Write([]byte("\n"))
if err != nil {
return fmt.Errorf("error finishing writing of state file %s: %w", stateFile, err)
}
return nil
}
Loading

0 comments on commit 767ac61

Please sign in to comment.