Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: db driven platform settings #1110

Merged
merged 87 commits into from
May 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
9aad646
wip: settings started
plyr4 Mar 7, 2024
9038758
chore: revert makefile
plyr4 Mar 19, 2024
0a4bf6a
Merge branch 'main' of github.com:go-vela/server into feat/platform-s…
plyr4 Mar 20, 2024
e779f5d
wip: platform settings using new types pattern
plyr4 Mar 21, 2024
be2edac
Merge branch 'main' of github.com:go-vela/server into feat/platform-s…
plyr4 Apr 1, 2024
4628b02
feat: boiler plate api code for settings endpoints
plyr4 Apr 1, 2024
4cf60eb
feat: wip settings object used for clone image
plyr4 Apr 2, 2024
0909e2f
Merge branch 'main' of github.com:go-vela/server into feat/platform-s…
plyr4 Apr 2, 2024
7e24e65
feat: wip settings embedded into compiler
plyr4 Apr 2, 2024
336dcf5
feat: query params to control output format, and clone image control
plyr4 Apr 3, 2024
a200311
Merge branch 'main' of github.com:go-vela/server into feat/platform-s…
plyr4 Apr 9, 2024
42726c8
wip: infra for queue routes and starlark exec limit
plyr4 Apr 10, 2024
04e43f0
wip: update queue and compiler using middleware
plyr4 Apr 11, 2024
648dd78
wip: debugging compile and publish panic
plyr4 Apr 11, 2024
3dba7ba
chore: merge with main
plyr4 Apr 16, 2024
c528210
Merge branch 'main' of github.com:go-vela/server into feat/platform-s…
plyr4 Apr 16, 2024
ee9b90f
wip: structure for settings done
plyr4 Apr 17, 2024
51c82bc
wip: restructure using nested settings
plyr4 Apr 17, 2024
e052235
enhance: reorganize into subtypes using db scan
plyr4 Apr 17, 2024
1f36713
fix: linter
plyr4 Apr 17, 2024
82fabca
chore: remove debug files
plyr4 Apr 17, 2024
17d6e68
chore: remove debug line
plyr4 Apr 17, 2024
d9ac8db
chore: cleanup, fmt, remove repo allowlist
plyr4 Apr 17, 2024
68aff5e
golangci-lint run --fix
plyr4 Apr 17, 2024
05d0389
fix: linter
plyr4 Apr 17, 2024
73f67f8
Merge branch 'main' into feat/platform-settings
plyr4 Apr 17, 2024
763f107
feat: schedule allowlist and more linting
plyr4 Apr 17, 2024
a612efa
fix: comma missing
plyr4 Apr 17, 2024
fd7f354
fix: compiler middleware test
plyr4 Apr 17, 2024
a6ae525
feat: delete settings
plyr4 Apr 17, 2024
50b8493
fix: uncomment delete code
plyr4 Apr 17, 2024
033ff65
Merge branch 'main' into feat/platform-settings
plyr4 Apr 17, 2024
47368ab
fix: create nested api types
plyr4 Apr 17, 2024
453f645
fix: check for nils
plyr4 Apr 17, 2024
dac5053
fix: remove sanitize but still crop fields
plyr4 Apr 19, 2024
6ba421a
Merge branch 'fix/owner-token' of github.com:go-vela/server into feat…
plyr4 Apr 22, 2024
8377d66
Merge branch 'main' into feat/platform-settings
plyr4 Apr 22, 2024
9e1af91
Merge branch 'main' into feat/platform-settings
plyr4 Apr 24, 2024
dfd5e6b
chore: merge with main
plyr4 Apr 24, 2024
ce6c60e
chore: merge with main
plyr4 Apr 24, 2024
be0d672
feat: types tests and general cleanup
plyr4 Apr 29, 2024
aaf0038
feat: db integration tests
plyr4 Apr 29, 2024
dfdcec5
chore: remove unneeded api endpoints
plyr4 Apr 29, 2024
1e0b4da
feat: db tests and general cleanup
plyr4 Apr 29, 2024
b0d9bbc
chore: lint
plyr4 Apr 29, 2024
af4acdd
feat: flag for resetting settings record on startup
plyr4 Apr 29, 2024
d322449
fix: get template depth in log, not pointer
plyr4 Apr 30, 2024
392cb5b
Merge branch 'main' of github.com:go-vela/server into feat/platform-s…
plyr4 May 1, 2024
be90441
enhance: timestamps for created and updated
plyr4 May 1, 2024
8c37621
chore: lint
plyr4 May 1, 2024
9e4a327
enhance: timestamps for created and updated
plyr4 May 1, 2024
e765740
enhance: validations for timestamps
plyr4 May 1, 2024
83215da
fix: test
plyr4 May 1, 2024
0d38a59
Merge branch 'main' into feat/platform-settings
plyr4 May 3, 2024
b12c907
Merge branch 'main' into feat/platform-settings
plyr4 May 3, 2024
d7ff645
fix: return updated settings
plyr4 May 3, 2024
cd3c3f9
Merge branch 'feat/platform-settings' of github.com:go-vela/server in…
plyr4 May 3, 2024
7ab84af
fix: return updated settings
plyr4 May 3, 2024
069da63
chore: lint
plyr4 May 3, 2024
9b47082
Merge branch 'main' into feat/platform-settings
plyr4 May 7, 2024
f5effff
enhance: move to db types, fix tests
plyr4 May 7, 2024
20f5691
fix: add jitter around initialization
plyr4 May 7, 2024
7bd45ca
fix: import order
plyr4 May 7, 2024
83f9380
fix: comment and jitter time
plyr4 May 7, 2024
8a25c07
Merge branch 'main' into feat/platform-settings
plyr4 May 7, 2024
7bcf015
chore: merge with main
plyr4 May 8, 2024
5a7fda9
fix: use refresh interval to avoid db spam
plyr4 May 8, 2024
b358b72
enhance: background go func that refreshes settings on an interval, c…
plyr4 May 8, 2024
297014f
chore: linter
plyr4 May 8, 2024
dee4488
Merge branch 'main' into feat/platform-settings
plyr4 May 8, 2024
c53ddbe
chore: typo
plyr4 May 8, 2024
4110ac2
chore: typos and missing test
plyr4 May 8, 2024
409a267
fix: typos, swagger, whitespace, 404s
plyr4 May 9, 2024
81fa5dd
fix: consistent variable naming and whitespace
plyr4 May 9, 2024
17a42d9
chore(queue): rename channels to routes
plyr4 May 9, 2024
eb3417b
fix: lint naming
plyr4 May 9, 2024
a56ea84
enhance: check record not found error instead of nil
plyr4 May 9, 2024
197e42f
enhance: add docker image validation
plyr4 May 10, 2024
1f8ef15
fix: typo
plyr4 May 10, 2024
da472a4
fix: lint fmt
plyr4 May 10, 2024
3a627ef
fix: test case flags
plyr4 May 10, 2024
f120029
fix: test case lint
plyr4 May 10, 2024
a43546e
fix: clone image compiler test
plyr4 May 10, 2024
4cad3e6
fix: dont update shared pointer
plyr4 May 10, 2024
b6e2672
fix: set ID
plyr4 May 10, 2024
e77df13
Merge branch 'main' into feat/platform-settings
plyr4 May 10, 2024
f54aa67
fix: test merge
plyr4 May 10, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
278 changes: 278 additions & 0 deletions api/admin/settings.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,278 @@
// SPDX-License-Identifier: Apache-2.0

package admin

import (
"fmt"
"net/http"
"time"

"github.com/gin-gonic/gin"
"github.com/sirupsen/logrus"

"github.com/go-vela/server/api/types/settings"
plyr4 marked this conversation as resolved.
Show resolved Hide resolved
"github.com/go-vela/server/compiler/native"
"github.com/go-vela/server/database"
"github.com/go-vela/server/internal/image"
"github.com/go-vela/server/queue"
cliMiddleware "github.com/go-vela/server/router/middleware/cli"
sMiddleware "github.com/go-vela/server/router/middleware/settings"
uMiddleware "github.com/go-vela/server/router/middleware/user"
"github.com/go-vela/server/util"
)

// swagger:operation GET /api/v1/admin/settings admin GetSettings
//
// Get the currently configured settings.
//
// ---
// produces:
// - application/json
// security:
// - ApiKeyAuth: []
// responses:
// '200':
// description: Successfully retrieved settings
// type: json
// schema:
// "$ref": "#/definitions/Platform"
// '404':
// description: Unable to retrieve settings
// schema:
// "$ref": "#/definitions/Error"

// GetSettings represents the API handler to
// captures settings stored in the database.
func GetSettings(c *gin.Context) {
// capture middleware values
s := sMiddleware.FromContext(c)

logrus.Info("Admin: reading settings")

// check captured value because we aren't retrieving settings from the database
// instead we are retrieving the auto-refreshed middleware value
if s == nil {
retErr := fmt.Errorf("settings not found")

util.HandleError(c, http.StatusNotFound, retErr)

return
}

c.JSON(http.StatusOK, s)
}

// swagger:operation PUT /api/v1/admin/settings admin UpdateSettings
//
// Update the platform settings singleton in the database.
//
// ---
// produces:
// - application/json
// parameters:
// - in: body
// name: body
// description: Payload containing settings to update
// required: true
// schema:
// "$ref": "#/definitions/Platform"
// security:
// - ApiKeyAuth: []
// responses:
// '200':
// description: Successfully updated platform settings in the database
// type: json
// schema:
// "$ref": "#/definitions/Platform"
// '400':
// description: Unable to update settings — bad request
// schema:
// "$ref": "#/definitions/Error"
// '404':
// description: Unable to retrieve platform settings to update
// schema:
// "$ref": "#/definitions/Error"
// '500':
// description: Unable to update platform settings in the database
// schema:
// "$ref": "#/definitions/Error"

// UpdateSettings represents the API handler to
// update the settings singleton stored in the database.
func UpdateSettings(c *gin.Context) {
// capture middleware values
s := sMiddleware.FromContext(c)
u := uMiddleware.FromContext(c)
ctx := c.Request.Context()

logrus.Info("Admin: updating settings")

// check captured value because we aren't retrieving settings from the database
// instead we are retrieving the auto-refreshed middleware value
if s == nil {
retErr := fmt.Errorf("settings not found")

util.HandleError(c, http.StatusNotFound, retErr)

return
}

// duplicate settings to not alter the shared pointer
_s := new(settings.Platform)
_s.Update(s)

// ensure we update the singleton record
_s.SetID(1)

// capture body from API request
input := new(settings.Platform)

err := c.Bind(input)
if err != nil {
retErr := fmt.Errorf("unable to decode JSON for settings: %w", err)

util.HandleError(c, http.StatusBadRequest, retErr)
plyr4 marked this conversation as resolved.
Show resolved Hide resolved

return
}

if input.Compiler != nil {
if input.CloneImage != nil {
// validate clone image
cloneImage := *input.CloneImage

_, err = image.ParseWithError(cloneImage)
if err != nil {
retErr := fmt.Errorf("invalid clone image %s: %w", cloneImage, err)

util.HandleError(c, http.StatusBadRequest, retErr)

return
}

_s.SetCloneImage(cloneImage)
}

if input.TemplateDepth != nil {
_s.SetTemplateDepth(*input.TemplateDepth)
}

if input.StarlarkExecLimit != nil {
_s.SetStarlarkExecLimit(*input.StarlarkExecLimit)
}
}

if input.Queue != nil {
if input.Queue.Routes != nil {
_s.SetRoutes(input.GetRoutes())
}
}

if input.RepoAllowlist != nil {
_s.SetRepoAllowlist(input.GetRepoAllowlist())
}

if input.ScheduleAllowlist != nil {
_s.SetScheduleAllowlist(input.GetScheduleAllowlist())
}

_s.SetUpdatedBy(u.GetName())

// send API call to update the settings
_s, err = database.FromContext(c).UpdateSettings(ctx, _s)
if err != nil {
retErr := fmt.Errorf("unable to update settings: %w", err)

util.HandleError(c, http.StatusInternalServerError, retErr)
plyr4 marked this conversation as resolved.
Show resolved Hide resolved

return
}

c.JSON(http.StatusOK, _s)
}

// swagger:operation DELETE /api/v1/admin/settings admin RestoreSettings
//
// Restore the currently configured settings to the environment defaults.
//
// ---
// produces:
// - application/json
// security:
// - ApiKeyAuth: []
// responses:
// '200':
// description: Successfully restored default settings in the database
// type: json
// schema:
// "$ref": "#/definitions/Platform"
// '404':
// description: Unable to retrieve settings to restore
// schema:
// "$ref": "#/definitions/Error"
// '500':
// description: Unable to restore settings in the database
// schema:
// "$ref": "#/definitions/Error"

// RestoreSettings represents the API handler to
// restore settings stored in the database to the environment defaults.
func RestoreSettings(c *gin.Context) {
// capture middleware values
s := sMiddleware.FromContext(c)
u := uMiddleware.FromContext(c)
cliCtx := cliMiddleware.FromContext(c)
ctx := c.Request.Context()

logrus.Info("Admin: restoring settings")

// check captured value because we aren't retrieving settings from the database
// instead we are retrieving the auto-refreshed middleware value
if s == nil {
retErr := fmt.Errorf("settings not found")

util.HandleError(c, http.StatusNotFound, retErr)

return
}

compiler, err := native.FromCLIContext(cliCtx)
if err != nil {
retErr := fmt.Errorf("unable to restore settings: %w", err)

util.HandleError(c, http.StatusInternalServerError, retErr)

return
}

queue, err := queue.FromCLIContext(cliCtx)
if err != nil {
retErr := fmt.Errorf("unable to restore settings: %w", err)

util.HandleError(c, http.StatusInternalServerError, retErr)

return
}

s.SetUpdatedAt(time.Now().UTC().Unix())
s.SetUpdatedBy(u.GetName())

// read in defaults supplied from the cli runtime
compilerSettings := compiler.GetSettings()
s.SetCompiler(compilerSettings)

queueSettings := queue.GetSettings()
s.SetQueue(queueSettings)

// send API call to update the settings
s, err = database.FromContext(c).UpdateSettings(ctx, s)
if err != nil {
retErr := fmt.Errorf("unable to update (restore) settings: %w", err)

util.HandleError(c, http.StatusInternalServerError, retErr)

return
}

c.JSON(http.StatusOK, s)
}
7 changes: 5 additions & 2 deletions api/repo/create.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import (
"github.com/go-vela/server/api/types"
"github.com/go-vela/server/api/types/actions"
"github.com/go-vela/server/database"
"github.com/go-vela/server/router/middleware/settings"
"github.com/go-vela/server/router/middleware/user"
"github.com/go-vela/server/scm"
"github.com/go-vela/server/util"
Expand Down Expand Up @@ -71,7 +72,8 @@ import (
func CreateRepo(c *gin.Context) {
// capture middleware values
u := user.Retrieve(c)
allowlist := c.Value("allowlist").([]string)
s := settings.FromContext(c)

defaultBuildLimit := c.Value("defaultBuildLimit").(int64)
defaultTimeout := c.Value("defaultTimeout").(int64)
maxBuildLimit := c.Value("maxBuildLimit").(int64)
Expand Down Expand Up @@ -197,6 +199,7 @@ func CreateRepo(c *gin.Context) {

return
}

r.SetPipelineType(input.GetPipelineType())
}

Expand All @@ -217,7 +220,7 @@ func CreateRepo(c *gin.Context) {
)

// ensure repo is allowed to be activated
if !util.CheckAllowlist(r, allowlist) {
if !util.CheckAllowlist(r, s.GetRepoAllowlist()) {
retErr := fmt.Errorf("unable to activate repo: %s is not on allowlist", r.GetFullName())

util.HandleError(c, http.StatusForbidden, retErr)
Expand Down
Loading
Loading