v24.4.0

24.4.0 (2024-06-04)

⚠ BREAKING CHANGES

• Cleanup API definitions, module imports (#534)

Features

• add 1password integration docs (#595) (29045ea)
• Add cluster admin cli (#644) (5b1de57)
• add crdb data source to grafana (#732) (f22694f)
• add env update error (#814) (48a5883)
• add hacky secondary ingress route for game lobbies (#567) (8bb6bd6)
• add internal api monolith (#641) (f25ffe4)
• Add managed OpenGB (#535) (9085d51)
• add opengb to bootstrap (#844) (ebd3c7b)
• add operation to list all clusters (#717) (1f4b169)
• add patch method to router (#744) (ed6596c)
• add pool filter to cluster dashboard (#830) (5436461)
• add provider api token to all linode calls (#613) (3882047)
• add provisioning dashboard (#733) (a1f9dcc)
• add ray id to error body (#833) (c115d6f)
• add region list/resolve per game (#633) (92275d8)
• Add script to run cargo clean (#700) (0f653e2)
• add toggle for load tests (#583) (a78d682)
• add vector http source (#800) (f4f2734)
• api-admin: add server destroy endpoint (#838) (4ff616b)
• bolt: list datacenter CLI command (#728) (c4a88de)
• bolt: update datacenters from CLI (#727) (083cd19)
• configurable drain ts per pool (#684) (f88c457)
• dynamic TLS generation (#635) (66e49dd)
• grafana: rivet logs dashboard (#724) (9a43f3a)
• infra: add ability to provision dev tunnel (#692) (659f8a1)
• Infra: Loops welcome email (b2e4006)
• nix: skip building bolt in nix with NIX_SKIP_BOLT (#664) (8e16a94)
• svc: resolve cluster name id op (#751) (58200ec)

Bug Fixes

• add last upload id (#745) (d10d917)
• add min count to autoscaler (#826) (9fe12a1)
• add patch to CORS (#848) (09f3ddc)
• add region to dns for path routing (#574) (e10ad25)
• add transacitons (#689) (f55b7e6)
• add transactions and locks (#696) (477ade5)
• api admin hub endpoint is incorrect (#660) (0aff347)
• api-status: auto-delete lobby after testing connection (#770) (9803f39)
• ats: don't send requests to ats nodes without install_complete_ts (#807) (618a429)
• bolt: copy & install git in docker for cluster build.rs (#769) (12bf1d4)
• bolt: correctly check for existing env var (#705) (ca4e48d)
• bolt: dont fully parse config when pulling (#816) (d22b08b)
• bolt: uncomment provisioning check (#749) (f25bead)
• bolt: update rust test package_id parsing (#622) (3d987ab)
• Change sdks linguist-vendored to linguist-generated (#662) (602749f)
• change test relative path (#754) (daf1d07)
• check for draining before installing/creating dns (#773) (cbe450b)
• chirp: add bypass for recursive messages (#708) (566088f)
• CI regression (#713) (636f0d3)
• claims (#672) (d61e290)
• clean up nomad jobs per test (#596) (6d7f0ee)
• Cleanup API definitions, module imports (#534) (0e0660a)
• cluster: delete dns record after failure to create (#827) (35fc6fe)
• cluster: don't taint servers that have cloud_destroy_ts (#839) (e5256f1)
• cluster: gg dns records leak if server destroyed before install comp...

v24.3.0

24.3.0 (2024-03-01)

Features

• bolt: add region filter to ssh command (#537) (af274a8)
• expose nomad dashboard via cloudflare tunnels (#543) (3a574c0)
• Main: Added Devcontainer files (9bb97db)
• mm: add config to opt-in individual games for host networking & root containers (#549) (be9ddd6)

Bug Fixes

• add checksum annotations to cloudflared deployment (#542) (f2d847b)
• bolt: clarify 1password service token warning (#541) (eb2e7d5)
• correct hcaptcha length (#548) (748aaa8)
• inaccessible admin routes (#555) (9896b09)
• revert to redis-rs v0.23.3 with panic patch (#552) (3780eaa)
• updated docs error url (#544) (7099658)

v24.2.0

Added

• Infra Added Better Uptime monitor
• Bolt Add Docker RUN cache to distributed deploys to improve deploy speeds
• Infra Prometheus VPA
• Infra Apache Traffic Server VPA
• api-cloud Admins can view all teams & games in a cluster
• Added automatic deploy CI for staging
• Infra Added compactor and GC to Loki
• api-status Test individual Game Guard nodes to ensure all nodes have the correct configuration
• Generate separate SDKs for runtime (lightweight, essentials for running a game) and full (heavy, includes cloud APIs)
• Metrics for cache operations as well as a Grafana dashboard
• Bolt Added namespace config and secrets sync with bolt config pull and bolt config push via 1Password
• GROUP_DEACTIVATED error now shows reasons for deactivation. Added docs for deactivation reasons
• /health/essential endpoint to test connectivity to all essential services
• Added error when trying to deploy a distributed cluster on a non-linux-x86 machine (not supported)

Changed

• api-status More comprehensive status check that both creates a lobby & connects to it
• More details in CLAIMS_MISSING_ENTITLEMENT error
• API Added 120s timeout to reading request body and writing response to all requests going through Traefik
• Infra Update Promtail logs to match k8s semantics
• Infra Added Cache-Control: no-cache to 400 responses from CDN
• [BREAKING] Infra Removed config-less hCaptcha. You are now required to provide a site key and
  secret key for the hCaptcha config in your game version matchmaker config for all future versions (old
  version will remain operational using our own hCaptcha site key).
• Internal Updated source hash calculation to use git diff and git rev-parse HEAD
• API Removed x-fern-* headers from generated TypeScript clients
• Implemented liveness probe to check connectivity to essential services
• Remove public-facing health check endpoints
• API Removed ability to choose a name id when creating a game. One will be generated based on the given display name

Fixed

• Bolt Prompt prod won't prompt if does not have user control
• Bolt Exclude copying bloat from infra/tf/ to distributed Docker builds
• Invalid JWT tokens now return explicit TOKEN_INVALID error instead of 500
• Infra Remove debug logging from traefik-tunnel
• Game lobby logs now ship even when the lobby fails immediately
• Fixed CLAIMS_MISSING_ENTITLEMENT not formatting correctly (reason given was ?)
• Added role ARN to exec commands in k8s-cluster-aws tf provider to properly authenticate
• Change email attached to Stripe on group ownership change
• Enable keep-alive on redis crate
• Update redis crate to mitigate panic on connection failure during AUTH
• Wrong grace period for GG config to update after mm::msg::lobby_ready

Security

• Resolve RUSTSEC-2024-0003

v24.1.0

Added

• Infra New job-runner crate responsible for managing the OCI bundle runtime & log shipping on the machine
• Infra Jobs now log an explicit rate message when logs are rate limited & truncated
• Infra infra-artifacts Terraform plan & S3 bucket used for automating building & uploading internal binaries, etc.
• Infra Aiven Redis provider
• Bolt bolt secret set <path> <value> command
• Bolt bolt.confirm_commands to namespace to confirm before running commands on a namespace
• watch-requests load test
• mm-sustain load test

Changed

• Matchmaker Allow excluding matchmaker.regions in order to enable all regions
• Matchmaker Lowered internal overhead of log shipping for lobbies
• Matchmaker Game mode names are now more lenient to include capital letters & underscores
• API Return API_REQUEST_TIMEOUT error after 50s (see docs/infrastructure/API_TIMEOUTS.md for context)
• API Move generated client APIs to sdks/
• Lower long poll timeout from 60s -> 40s
• Bolt Moved additional project roots to Bolt.toml
• types Support multiple project roots for reusing Protobuf types
• Infra Switch from AWS ELB to NLB to work around surge queue length limitation
• Infra Loki resources are now configurable
• pools Allow infinite Redis reconnection attempts
• pools Set Redis client names
• pools Ping Redis every 15 seconds
• pools Enable test_before_acquire on SQLx
• pools Decrease SQLx idle_timeout to 3 minutes
• pools Set ClickHouse idle_timeout to 15 seconds
• api-helper Box path futures for faster compile times
• Upgrade async-nats
• test-mm-lobby-echo now handles SIGTERM and exits immediately, allows for less resource consumption while testing lobbies
• mm Dynamically sleep based on lobby's create_ts for Treafik config to update
• Infra Update Traefik tunnel client & server to v3.0.0-beta5
• Infra Update Traefik load balancer to v2.10.7

Security

• Resolve RUSTSEC-2023-0044

Fixed

• Infra runc rootfs is now a writable file system
• Matchmaker Logs not shipping if lobby exits immediately
• Matchmaker Returning lnd-atl instead of dev-lcl as the mocked mocked region ID in the region list
• API 520 error when long polling
• api-cloud Returning wrong domain for domains.cdn
• Infra Fix Prometheus storage retention conversion between mebibytes and megabytes
• Infra Fix typo in Game Guard Traefik config not exposing API endpoint
• Infra Kill signal for servers was SIGINT instead of SIGTERM
• Infra NATS cluster not getting enabled
• Infra Redis Kubernetes error when using non-Kubernetes provider
• api-helper Remove excess logging
• user_identity.identities not getting purged on create & delete
• Bolt Error when applying Terraform when a plan is no longer required
• api-helper Instrument path futures
• Infra CNI ports not being removed from the nat iptable, therefore occasionally causing failed connections
• Infra Disable nativeLB for Traefik tunnel
• Infra Update default Nomad storage to 64Gi
• Infra Tunnel now exposes each Nomad server individually so the Nomad client can handle failover natively instead of relying on Traefik
• Infra Traefik tunnel not respecting configured replicas
• Bolt ClickHouse password generation now includes required special characters

v23.2.0-rc1

Added

• Infra Lobby tagging system for filtering lobbies in /find
• Infra Dynamically configurable max player count in /find and /create
• Bolt Added bolt admin login to allow for logging in without an email provider setup. Automatically turns the user into an admin for immediate access to the developer dashboard.
• Bolt Fixed bolt db migrate create
• Infra Added user-admin-set service for creating an admin user

Changed

• Bolt Removed bolt admin team-dev create. You can use bolt admin login and the hub to create a new dev team
• Infra Turnstile CAPTCHA_CAPTCHA_REQUIRED responses now include a site key
• Infra Turnstile is no longer configurable by domain (instead configured by Turnstile itself)
• Infra Job log aggregating to use Vector under the hood to insert directly into ClickHouse
• Matchmaker Players automatically remove after extended periods of time to account for network failures

Fixed

• Infra Job logs occasionally returning duplicate log lines
• Matchmaker /list returning no lobbies unless include_state query parameter is true
• Matchmaker Players remove correctly when the player fails to be inserted into the Cockroach database and only exists in Redis

v23.1.0

Fixed

• prm-job configuration now gets applied correctly again

v23.1.0-rc4

Changed

• Revert Fern TypeScript generator to 0.5.6 to fix bundled export

v23.1.0-rc3

Changed

• Don't publish internal Fern package on tag to prevent duplicate pushes

v23.1.0-rc2

Changed

• Update to Fern 0.15.0-rc7
• Update Fern TypeScript, Java, and Go generators

v23.1.0-rc1

Added

• Matchmaker Support custom lobbies
• Matchmaker Support lobby state
• Matchmaker Support external verification
• Library Support Java library
• Library Support Go library
• Cloud Support multipart uploads for builds
• Infra Support configuring multiple S3 providers
• Infra Support multipart uploads
• Infra Replace Promtail-based log shipping with native Loki Docker driver
• Infra Local Traefik Cloudflare proxy daemon for connecting to Cloudflare Access services
• Infra Upload service builds to default S3 provider instead of hardcoded bucket
• Infra Enable Apache Traffic Server pull-through cache for Docker builds
• Bolt Support for connecting to Redis databases with bolt redis sh
• Bolt Confirmation before running any command in the production namespace
• Bolt --start-at flag for all infra commands
• Bolt Explicit database dependencies in services to reduce excess database pools

Changed

• Infra Update CNI plugins to 1.3.0
• Infra Update ClickHouse to 23.7.2.25
• Infra Update Cockroach to 23.1.7
• Infra Update Consul Exporter to 1.9.0
• Infra Update Consul to 1.16.0
• Infra Update Imagor to 1.4.7
• Infra Update NATS server to 2.9.20
• Infra Update Node Exporter server to 1.6.0
• Infra Update Nomad to 1.6.0
• Infra Update Prometheus server to 2.46.0
• Infra Update Redis Exporter to 1.52.0
• Infra Update Redis to 7.0.12
• Infra Update Treafik to 2.10.4
• Bolt PostHog events are now captured in a background task
• Bolt Auto-install rsync on Salt Master
• Bolt Recursively add dependencies from overridden services when using additional roots
• KV Significantly rate limit of all endpoints

Security

• Resolve RUSTSEC-2023-0044
• Resolve RUSTSEC-2022-0093
• Resolve RUSTSEC-2023-0053

Fixed

• Portal Skip captcha if no Turnstile key provided
• Infra Missing dpenedency on mounting volumn before setting permissions of /var/* for Cockroach, ClickHouse, Prometheus, and Traffic Server
• Chrip Empty message parameters now have placeholder so NATS doesn't throw an error
• Chrip Messages with no parameters no longer have a trailing dot
• Bolt Correctly resolve project root when building services natively
• Bolt Correctly determine executable path for ExecServiceDriver::UploadedBinaryArtifact with different Cargo names