Skip to content

feat(runtime/go): smart ShouldRebuild via go list -deps#6823

Open
alessandrolattao wants to merge 2 commits intoanomalyco:devfrom
alessandrolattao:feat/golang-runtime-smart-rebuild
Open

feat(runtime/go): smart ShouldRebuild via go list -deps#6823
alessandrolattao wants to merge 2 commits intoanomalyco:devfrom
alessandrolattao:feat/golang-runtime-smart-rebuild

Conversation

@alessandrolattao
Copy link
Copy Markdown

Why

The Go runtime today rebuilds every Lambda in the same go.mod tree whenever any file changes. That's fine with the documented layout (one go.mod per handler), but as soon as a project chooses a single-module monorepo with many handlers — share a services/shared package across N Lambdas, keep one set of dependencies, one go.mod tidy, etc. — sst dev becomes unusable: editing one handler triggers a rebuild of every other handler in the module.

Side effect of one-go.mod-per-Lambda also pays a recurring cost on every CI lint and CI test pass that scales linearly with the Lambda count, since each handler is its own golangci-lint + go test invocation.

This is the same problem the Node runtime already solved: it uses esbuild's metafile to know exactly which files compile into each bundle and rebuilds only the Lambdas whose actual deps changed.

What

Capture each handler's transitive import graph at Build time via go list -deps -json and use it as the rebuild scope. ShouldRebuild fires only when the changed file is in that handler's graph. Works in every layout (per-Lambda, per-domain, single-module monorepo).

Side fix: go build arguments now use a ./ prefix on the source path so layouts where the source is a sub-path don't fail with package <path> is not in std. With the legacy one-go.mod-per-Lambda layout src was always ., so the bug never surfaced.

The previous directory-scope behavior was deliberately removed rather than kept as a fallback. In a single-module repo it would silently fan out to every Lambda when captureDeps failed, hiding a real problem behind a worse one. A captureDeps failure now logs a warning with full context (goflags, gomodcache) and disables ShouldRebuild for that function until the next successful Build.

Tests

`pkg/runtime/golang/golang_test.go` — pure unit tests for `Properties` JSON contract, the goarch mapping, `ShouldRebuild` edge cases (non-go file, missing graph, file hit/miss, relative path resolution), the `GOMODCACHE`/stdlib filter via a fixture JSON stream, a 50×50 race-detector concurrency probe, and a cross-OS `isUnderDir` helper.

`pkg/runtime/golang/build_test.go` — hermetic integration tests against the real `go list` toolchain (env scrubbed: `GOPROXY=off`, `GOTOOLCHAIN=local`, redirected `GOMODCACHE`/`GOCACHE`/`HOME`). Skipped under `-short` and when `go` is not in `PATH`.

`go test ./pkg/runtime/golang/... -race`: 1.6s full, 1.0s under `-short`. `go vet` clean.

Tested manually

Validated end-to-end on a real repo with ~60 Go Lambdas consolidated into one root `services/go.mod`. With this patch:

  • editing a single Lambda's `handler.go` rebuilds only that Lambda;
  • editing a file in the shared package rebuilds only the Lambdas that import it (cross-Lambda fan-out is correct);
  • editing a `_test.go` triggers no rebuild (test files don't compile into the binary);
  • `bun run go-lint` time on the same repo dropped from ~10s to <1s warm cache (one `golangci-lint` invocation over the whole module instead of 60).

The Go runtime today rebuilds every Lambda in the same go.mod tree
whenever any file changes. Fine when you have one go.mod per Lambda
(the doc'd layout), but unworkable when you choose a single-module
monorepo with many handlers — in that case `sst dev` rebuilds every
Lambda on every save and the dev loop falls apart.

This change captures each handler's transitive import graph at Build
time (`go list -deps -json`) and uses it as the rebuild scope.
ShouldRebuild fires only on files actually in that graph — same idea
the Node runtime already uses with its esbuild metafile. Works in
every layout (per-Lambda, per-domain, single-module).

Also fixes a `go build` bug exposed by sub-path handlers: the source
arg now has a `./` prefix so a non-`.` source path doesn't get
treated as a stdlib package import.

Tests cover ShouldRebuild edges, the GOMODCACHE/stdlib filter, a
race-detector concurrency probe, and hermetic `go list` integration
fixtures (skipped under -short).
Copilot AI review requested due to automatic review settings April 28, 2026 12:22
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves the Go runtime’s incremental rebuild behavior in sst dev by capturing each handler’s transitive dependency file set via go list -deps -json during Build, and using that set in ShouldRebuild to rebuild only the functions actually affected by a file change. It also fixes go build invocation for sub-path handlers by forcing ./-prefixed relative package paths, and adds unit + integration tests for the new dependency-capture and rebuild-scope logic.

Changes:

  • Capture per-function dependency graphs at build time (go list -deps -json) and use them to scope ShouldRebuild.
  • Fix go build argument to use ./-prefixed relative handler paths to avoid stdlib-package misresolution.
  • Add unit tests for graph parsing / path helpers / concurrency and hermetic integration tests against the real Go toolchain.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File Description
pkg/runtime/golang/golang.go Implements dependency capture, new rebuild decision logic, GOMODCACHE filtering, and path fix for go build.
pkg/runtime/golang/golang_test.go Adds unit tests for Properties contract, arch mapping, ShouldRebuild, isUnderDir, and go-list parsing filters.
pkg/runtime/golang/build_test.go Adds integration tests exercising captureDeps with real go list in a scrubbed environment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread pkg/runtime/golang/golang.go
Comment thread pkg/runtime/golang/golang.go Outdated
Comment thread pkg/runtime/golang/golang.go
Comment thread pkg/runtime/golang/golang_test.go Outdated
Comment thread pkg/runtime/golang/golang.go Outdated
The previous sync.Once made the GOMODCACHE filter silently inert if
the very first Build had a degenerate env: a transient `go env`
failure locked the cached value to "" for the lifetime of the
process. Replace it with a Mutex + resolved flag, slog.Warn on
failure, and re-try on the next call.

Tag rebuild log lines with a `reason` (file_in_graph, gomod_change,
new_file_in_pkg) so dev-loop traces show why each rebuild fires.
Demote the per-build "running go build" line to Debug. Drop the
unused per-handler `directories` map now that the import graph
covers it.

Add GOWORK=off and GOSUMDB=off to the integration test env so
`go list` can't pick up the host's go.work or hit the checksum DB.
@alessandrolattao
Copy link
Copy Markdown
Author

alessandrolattao commented Apr 28, 2026

I've just realized that the current version fails to trigger a rebuild for Lambdas when code shared via workspaces/replaces is modified, even if the Lambda depends on it. This update also fixes that issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants