E

Dockerfile Setup for Go Project

Summary

The user discusses a Dockerfile setup for a Go project, specifically the caching behavior of the go generate step and its effects on dependency management. They note that the modcache in /go/modcache only retains changes from the deps target, leading to redownloads of dependencies when the source changes. The user questions the need for an additional CACHE command in the src target to retain changes and clarifies that without it, the cache is "read only." They mention that using CACHE --persist --sharing shared /go allows changes to be retained. The user also discusses the inheritance of WORKDIR and the relevance of the --keep-ts option, stating it does not impact Go builds, although they acknowledge it can cause issues in other tools like Cargo. They seek validation from Earthly engineers on the expected behavior and suggest improvements in documentation.

Status
resolved
Tags
    Source
    #earthly
      i

      ingwar

      7/31/2024

      Sadly I wrote it only on slack.. I should write it into the lib..

      j

      joshua.gilman

      7/31/2024

      Good to know. I’ve only seen it break for Cargo because it uses file timestamps for breaking the cache. I thought Go was smarter than that because I’ve not yet run into a case where I needed to use that flag.

      i

      ingwar

      7/31/2024

      I wrote pipeline for golang few times..

      i

      ingwar

      7/31/2024

      > And using --keep-ts depends on the underlying tool. For Go it doesn’t make a difference. It makes a difference.. Without it it breaks sometimes.. and breaks in ugly way.. some files are old, and some are new..

      j

      joshua.gilman

      7/31/2024

      The question here is more for the Earthly engineers to validate this is the expected behavior and maybe add some additional documentation if so.

      j

      joshua.gilman

      7/31/2024

      Yes. As I already mentioned, adding the additional CACHE command makes the target behave as expected. Meaning the modcache folder maintains the modules downloaded by go generate and subsequent runs don't redownload them.

          FROM golang:1.22.4-alpine3.19
      
          WORKDIR /work
      
          RUN mkdir -p /go/cache && mkdir -p /go/modcache
          ENV GOCACHE=/go/cache
          ENV GOMODCACHE=/go/modcache
          CACHE --persist --sharing shared /go
      
          COPY go.mod go.sum .
          RUN go mod download
      
      src:
          FROM +deps
      
          CACHE --persist --sharing shared /go
      
          COPY . .
          RUN go generate ./...```
      
      b

      brandon356

      7/31/2024

      Are you sure about --keep-ts? go builds don't care Earthly doesn't know that step does not need to be re-run

      b

      brandon356

      7/31/2024

      oh right

      j

      joshua.gilman

      7/31/2024

      Not sure I understand your question. The WORKDIR is inherited.

      b

      brandon356

      7/31/2024

      Also, why is WORKDIR not lifted to be the same in deps and src?

      j

      joshua.gilman

      7/31/2024

      And using --keep-ts depends on the underlying tool. For Go it doesn't make a difference.

      j

      joshua.gilman

      7/31/2024

      No, I've verified the behavior locally:

      • Without adding an additional CACHE command to the src target, the cache appears to only be "read only." Meaning it doesn't persist changes made in the src step. • Adding an additional CACHE --persist --sharing shared /go to the beginning of the src target modifies the behavior so that changes made during the target are properly persisted. I'm wondering if this is the intended behavior because it's very subtle and not really documented well.

      b

      brandon356

      7/31/2024

      And you when you copy something inside earthly and use cache you always need to add --keep-ts

      b

      brandon356

      7/31/2024

      It seems to me you need to cache go/modcache seperately?

      j

      joshua.gilman

      7/31/2024
          FROM golang:1.22.4-alpine3.19
      
          WORKDIR /work
      
          RUN mkdir -p /go/cache && mkdir -p /go/modcache
          ENV GOCACHE=/go/cache
          ENV GOMODCACHE=/go/modcache
          CACHE --persist --sharing shared /go
      
          COPY go.mod go.sum .
          RUN go mod download
      
      src:
          FROM +deps
      
          COPY . .
          RUN go generate ./...```
      In the above example, the `go generate` step is populating the modcache with additional dependencies (because I use `go run <http://github.com/x/y|github.com/x/y>`). However, the cache in `/go/modcache` is only persisting the changes from the `deps` target causing the `src` step to redownload the dependencies every time the source changes.
      
      Do I need to add an additional `CACHE` command in the `src` target? Or am I misunderstanding how the sharing is supposed to work?