Summary
The user discusses a Dockerfile setup for a Go project, specifically the caching behavior of the go generate
step and its effects on dependency management. They note that the modcache in /go/modcache
only retains changes from the deps
target, leading to redownloads of dependencies when the source changes. The user questions the need for an additional CACHE
command in the src
target to retain changes and clarifies that without it, the cache is "read only." They mention that using CACHE --persist --sharing shared /go
allows changes to be retained. The user also discusses the inheritance of WORKDIR
and the relevance of the --keep-ts
option, stating it does not impact Go builds, although they acknowledge it can cause issues in other tools like Cargo. They seek validation from Earthly engineers on the expected behavior and suggest improvements in documentation.
ingwar
Sadly I wrote it only on slack.. I should write it into the lib..
ingwar
joshua.gilman
Good to know. I’ve only seen it break for Cargo because it uses file timestamps for breaking the cache. I thought Go was smarter than that because I’ve not yet run into a case where I needed to use that flag.
ingwar
I wrote pipeline for golang few times..
ingwar
> And using --keep-ts
depends on the underlying tool. For Go it doesn’t make a difference.
It makes a difference.. Without it it breaks sometimes.. and breaks in ugly way.. some files are old, and some are new..
joshua.gilman
The question here is more for the Earthly engineers to validate this is the expected behavior and maybe add some additional documentation if so.
joshua.gilman
Yes. As I already mentioned, adding the additional CACHE
command makes the target behave as expected. Meaning the modcache
folder maintains the modules downloaded by go generate
and subsequent runs don't redownload them.
FROM golang:1.22.4-alpine3.19
WORKDIR /work
RUN mkdir -p /go/cache && mkdir -p /go/modcache
ENV GOCACHE=/go/cache
ENV GOMODCACHE=/go/modcache
CACHE --persist --sharing shared /go
COPY go.mod go.sum .
RUN go mod download
src:
FROM +deps
CACHE --persist --sharing shared /go
COPY . .
RUN go generate ./...```
brandon356
Are you sure about --keep-ts
? go builds don't care Earthly doesn't know that step does not need to be re-run
brandon356
oh right
joshua.gilman
Not sure I understand your question. The WORKDIR
is inherited.
brandon356
Also, why is WORKDIR not lifted to be the same in deps and src?
joshua.gilman
And using --keep-ts
depends on the underlying tool. For Go it doesn't make a difference.
joshua.gilman
No, I've verified the behavior locally:
• Without adding an additional CACHE
command to the src
target, the cache appears to only be "read only." Meaning it doesn't persist changes made in the src
step.
• Adding an additional CACHE --persist --sharing shared /go
to the beginning of the src
target modifies the behavior so that changes made during the target are properly persisted.
I'm wondering if this is the intended behavior because it's very subtle and not really documented well.
brandon356
And you when you copy something inside earthly and use cache you always need to add --keep-ts
brandon356
It seems to me you need to cache go/modcache seperately?
joshua.gilman
FROM golang:1.22.4-alpine3.19
WORKDIR /work
RUN mkdir -p /go/cache && mkdir -p /go/modcache
ENV GOCACHE=/go/cache
ENV GOMODCACHE=/go/modcache
CACHE --persist --sharing shared /go
COPY go.mod go.sum .
RUN go mod download
src:
FROM +deps
COPY . .
RUN go generate ./...```
In the above example, the `go generate` step is populating the modcache with additional dependencies (because I use `go run <http://github.com/x/y|github.com/x/y>`). However, the cache in `/go/modcache` is only persisting the changes from the `deps` target causing the `src` step to redownload the dependencies every time the source changes.
Do I need to add an additional `CACHE` command in the `src` target? Or am I misunderstanding how the sharing is supposed to work?