E

Dockerfile Setup for Go Project

Summary

The user discusses a Dockerfile setup for a Go project, specifically the caching behavior of the go generate step and its effects on dependency management. They note that the modcache in /go/modcache only retains changes from the deps target, leading to redownloads of dependencies when the source changes. The user questions the need for an additional CACHE command in the src target to retain changes and clarifies that without it, the cache is "read only." They mention that using CACHE --persist --sharing shared /go allows changes to be retained. The user also discusses the inheritance of WORKDIR and the relevance of the --keep-ts option, stating it does not impact Go builds, although they acknowledge it can cause issues in other tools like Cargo. They seek validation from Earthly engineers on the expected behavior and suggest improvements in documentation.

Status
resolved
Tags
  • Earthly
  • Dockerfile
  • Go Project
  • Caching
  • Docker
  • Go
Source
#earthly
    i

    ingwar

    7/31/2024

    Sadly I wrote it only on slack.. I should write it into the lib..

    j

    joshua.gilman

    7/31/2024

    Good to know. I’ve only seen it break for Cargo because it uses file timestamps for breaking the cache. I thought Go was smarter than that because I’ve not yet run into a case where I needed to use that flag.

    i

    ingwar

    7/31/2024

    I wrote pipeline for golang few times..

    i

    ingwar

    7/31/2024

    > And using --keep-ts depends on the underlying tool. For Go it doesn’t make a difference. It makes a difference.. Without it it breaks sometimes.. and breaks in ugly way.. some files are old, and some are new..

    j

    joshua.gilman

    7/31/2024

    The question here is more for the Earthly engineers to validate this is the expected behavior and maybe add some additional documentation if so.

    j

    joshua.gilman

    7/31/2024

    Yes. As I already mentioned, adding the additional CACHE command makes the target behave as expected. Meaning the modcache folder maintains the modules downloaded by go generate and subsequent runs don't redownload them.

        FROM golang:1.22.4-alpine3.19
    
        WORKDIR /work
    
        RUN mkdir -p /go/cache && mkdir -p /go/modcache
        ENV GOCACHE=/go/cache
        ENV GOMODCACHE=/go/modcache
        CACHE --persist --sharing shared /go
    
        COPY go.mod go.sum .
        RUN go mod download
    
    src:
        FROM +deps
    
        CACHE --persist --sharing shared /go
    
        COPY . .
        RUN go generate ./...```
    
    b

    brandon356

    7/31/2024

    Are you sure about --keep-ts? go builds don't care Earthly doesn't know that step does not need to be re-run

    b

    brandon356

    7/31/2024

    oh right

    j

    joshua.gilman

    7/31/2024

    Not sure I understand your question. The WORKDIR is inherited.

    b

    brandon356

    7/31/2024

    Also, why is WORKDIR not lifted to be the same in deps and src?

    j

    joshua.gilman

    7/31/2024

    And using --keep-ts depends on the underlying tool. For Go it doesn't make a difference.

    j

    joshua.gilman

    7/31/2024

    No, I've verified the behavior locally:

    • Without adding an additional CACHE command to the src target, the cache appears to only be "read only." Meaning it doesn't persist changes made in the src step. • Adding an additional CACHE --persist --sharing shared /go to the beginning of the src target modifies the behavior so that changes made during the target are properly persisted. I'm wondering if this is the intended behavior because it's very subtle and not really documented well.

    b

    brandon356

    7/31/2024

    And you when you copy something inside earthly and use cache you always need to add --keep-ts

    b

    brandon356

    7/31/2024

    It seems to me you need to cache go/modcache seperately?

    j

    joshua.gilman

    7/31/2024
        FROM golang:1.22.4-alpine3.19
    
        WORKDIR /work
    
        RUN mkdir -p /go/cache && mkdir -p /go/modcache
        ENV GOCACHE=/go/cache
        ENV GOMODCACHE=/go/modcache
        CACHE --persist --sharing shared /go
    
        COPY go.mod go.sum .
        RUN go mod download
    
    src:
        FROM +deps
    
        COPY . .
        RUN go generate ./...```
    In the above example, the `go generate` step is populating the modcache with additional dependencies (because I use `go run <http://github.com/x/y|github.com/x/y>`). However, the cache in `/go/modcache` is only persisting the changes from the `deps` target causing the `src` step to redownload the dependencies every time the source changes.
    
    Do I need to add an additional `CACHE` command in the `src` target? Or am I misunderstanding how the sharing is supposed to work?