E

Challenges with SSH and CI Authentication

Summary

The text addresses challenges in executing commands that require SSH access to private module repositories, both locally and in GitHub Actions. It proposes solutions like passing passwords as arguments, using conditionals, and adjusting GitHub authentication settings. The author seeks alternative methods for improving CI authentication and shares a snippet for configuring GO environment parameters and caches. There is a discussion about embedding HTTPS git credentials in Docker layers, clarifying that credentials may appear in commands but are not saved. The author is interested in setting up a Go build within actions and asks about a repository for common Earthly functions, as well as the necessity of the --keep-ts option for caching in Earthly, noting that Golang uses timestamps for caching, and without it, static timestamps can cause issues.

Status
resolved
Tags
    Source
    #earthly
      i

      ingwar

      7/31/2024

      and without it Earthly sets static timestamp on them and caches dont work (actually they break stuff, not working wouldynt be that bad)..

      i

      ingwar

      7/31/2024

      Because golang uses timestamps on files for caching purposes..

      b

      brandon356

      7/31/2024

      Also I am curious on the reason --keep-ts is required when using a cache in Earthly

      b

      brandon356

      7/31/2024

      Thank you! This makes sense now. I've been trying to figure out how to set up a Go build inside actions and was surprised Rust was the only language-specific docs in Earthly. Is there a repo/marketplace for public/common Earthly functions like this somewhere?

      i

      ingwar

      7/31/2024

      RUN --secret GIT_USER=GITHUB_USER --secret GIT_PASS=GITHUB_TOKEN eval $cmd and that line has credentials but they are never saved (at least as you dont save them as part of cmd)

      i

      ingwar

      7/31/2024

      RUN git config --global credential.helper '!p() { printf "username=${GIT_USER}\\npassword=${GIT_PASS}"; }; p' that line is saved as layer, but it dont have credentials..

      i

      ingwar

      7/31/2024

      I wrote it so long ago that I had to loo whats there..

      i

      ingwar

      7/31/2024

      nooo.. actually not..

      i

      ingwar

      7/31/2024

      Actually yes.. but we always use it in our build images.. that are temporary..

      b

      brandon356

      7/31/2024

      The issue with this approach is that you embed the https git credentials in a Docker layer right? <@U02P31BPR6X>

      n

      nicholas.thomson

      6/18/2024

      Wow this is awesome. Thank you for snippets!

      i

      ingwar

      6/18/2024

      Cause compile just works instantaneously..

      i

      ingwar

      6/18/2024

      with that setup you dont need to do go mod download dance..

      i

      ingwar

      6/18/2024

      Only thing to remember (that we learned hard way) is that when you cpy something inside earthly and use cache you always need to add --keep-ts

      i

      ingwar

      6/18/2024

      And if your using GO then life changing thing for us was

      # Call example:
      # DO commons+KTB_GO_PREPARE
      KTB_GO_PREPARE:
          FUNCTION
          ENV GOCACHE=/go-cache
          ENV GOMODCACHE=/go-mod-cache
          ENV GOPRIVATE=<http://github.com/some|github.com/some>
          CACHE --sharing=shared --id=kbt_go_cache $GOCACHE
          CACHE --sharing=shared --id=kbt_go_mod_cache $GOMODCACHE```
      
      i

      ingwar

      6/18/2024

      CI depending from setup use plain or ssh..

      i

      ingwar

      6/18/2024

      Default config is ssh so and thats what most people use at local machines..

      i

      ingwar

      6/18/2024

      We have something like that..

      # Its mostly used for go programs that reach private repositories.
      # Call example:
      # DO commons+CONFIGURE_GITHUB --GITHUB_AUTH=ssh
      # DO commons+CONFIGURE_GITHUB --GITHUB_AUTH=plain
      CONFIGURE_GITHUB:
          FUNCTION
          ARG GITHUB_AUTH=ssh
          ENV GITHUB_AUTH=$GITHUB_AUTH
          IF [ "$GITHUB_AUTH" = "plain" ]
              RUN git config --global credential.helper '!p() { printf "username=${GIT_USER}\\npassword=${GIT_PASS}"; }; p'
          ELSE
              RUN git config --global url."<ssh://git@github.com/>".insteadOf <https://github.com/>
              RUN mkdir -p -m 0600 ~/.ssh &amp;&amp; ssh-keyscan <http://github.com|github.com> &gt;&gt; ~/.ssh/known_hosts
          END
      
      # Run any FUNCTION with github configured
      # Call example:
      # DO commons+RUN_WITH_GITHUB --cmd="git clone <https://github.com/some/repo>"
      RUN_WITH_GITHUB:
          FUNCTION
          ARG --required cmd
          ARG DEBUG_PIPELINE=false
          IF [ "$GITHUB_AUTH" = "plain" ]
              RUN --secret GIT_USER=GITHUB_USER --secret GIT_PASS=GITHUB_TOKEN eval $cmd
          ELSE
              RUN --ssh eval $cmd
          END
      
      # Run FUNCTION with github configured
      # Its mostly used for go programs that reach private repositories.
      # Call example:
      # DO commons+GO_WITH_GITHUB --cmd="go build"
      GO_WITH_GITHUB:
          FUNCTION
          DO +KTB_GO_PREPARE
          ARG --required cmd
          ARG DESTDIR=/build/output
          ARG DEBUG_PIPELINE=false
          ARG GO_TEST_FLAGS
          IF [ "$GITHUB_AUTH" = "plain" ]
              RUN --secret GIT_USER=GITHUB_USER --secret GIT_PASS=GITHUB_TOKEN \
                  eval $cmd
          ELSE
              RUN --ssh eval $cmd
          END```
      
      n

      nicholas.thomson

      6/17/2024

      I just resorted to passing the password as an ARG, and then using a conditional to call -ssh only if the ARG is not defined. It's a bit hacky, so still open to suggestions

      n

      nicholas.thomson

      6/17/2024

      Hey earthly folks :wave: I have a Earthfile with a target for running go mod download that needs to pass through -ssh so that go can access my private module repositories. Everything works great locally! Now when I try to run that same step as part of the build process in a Github action, earthly fails with failed: no SSH key "" forwarded from the client. I would expect this since GHA doesn't have an SSH socket running as far as I know. Is it possible to conditionally use SSH, or is there a better method for authenticating with private git repositories that can support being run in CI?