Earthly

No problem, hope it works out well!

cool! thanks again. I'll give it a shot. my steps are pretty much just a COPY with nothing actually being built on each iteration of the loop so i don't think it will be too bad, but it is cool to know that that is configurable at the daemon level.

buildkit has a max number of steps it can run in parallel (configurable at the daemon) and will queue steps if it needs to. In some cases it can struggle if the load is too high, especially if those steps demand a lot of CPU/mem. I would give give it a try and see how it performs first

i guess a better follow up question, how would i set those limits lol

nice, agreed. that would be a fitting test for my understanding of build.

I've done this with docker in the past and the daemon is pretty good about queing things or waiting for other tasks to finish.

if i do a bunch of these jobs in parallel does the earthly buildkit try to throttle or limit itself so that it doesn't completely brick the host it is running on?

I might consider a separate target and call it with BUILD , it may be faster due to the parallelism

Cool. would i call another job/stage with build args from within the for loop or should i put a SAVE IMAGE directive in the body of the for loop?

It sounds like the FOR statement would be well suited for this. Hard to say for sure without understanding more about your particular use case, but I would give it a try

I guess to be more specific. Using a list of versions in a text file, can I create and push an image for each line in said file using earthly?

Thank you so much for the clarification.

My last question is an odd one.

I have a rather large number of "base" containers (50+) that contain an app. These are built outside my pipeline. I need to drop in a simple webserver that i've built that invokes the app in those base containers.

I don't need to rebuild my webserver for each container, i just need to essentially copy the binary in, commit the image, and push with a version tag.

Does that sound like something earthly can do? i see the For loop, but i would essentially need to read in a list of versions and push an image for each. since that list could be variable, i'm not sure how the for loop would handle that as artifacts within for loops seem to not be shared.

Yeah, probably

Gotcha. and i suppose that with regards to parallelism and what level of complexity is needed, that's going to depend on the needs my project.

First part is accurate understanding.

For the second part, you are right. I was talking about output (artifacts locally, images pushed to registry).

So when you COPY an artifact, that would trigger building that target, and making the artifact available in the target that invokes COPY, but if you want to save the artifact with SAVE ARTIFACT .. AS LOCAL, it needs to be referred to with a BUILD.

Ok so on the first point, if i had an +all block or something that had dependencies on other stages, those stages would get built in parallel if they were called with BUILD, but if i did not specify BUILD those stages would only get invoked/ran when it reaches a reference to the stage in something like a COPY statement? is this an accurate understanding?

But on the second statement, you're saying that if I invoke a stage that calls several jobs that would save artifacts, those artifacts would not get "output" (either locally or as an image) without using BUILD ? would artifacts in the pipeline still be available though? A simple example being a job that creates a scratch container with a golang binary. You have one stage that builds the binary and the other that copies it into the container.

In the example, I don't need a build directive, i can use COPY. That's where i get confused.

For me, the important detail is that only targets referred to by a BUILD is considered when doing SAVE IMAGE and SAVE ARTIFACT

One important benefit to the BUILD statement is that it invokes in parallel. So for example, these would run at the same time:

BUILD +service-b```

:wave: Hello, team! I'm looking for some clarity around the BUILD directive in an earthly file. I'm still operating with docker brain here. I'm really just unsure when or why i would need to use it.

If a DAG is getting creating under the hood that would invoke a dependent target when i invoke a target-ref, why would i need and explcit call to BUILD?