Earthly

If you run your earthly command with -i, it should drop you into an interactive shell where you can troubleshoot why compose is failing to come up… check the image architectures, file permissions, etc.

If you haven’t already, I’d read the <https://docs.earthly.dev/docs/guides/docker-in-earthly#limitations-of-docker-in-earthly|“Limitations of Docker in Earthly” section> and <https://docs.earthly.dev/docs/guides/integration|“Integration Testing” guide> from the Earthly docs

I would investigate this part as well. I would look at the earthly-dind image and see how they do that and try to do the same

can you share your docker compose file?

i'm guessing its the docker-ce etc

ADD bz-internal.ca.crt /usr/local/share/ca-certificates/bz-internal.ca.crt

RUN apt-get update && \
    apt-get upgrade -y && \
    apt-get install -y --no-install-recommends \
      wget iptables build-essential \
      nodejs npm ca-certificates curl \
      && \
      install -m 0755 -d /etc/apt/keyrings && \
      curl -fsSL <https://download.docker.com/linux/debian/gpg> -o /etc/apt/keyrings/docker.asc &amp;&amp; \
      chmod a+r /etc/apt/keyrings/docker.asc &amp;&amp; \
# Add the repository to Apt sources:
    echo \
    "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] <https://download.docker.com/linux/debian> \
    $(. /etc/os-release &amp;&amp; echo "$VERSION_CODENAME") stable" | \
    tee /etc/apt/sources.list.d/docker.list &gt; /dev/null &amp;&amp; \
    apt-get update &amp;&amp; \
    apt-get install -y docker-ce docker-ce-cli <http://containerd.io|containerd.io> &amp;&amp; \
    update-ca-certificates &amp;&amp; \
    npm i -g snyk &amp;&amp; \
    
    # Clean up apt
    apt-get remove -y wget &amp;&amp; \
    apt-get autoremove -y &amp;&amp; \
    rm -rf /var/lib/apt/lists/*```

May be you can share at least the base image that was used to build http://eu.gcr.io/bz-kube/bluezinc/base/8-sdk|eu.gcr.io/bz-kube/bluezinc/base/8-sdk? That might help pointing in some direction

you're right, i've tried it with the ms one locally and its fine...

In my mind, the issue is in the image http://eu.gcr.io/bz-kube/bluezinc/base/8-sdk|eu.gcr.io/bz-kube/bluezinc/base/8-sdk . Unfortunately, I don't have access to it to try this hypothesis

your example works fine locally for me too

    FROM +src
    DO <http://github.com/earthly/lib+INSTALL_DIND|github.com/earthly/lib+INSTALL_DIND>
    WORKDIR /build   
    COPY docker-compose.yaml .
    COPY ../scripts+sql-image/data .docker-data/mssql
    COPY ci ci
    WITH DOCKER --compose docker-compose.yaml
        RUN echo "hello" &amp;&amp; dotnet test Caseflow.All.sln
    END```

Earthfile isn't public, so can't share currently... its a reasonably complex build process i'm trying to implement... just to validate my approach....

I'm

1. building a sql server volume containing and mdf file

Agreed, I tried a very simple exemple that I assume reproduce what you describe.

test:
<http://FROMmcr.microsoft.com/dotnet/sdk:8.0|FROMmcr.microsoft.com/dotnet/sdk:8.0>
WITH DOCKER
RUNdocker run alpine echo hello from docker in earthly
END

On earthly on colima on apple silicon S3

I run

$ earthly -P +test

And I get a successful result, as you can see in -> https://cloud.earthly.dev/builds/14553ee7-08c6-42c7-8e9f-10c215019505

I bet that the issue is in some part of your setup that we did not see yet.

something I can run like earthly -P <http://github.com/chrismcv/xxxxxxx+target|github.com/chrismcv/xxxxxxx+target>

<@U077G761B2Q> I run Earthly + colima on an apple silicon mac - is your Earthfile public? if not would you be willing to publish to github a quick PoC I can run over here?

yeah

dotnet-sdk image ? Like this one http://mcr.microsoft.com/dotnet/sdk:8.0|mcr.microsoft.com/dotnet/sdk:8.0 ?

I'm running my own, I was using a dotnet-sdk image.

The following helps:

DO <http://github.com/earthly/lib+INSTALL_DIND|github.com/earthly/lib+INSTALL_DIND>```
but doesn't get me anywhere useful:
```   +integration-test *failed* | ERROR: for bz-sql  Cannot start service mssql: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe: unknown
   +integration-test *failed* | Creating bz-redis     ... error

   +integration-test *failed* | ERROR: for bz-redis  Cannot start service redis: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe: unknown
   +integration-test *failed* | Creating bz-search    ... error```

Ahhhh, you mean you are running a target from earthly repository, like earthly -P <http://github.com/earthly/earthly/examples/integration-test+integration-test|github.com/earthly/earthly/examples/integration-test+integration-test> ?

I thought you were running some integration test in some project of yours.

From what I can see in the earthly repo, there are only three targets called integration-test

examples/integration-test/Earthfile:integration-test:
examples/integration-test/Earthfile-    FROM +project-files
examples/integration-test/Earthfile-    COPY src src
examples/integration-test/Earthfile-    COPY docker-compose.yml ./
examples/integration-test/Earthfile-    WITH DOCKER --compose docker-compose.yml
examples/integration-test/Earthfile-        RUN while ! pg_isready --host=localhost --port=5432 --dbname=iso3166 --username=postgres; do sleep 1; done ;\
examples/integration-test/Earthfile-            sbt it:test
examples/integration-test/Earthfile-    END
examples/go/Earthfile:integration-test:
examples/go/Earthfile-    FROM +deps
examples/go/Earthfile-    COPY main.go .
examples/go/Earthfile-    COPY main_integration_test.go .
examples/go/Earthfile-    COPY docker-compose.yml ./
examples/go/Earthfile-    WITH DOCKER --compose docker-compose.yml
examples/go/Earthfile-        RUN CGO_ENABLED=0 go test <http://github.com/earthly/earthly/examples/go|github.com/earthly/earthly/examples/go>
examples/go/Earthfile-    END
examples/typescript-node/Earthfile:integration-test:
examples/typescript-node/Earthfile-    FROM +test-setup
examples/typescript-node/Earthfile-    COPY +build/dist dist
examples/typescript-node/Earthfile-    WITH DOCKER --load typescript-node-example:latest=+docker
examples/typescript-node/Earthfile-        RUN docker run --rm -d -p 8080:8080 typescript-node-example:latest &amp;&amp; npm run test:integration
examples/typescript-node/Earthfile-    END
examples/typescript-node/Earthfile-
examples/typescript-node/Earthfile-

Which one are you trying to run?

is this on the Earthly repo?

That's strange. On my Mac S3, also with colima, everything work well with earthly -P. Would you mind sharing an example to help reproduce the issue ?

hi, also getting an issue trying integration tests, using Mac M1, and colima.... earthly -P +integration-test the -P doesn't seem to be enough

   +integration-test *failed* | Repeating the failure error...
   +integration-test *failed* | --> WITH DOCKER (install deps)
   +integration-test *failed* | Warning: Docker-in-Earthly needs to be run as root user
   +integration-test *failed* | jq is missing. Attempting to install automatically.
   +integration-test *failed* | Reading package lists...
   +integration-test *failed* | E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)
   +integration-test *failed* | ERROR Earthfile:70:8
   +integration-test *failed* |       The command
   +integration-test *failed* |           WITH DOCKER (install deps)
   +integration-test *failed* |       did not complete successfully. Exit code 100
   +integration-test |       The command
   +integration-test |           WITH DOCKER RUN
   +integration-test |       did not complete successfully. Exit code 100```