AWS ECS Fargate + Alpine

Quick warning, this post is very technical. And probably boring.

TLDR

We run our primary API and many micro sites on top of AWS ECS Fargate, with an Elixir Docker image containing our application. Previously, that was an Alpine based image, but we have had to move back into the land of Debian due to an issue with musl vs glibc. One day, new deploys would fail to launch, never passing health checks and crashing. Eventually, they would pass, intermittently.

Bad Timing

A few days prior, we had bumped some dependency versions, including our Alpine Linux version. Seeing the startup error, it seemed logical to roll this back!

sys/unix/sys_signal_stack.c:101:sys_sigaltstack(): Internal error: Failed to set alternate signal stack
...
"StitchWorkflow.Release.ping_db"
...

On container startup, there was a low level error that meant the container failed to talk to our Database, forcing the ECS Task to crash. We never had any down time, technically, since our rolling deployment would eventually go green. Suspicious. Especially since this rollback didn’t work at all! Alpine 3.21 all the way through 3.24 were now failing, intermittently.

The hunt

Interestingly, this error actually has a stack trace, which was very handy! A quick search on Erlang’s Github for the file sys_signal_stack points us to the runtime function! Not to spoil the root cause too early, but a fix was just merged to main at the end of June, so look for a tagged older version (we are on OTP 27 at this time). Look, our exact error message!

static void sys_sigaltstack(void *ss_sp) {
    stack_t ss;

    ss.ss_sp = ss_sp;
    ss.ss_flags = 0;
    ss.ss_size = SIGSTKSZ;

    if (sigaltstack(&ss, NULL) < 0) {
        ERTS_INTERNAL_ERROR("Failed to set alternate signal stack");
    }
}

SIGSTKSZ is a compile-time value, and it hasn’t changed in any Alpine/musl versions. It has been 8192 since 2014, and still is!

The cause

This was very strange, and a lower level C or hardware problem then I was used to lately. So I asked my good friend Claude, and we found it.

echo "cpu wide-vector flags: $(grep -m1 '^flags' /proc/cpuinfo | grep -o 'avx512[a-z]*\|amx_[a-z]*' | sort -u | tr '\n' ' ')"
# ...and, only on failure, re-run under: strace -f -e trace=sigaltstack

The failing host came back as:

kernel: 6.1.174
cpu wide-vector flags: amx_bf amx_int amx_tile avx512 avx512bw avx512cd avx512dq avx512f avx512ifma avx512vbmi avx512vl

There it is: AMX. amx_tile etc. means an Intel Sapphire Rapids-class chip. And the killer number:

AMX tile register state alone is 8192 bytes in the CPU’s save area.
Add AVX-512 (~2.5 KB) plus the rest of the signal frame, and the kernel’s required minimum is ~11–12 KB.
musl handed the VM an 8192-byte stack. Kernel says “not enough” → sigaltstack rejected → VM aborts.

AVX-512 alone fits comfortably in 8192 (its frame is ~3.5 KB), which is why plain AVX-512 hosts booted green and only the AMX hosts blew up. That’s the entire source of the intermittency.

So AWS providing newer upgraded hardware caused our crash!