"mmap 131072 bytes at (nil)" with 'stack test' when using hint (+ servant + hspec)

[jwaldmann]

Hi. We use hint, inside a servant app. We have hspec tests for the app. When I run them with stack test, I get

servant-tests: mmap 131072 bytes at (nil): Cannot allocate memory
servant-tests: Try specifying an address with +RTS -xm<addr> -RTS
servant-tests: internal error: m32_allocator_init: Failed to map
    (GHC version 8.10.4 for x86_64_unknown_linux)

The error does not happen when I call the executable (.stack-work/dist/x86_64-linux/Cabal-3.2.1.0/build/servant-tests/servant-tests) directly.

I could look into this more closely, but I wanted to ask first whether you've seen this before, as this would help me isolate the cause.

[gelisam]

That doesn't ring a bell, no, but googling found a few hits:

https://gitlab.haskell.org/ghc/ghc/-/issues/19421
sol/doctest#271

They're both in the doctest project, which makes sense because doctest is a wrapper around the ghci executable, which is a wrapper around the ghc library, and hint is also a wrapper around the ghc library. I am thus guessing that the bug is ultimately in ghc itself.

[jwaldmann]

Thanks for the pointer. I will try to reduce my test case. (It's https://gitlab.imn.htwk-leipzig.de/autotool/all0/-/blob/master/server-implementation-servant/test/ApiSpec.hs#L207 but that pulls in a lot of depencies.)

[gelisam]

Since the documentation for the -xm flag mentioned by the error message says that it's for fixing OS-specific bugs, could you please share your OS version and system architecture? It will hopefully help others who stumble upon this issue in the future.

That documentation also says that the flag is only useful in GHCi, so I guess it's not surprising that this specific symptom only occurs with interpreted code and not when running the same code inside an executable.

Since the code you linked to says it tests a timeout, I have a hypothesis. Perhaps the code which is interrupted by the timeout is a busy loop which allocates more and more memory, and thus uses up the portion of the lower 2Gb which the documentation says the RTS must use? What does the memory profile of your program look like while waiting for the timeout to trigger?

[jwaldmann]

Fedora GNU/Linux on x86_64

the code which is interrupted by the timeout is a busy loop which allocates more and more memory,

ah yes! something like that is happening, it evaluates test = not test https://gitlab.imn.htwk-leipzig.de/autotool/all0/-/blob/master/server-implementation-servant/test/ApiSpec.hs#L215 (certainly "busy" but should not allocate?)

[jwaldmann]

I isolated this test case https://gitlab.imn.htwk-leipzig.de/autotool/all0/-/tree/master/hint-check but I give up (on debugging) for now, as it seems a Heisenbug to me.

[gelisam]

"Could not load module Data.ByteString, it is a member of the hidden package bytestring-0.10.12.0" sounds a lot like #128! That symptom should be pretty consistent though, you're saying it disappears when you try to debug it? What kind of debugging actions are causing it to disappear?

[yaitskov]

I hit same issues recently after upgrading nixpkgs from release-20.09 to release-21.05 same ghc-8.10.4. The bug is reproducible only on Linux when I launch Haskell interpreter and MacOS BigSur is ok.
My current workaround is to set -xm with something.

my-app +RTS -xm50 -RTS ...

Once xm is mentioned the failure doesn't happen.
I want to set this flag at compile time with -with-rtsopts, but
app fails and complains about unrecognized flag!

my-app: unexpected RTS argument: "-xm50"

P.S. my case happens not during tests, but at regular program run.