Build failed because docker is too old or is not working properly: json.decoder.JSONDecodeError #2074

mcleantom · 2024-11-06T15:22:41Z

Description

I am running cibuildwheel version 2.21.3 and I am trying to see if my project will compile for linux wheels, from windows, using the command:

cibuildwheel --platform linux --debug-traceback

Which fails with the error

Build failed because docker is too old or is not working properly.

Because of a json decode error:


     _ _       _ _   _       _           _
 ___|_| |_ _ _|_| |_| |_ _ _| |_ ___ ___| |
|  _| | . | | | | | . | | | |   | -_| -_| |
|___|_|___|___|_|_|___|_____|_|_|___|___|_|

cibuildwheel version 2.21.3

Build options:
  platform: linux
  allow_empty: False
  architectures: x86_64
  build_selector:
    build_config: cp*
    skip_config: cp*-musllinux* pp*
    requires_python: >=3.11
    prerelease_pythons: False
    free_threaded_support: False
  output_dir: C:\Users\tom.mclean\src\Nemo\wheelhouse
  package_dir: C:\Users\tom.mclean\src\Nemo
  test_selector:
    skip_config:
  before_all:
    rm -rf vcpkg
    rm -rf vcpkg_installed
    yum -y install curl zip unzip tar
    git clone https://github.com/microsoft/vcpkg.git
    ./vcpkg/bootstrap-vcpkg.sh
    ./vcpkg/vcpkg install
  before_build:
  before_test:
  build_frontend: None
  build_verbosity: 0
  config_settings:
  container_engine: docker
  dependency_constraints: pinned
  environment:
  manylinux_images:
    x86_64: quay.io/pypa/manylinux_2_28_x86_64:2024.10.07-1
    i686: manylinux_2_28
    pypy_x86_64: quay.io/pypa/manylinux_2_28_x86_64:2024.10.07-1
    aarch64: quay.io/pypa/manylinux_2_28_aarch64:2024.10.07-1
    ppc64le: quay.io/pypa/manylinux_2_28_ppc64le:2024.10.07-1
    s390x: quay.io/pypa/manylinux_2_28_s390x:2024.10.07-1
    pypy_aarch64: quay.io/pypa/manylinux_2_28_aarch64:2024.10.07-1
    pypy_i686: manylinux_2_28
  musllinux_images:
    x86_64: quay.io/pypa/musllinux_1_2_x86_64:2024.10.07-1
    i686: quay.io/pypa/musllinux_1_2_i686:2024.10.07-1
    aarch64: quay.io/pypa/musllinux_1_2_aarch64:2024.10.07-1
    ppc64le: quay.io/pypa/musllinux_1_2_ppc64le:2024.10.07-1
    s390x: quay.io/pypa/musllinux_1_2_s390x:2024.10.07-1
    armv7l: quay.io/pypa/musllinux_1_2_armv7l:2024.10.07-1
  repair_command: auditwheel repair -w {dest_dir} {wheel}
  test_command:
  test_extras:
  test_requires:

Cache folder: C:\Users\tom.mclean\AppData\Local\pypa\cibuildwheel\Cache

Here we go!

Starting container image quay.io/pypa/manylinux_2_28_x86_64:2024.10.07-1...

info: This container will host the build for cp311-manylinux_x86_64, cp312-manylinux_x86_64, cp313-manylinux_x86_64...
+ docker version -f '{{json .}}'

                                                                       ✕ 0.14s
Error: Build failed because docker is too old or is not working properly.

Traceback (most recent call last):
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\oci_container.py", line 105, in _check_engine_version
    version_info = json.loads(version_string.strip())
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\__main__.py", line 49, in main
    main_inner(global_options)
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\__main__.py", line 184, in main_inner
    build_in_directory(args)
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\__main__.py", line 351, in build_in_directory
    platform_module.build(options, tmp_path)
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\linux.py", line 460, in build
    with OCIContainer(
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\oci_container.py", line 230, in __enter__
    _check_engine_version(self.engine)
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\oci_container.py", line 145, in _check_engine_version
    raise OCIEngineTooOldError(msg) from e
cibuildwheel.errors.OCIEngineTooOldError: Build failed because docker is too old or is not working properly.

The output of my docker version command is:

{
   "Client":{
      "CloudIntegration":"v1.0.35+desktop.10",
      "Version":"25.0.2",
      "ApiVersion":"1.44",
      "DefaultAPIVersion":"1.44",
      "GitCommit":"29cf629",
      "GoVersion":"go1.21.6",
      "Os":"windows",
      "Arch":"amd64",
      "BuildTime":"Thu Feb  1 00:24:09 2024",
      "Context":"default"
   },
   "Server":{
      "Platform":{
         "Name":"Docker Desktop 4.27.1 (136059)"
      },
      "Components":[
         {
            "Name":"Engine",
            "Version":"25.0.2",
            "Details":{
               "ApiVersion":"1.44",
               "Arch":"amd64",
               "BuildTime":"Thu Feb  1 00:23:17 2024",
               "Experimental":"false",
               "GitCommit":"fce6e0c",
               "GoVersion":"go1.21.6",
               "KernelVersion":"5.15.133.1-microsoft-standard-WSL2",
               "MinAPIVersion":"1.24",
               "Os":"linux"
            }
         },
         {
            "Name":"containerd",
            "Version":"1.6.28",
            "Details":{
               "GitCommit":"ae07eda36dd25f8a1b98dfbf587313b99c0190bb"
            }
         },
         {
            "Name":"runc",
            "Version":"1.1.12",
            "Details":{
               "GitCommit":"v1.1.12-0-g51d5e94"
            }
         },
         {
            "Name":"docker-init",
            "Version":"0.19.0",
            "Details":{
               "GitCommit":"de40ad0"
            }
         }
      ],
      "Version":"25.0.2",
      "ApiVersion":"1.44",
      "MinAPIVersion":"1.24",
      "GitCommit":"fce6e0c",
      "GoVersion":"go1.21.6",
      "Os":"linux",
      "Arch":"amd64",
      "KernelVersion":"5.15.133.1-microsoft-standard-WSL2",
      "BuildTime":"2024-02-01T00:23:17.000000000+00:00"
   }
}

So I am not sure why it is failing.

Build log

No response

CI config

No response

The text was updated successfully, but these errors were encountered:

mcleantom · 2024-11-06T16:10:10Z

I could run the command from inside WSL, but I thought this command had previously worked when ran from windows

mcleantom · 2024-11-06T19:35:15Z

Actually it should work from windows as per #1782 so not sure whats happening

Czaki · 2024-11-06T23:23:40Z

@mcleantom could you run the cibuildwheel using traceback-with-variables

$ traceback-with-variables cibuildwheel --platform linux --debug-traceback

It should add additional context to traceback to allow understand the problem.

mcleantom · 2024-11-07T08:46:44Z

@Czaki Hey, here is the output of that:

Traceback with variables (most recent call last):
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\pathlib.py", line 1059, in read_text
    return f.read()
      self = WindowsPath('C:/Users/tom.mclean/AppData/Local/miniconda3/envs/nemo/Scripts/cibuildwheel.EXE')
      encoding = 'locale'
      errors = None
      f = <_io.TextIOWrapper name='C:\\Users\\tom.mclean\\AppData\\Local\\miniconda3\\envs\\nemo\\Scripts\\cibuildwheel.EXE' mode='r' encoding='cp1252'>
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
      self = <encodings.cp1252.IncrementalDecoder object at 0x0000029F4C652950>(errors='strict')
      input = b'MZ\x90\x00\x03\x00\x00\x00\x04\x00\x00\x00\xff\xff\x00\x00\xb8\x00\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xf8\x00\x00\x00\x0e\x1f\xba\x0e\x00\xb4\t\xcd!\xb8\x01L\xcd!This program cannot be run in DOS mode.\r\r\n$\x00\x00\x00\x00\x00\x00\x00\xa3\xfa`v\xe7\x9b\x0e%\xe7\x9b\x0e%\xe7\x9b\x0e%t\xd5\x96%\xe6\x9b\x0e%\xfc\x06\xa5%\xc6\x9b\x0e%\xfc\x06\xa4%\x91\x9b\x0e%\xfc\x06\x90%\xee\x9b\x0e%\xee\xe3\x9d%\xe2\x9b\x0e%\xe7\x9b\x0f%\xb8\x9b\x0e%\xfc\x06\xa0%\xe6\x9b\x0e%\xfc\x06\x94%\xe6\x9b\x0e%\xfc\x06\x93%\xe6\x9b\x0e%Rich\xe7\x9b\x0e%\x00\x00\x00\x00\x00\x00\x00\x00\x0...\', \'\', sys.argv[0])\n    sys.exit(main())\nPK\x01\x02\x14\x00\x14\x00\x00\x00\x00\x00\xf5zfY\x1aa\x8bP\xc9\x00\x00\x00\xc9\x00\x00\x00\x0b\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80\x01\x00\x00\x00\x00__main__.pyPK\x05\x06\x00\x00\x00\x00\x01\x00\x01\x009\x00\x00\x00\xf2\x00\x00\x00\x00\x00'
      final = True
builtins.UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 2: character maps to <undefined>

Czaki · 2024-11-07T08:53:54Z

Hm. It looks like it invoke exe instead of a python file. What with:

$ traceback-with-variables cibuildwheel.__main__ --platform linux --debug-traceback

mcleantom · 2024-11-07T08:56:34Z

That output is:

(nemo) C:\Users\tom.mclean\src\Nemo>traceback-with-variables cibuildwheel.__main__ --platform linux --debug-traceback

     _ _       _ _   _       _           _
 ___|_| |_ _ _|_| |_| |_ _ _| |_ ___ ___| |
|  _| | . | | | | | . | | | |   | -_| -_| |
|___|_|___|___|_|_|___|_____|_|_|___|___|_|

cibuildwheel version 2.21.2

Build options:
  platform: linux
  allow_empty: False
  architectures: x86_64
  build_selector:
    build_config: cp*
    skip_config: cp*-musllinux* pp*
    requires_python: >=3.11
    prerelease_pythons: False
    free_threaded_support: False
  output_dir: C:\Users\tom.mclean\src\Nemo\wheelhouse
  package_dir: C:\Users\tom.mclean\src\Nemo
  test_selector:
    skip_config:
  before_all:
    rm -rf vcpkg
    rm -rf vcpkg_installed
    yum -y install curl zip unzip tar
    git clone https://github.com/microsoft/vcpkg.git
    ./vcpkg/bootstrap-vcpkg.sh
    ./vcpkg/vcpkg install
  before_build:
  before_test:
  build_frontend: None
  build_verbosity: 0
  config_settings:
  container_engine: docker
  dependency_constraints: pinned
  environment:
  manylinux_images:
    x86_64: quay.io/pypa/manylinux_2_28_x86_64:2024.10.01-1
    i686: manylinux_2_28
    pypy_x86_64: quay.io/pypa/manylinux_2_28_x86_64:2024.10.01-1
    aarch64: quay.io/pypa/manylinux_2_28_aarch64:2024.10.01-1
    ppc64le: quay.io/pypa/manylinux_2_28_ppc64le:2024.10.01-1
    s390x: quay.io/pypa/manylinux_2_28_s390x:2024.10.01-1
    pypy_aarch64: quay.io/pypa/manylinux_2_28_aarch64:2024.10.01-1
    pypy_i686: manylinux_2_28
  musllinux_images:
    x86_64: quay.io/pypa/musllinux_1_2_x86_64:2024.10.01-1
    i686: quay.io/pypa/musllinux_1_2_i686:2024.10.01-1
    aarch64: quay.io/pypa/musllinux_1_2_aarch64:2024.10.01-1
    ppc64le: quay.io/pypa/musllinux_1_2_ppc64le:2024.10.01-1
    s390x: quay.io/pypa/musllinux_1_2_s390x:2024.10.01-1
    armv7l: quay.io/pypa/musllinux_1_2_armv7l:2024.10.01-1
  repair_command: auditwheel repair -w {dest_dir} {wheel}
  test_command:
  test_extras:
  test_requires:

Cache folder: C:\Users\tom.mclean\AppData\Local\pypa\cibuildwheel\Cache

Here we go!

Starting container image quay.io/pypa/manylinux_2_28_x86_64:2024.10.01-1...

info: This container will host the build for cp311-manylinux_x86_64, cp312-manylinux_x86_64, cp313-manylinux_x86_64...
+ docker version -f '{{json .}}'

                                                                       ✕ 0.19s
Error: Build failed because docker is too old or is not working properly.

Traceback (most recent call last):
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\oci_container.py", line 105, in _check_engine_version
    version_info = json.loads(version_string.strip())
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\__main__.py", line 49, in main
    main_inner(global_options)
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\__main__.py", line 184, in main_inner
    build_in_directory(args)
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\__main__.py", line 351, in build_in_directory
    platform_module.build(options, tmp_path)
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\linux.py", line 460, in build
    with OCIContainer(
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\oci_container.py", line 230, in __enter__
    _check_engine_version(self.engine)
  File "C:\Users\tom.mclean\AppData\Local\miniconda3\envs\nemo\Lib\site-packages\cibuildwheel\oci_container.py", line 145, in _check_engine_version
    raise OCIEngineTooOldError(msg) from e
cibuildwheel.errors.OCIEngineTooOldError: Build failed because docker is too old or is not working properly.

Czaki · 2024-11-07T09:08:03Z

Ok. So this module do not overwrite traceback.print_exc(file=sys.stderr) function.

cibuildwheel/cibuildwheel/__main__.py

Line 58 in e18a6e9

traceback.print_exc(file=sys.stderr)

I need to think about other debug options..

mcleantom · 2024-11-07T09:26:05Z

I cloned the repo and ran it within pycharm debugger so that it picks up the cibuildwheel that I cloned and it ran fine. Not sure why its not working when I just run from the terminal

Even if I use pip install . inside the cloned repo, then run the terminal command it doesnt work.

I modified the code to print the version string variable and it says:

+ docker version -f '{{json .}}'
failed to get console mode for stdout: The handle is invalid.
{"Client":{"CloudIntegration":"v1.0.35+desktop.10","Version":"25.0.2","ApiVersion":"1.44","DefaultAPIVersion":"1.44","GitCommit":"29cf629","GoVersion":"go1.21.6","Os":"windows","Ar
ch":"amd64","BuildTime":"Thu Feb  1 00:24:09 2024","Context":"default"},"Server":{"Platform":{"Name":"Docker Desktop 4.27.1 (136059)"},"Components":[{"Name":"Engine","Version":"25.
0.2","Details":{"ApiVersion":"1.44","Arch":"amd64","BuildTime":"Thu Feb  1 00:23:17 2024","Experimental":"false","GitCommit":"fce6e0c","GoVersion":"go1.21.6","KernelVersion":"5.15.
133.1-microsoft-standard-WSL2","MinAPIVersion":"1.24","Os":"linux"}},{"Name":"containerd","Version":"1.6.28","Details":{"GitCommit":"ae07eda36dd25f8a1b98dfbf587313b99c0190bb"}},{"N
ame":"runc","Version":"1.1.12","Details":{"GitCommit":"v1.1.12-0-g51d5e94"}},{"Name":"docker-init","Version":"0.19.0","Details":{"GitCommit":"de40ad0"}}],"Version":"25.0.2","ApiVer
sion":"1.44","MinAPIVersion":"1.24","GitCommit":"fce6e0c","GoVersion":"go1.21.6","Os":"linux","Arch":"amd64","KernelVersion":"5.15.133.1-microsoft-standard-WSL2","BuildTime":"2024-02-01T00:23:17.000000000+00:00"}}

mcleantom · 2024-11-07T09:29:39Z

Seems to be related to docker/for-win#13891

Adding this:

version_string = version_string.strip("failed to get console mode for stdout: The handle is invalid.\n")

"fixes" it, but I guess I should downgrade/upgrade docker

Czaki · 2024-11-07T11:56:43Z

Hm.
Reading pointed issue suggest that failed to get console mode for stdout: The handle is invalid is printed to stderr.

Maybe we should prepare stderr handler that may handle it so it will not be rediretedet to stdout?

@henryiii?

mcleantom changed the title ~~Build failed because docker is too old or is not working properly.~~ Build failed because docker is too old or is not working properly: json.decoder.JSONDecodeError Nov 6, 2024

mcleantom closed this as completed Nov 6, 2024

mcleantom reopened this Nov 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build failed because docker is too old or is not working properly: json.decoder.JSONDecodeError #2074

Build failed because docker is too old or is not working properly: json.decoder.JSONDecodeError #2074

mcleantom commented Nov 6, 2024

mcleantom commented Nov 6, 2024

mcleantom commented Nov 6, 2024

Czaki commented Nov 6, 2024

mcleantom commented Nov 7, 2024

Czaki commented Nov 7, 2024

mcleantom commented Nov 7, 2024

Czaki commented Nov 7, 2024

mcleantom commented Nov 7, 2024 •

edited

Loading

mcleantom commented Nov 7, 2024 •

edited

Loading

Czaki commented Nov 7, 2024

Build failed because docker is too old or is not working properly: json.decoder.JSONDecodeError #2074

Build failed because docker is too old or is not working properly: json.decoder.JSONDecodeError #2074

Comments

mcleantom commented Nov 6, 2024

Description

Build log

CI config

mcleantom commented Nov 6, 2024

mcleantom commented Nov 6, 2024

Czaki commented Nov 6, 2024

mcleantom commented Nov 7, 2024

Czaki commented Nov 7, 2024

mcleantom commented Nov 7, 2024

Czaki commented Nov 7, 2024

mcleantom commented Nov 7, 2024 • edited Loading

mcleantom commented Nov 7, 2024 • edited Loading

Czaki commented Nov 7, 2024

mcleantom commented Nov 7, 2024 •

edited

Loading

mcleantom commented Nov 7, 2024 •

edited

Loading