Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't push data to datanode with webhdfs #144

Open
GuillaumeDrillaud opened this issue Mar 10, 2023 · 1 comment
Open

Can't push data to datanode with webhdfs #144

GuillaumeDrillaud opened this issue Mar 10, 2023 · 1 comment

Comments

@GuillaumeDrillaud
Copy link

GuillaumeDrillaud commented Mar 10, 2023 •
edited
Loading

Here is what I can do :

  • I can write/read data using hdfs command line
  • I can list a dir using webhdfs (hdfscli and curl)

Here are what I can't do :

  • I can't push data using curl or hdfscli
  • I can't push data using the python module "hdfs"

To summarize, I can access namenode with anything, but i can't access datanode with webhdfs.

Here are some errors i get :

gdrillaud@nanw91137:~$ curl -v -i -X PUT -T /export/home/workingArea/data/platform/storage/storage_api/fastAPI/data/to_send/LN-1803_PN-5328_PI-1_SN-1936876_Year-2022_Jday-234_00-00UTC_01.segd  "http://10.20.141.17:9870/webhdfs/v1/Basalt-Files/?op=CREATE"
*   Trying 10.20.141.17:9870...
* Connected to 10.20.141.17 (10.20.141.17) port 9870 (#0)
> PUT /webhdfs/v1/Basalt-Files/?op=CREATE HTTP/1.1
> Host: 10.20.141.17:9870
> User-Agent: curl/7.74.0
> Accept: */*
> Content-Length: 173520544
> Expect: 100-continue
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 100 Continue
HTTP/1.1 100 Continue

* Mark bundle as not supporting multiuse
< HTTP/1.1 307 Temporary Redirect
HTTP/1.1 307 Temporary Redirect
< Date: Fri, 10 Mar 2023 14:45:10 GMT
Date: Fri, 10 Mar 2023 14:45:10 GMT
< Cache-Control: no-cache
Cache-Control: no-cache
< Expires: Fri, 10 Mar 2023 14:45:10 GMT
Expires: Fri, 10 Mar 2023 14:45:10 GMT
< Date: Fri, 10 Mar 2023 14:45:10 GMT
Date: Fri, 10 Mar 2023 14:45:10 GMT
< Pragma: no-cache
Pragma: no-cache
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< X-FRAME-OPTIONS: SAMEORIGIN
X-FRAME-OPTIONS: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
X-XSS-Protection: 1; mode=block
< Location: http://e26472ec19ff:9864/webhdfs/v1/Basalt-Files/?op=CREATE&namenoderpcaddress=namenode:9900&createflag=&createparent=true&overwrite=false
Location: http://e26472ec19ff:9864/webhdfs/v1/Basalt-Files/?op=CREATE&namenoderpcaddress=namenode:9900&createflag=&createparent=true&overwrite=false
< Content-Type: application/octet-stream
Content-Type: application/octet-stream
< Content-Length: 0
Content-Length: 0

* HTTP error before end of send, stop sending
<
* Closing connection 0

hdfscli upload "/export/home/workingArea/data/platform/storage/storage_api/fastAPI/data/to_send/LN-1803_PN-5328_PI-1_SN-1936876_Year-2022_Jday-234_00-00UTC_01.segd" "/Basalt-Files/" --alias=dev
ERROR   Error while uploading. Attempting cleanup.
Traceback (most recent call last):
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/client.py", line 650, in upload
    _upload(path_tuple)
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/client.py", line 581, in _upload
    self.write(_temp_path, wrap(reader, chunk_size, progress), **kwargs)
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/client.py", line 527, in write
    consumer(data)
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/client.py", line 516, in consumer
    res = self._request(
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/client.py", line 209, in _request
    return self._session.request(
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/requests/adapters.py", line 521, in send
    low_conn.endheaders()
  File "/usr/local/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f94877b1910>: Failed to establish a new connection: [Errno 111] Connection refused
ERROR   Unexpected exception.
Traceback (most recent call last):
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/config.py", line 195, in wrapper
    return func(*args, **kwargs)
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/__main__.py", line 257, in main
    client.upload(
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/client.py", line 661, in upload
    raise err
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/client.py", line 650, in upload
    _upload(path_tuple)
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/client.py", line 581, in _upload
    self.write(_temp_path, wrap(reader, chunk_size, progress), **kwargs)
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/client.py", line 527, in write
    consumer(data)
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/client.py", line 516, in consumer
    res = self._request(
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/hdfs/client.py", line 209, in _request
    return self._session.request(
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/requests/adapters.py", line 521, in send
    low_conn.endheaders()
  File "/usr/local/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/export/home/workingArea/data/.local/share/virtualenvs/storage-C30ZN5Fr/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f94877b1910>: Failed to establish a new connection: [Errno 111] Connection refused
@dedkot01
Copy link

This answer helped me: https://stackoverflow.com/a/64981792/12023127

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@GuillaumeDrillaud @dedkot01