[bitnami/spark] Unable to run pyspark interactive shell

[jonatronblah]

Name and Version

bitnami/spark:3.3

What architecture are you using?

amd64

What steps will reproduce the bug?

Forgive me as I am very new to spark, but could not solve this issue with google.
Using the following docker compose file:

version: '2'

services:
  spark:
    image: docker.io/bitnami/spark:3.3
    environment:
      - SPARK_MODE=master
      - SPARK_RPC_AUTHENTICATION_ENABLED=no
      - SPARK_RPC_ENCRYPTION_ENABLED=no
      - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
      - SPARK_SSL_ENABLED=no
      - SPARK_USER=spark
    ports:
      - '8080:8080'
  spark-worker:
    image: docker.io/bitnami/spark:3.3
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER_URL=spark://spark:7077
      - SPARK_WORKER_MEMORY=1G
      - SPARK_WORKER_CORES=1
      - SPARK_RPC_AUTHENTICATION_ENABLED=no
      - SPARK_RPC_ENCRYPTION_ENABLED=no
      - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
      - SPARK_SSL_ENABLED=no
      - SPARK_USER=spark

Attempting to run pyspark interactive shell with the following command: ./bin/pyspark

What is the expected behavior?

Expect the pyspark interactive shell to run.

What do you see instead?

Always receive the following error: "Error: pyspark does not support any application options."

Additional information

I can submit pyspark jobs successfully, and activate the scala interactive shell just fine. Thanks for any help.

[marcosbc]

Hi @jonatronblah, you seem to be right as the tool does not work with the current Bitnami container image for Spark 3.3.

However, I tried the upstream apache/spark container for 3.3 and it does not work either (although the error is similar). But it seems to work fine for the upstream 3.4 image.

We are already working on adding the new 3.4 version soon to Bitnami, so once it is released, we can check whether the issue is resolved (or not), and escalate it if needed.

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[eugeniatso]

I had the same issue. I was able to solve the issue by putting the commands

pip install pyspark --> push enter
python --> push enter
import pyspark --> push enter
pyspark --> push enter

I hope this helps you too. If not, sorry :/ I'm pretty new to this too.

[thereallisabui]

Just wanna report that it also did not work for 3.4 and 3.5. This is not 'solved'.

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[dermoritz]

i have the same issue - any workarounds?

[ArmirKS]

currently facing the same issue

[dermoritz]

an workaround i found here: #38139 (comment)