Unsupported OP_QUERY command

[Adityak4201]

I have mongoDB 6.0. When I am trying to connect it from redash 8.0, I am getting error:

Unsupported OP_QUERY command: connectionStatus. The client driver may require an upgrade. For more details, see https://dochub.mongodb.org/core/legacy-opcode-removal.

After going through redash's code, I realized they are using pymongo 3.9.0 which only supports till mongo 4.2. Please upgrade the pymongo version to 4.3.3

[justinclift]

Interesting. Yeah, that pymongo version of 3.9.0 is even in the latest Redash source code:

redash/requirements_all_ds.txt

Line 9 in d63ef90

pymongo[tls,srv]==3.9.0

We're updating other dependencies over the last days and weeks anyway, so we might as well try a newer pymongo and see if it builds without needing other changes.

[justinclift]

@Adityak4201 I've just created a PR with that change, mostly to see if it passes our test suite. No idea if it'll work or not, but it's worth trying out just in case. 😄

[justinclift]

The newer PyMongo passed the current Redash test suite, though I'm not sure how much that actually tests the Mongo part of things. That being said, I've merged it the PR into our master branch, so it's currently re-running the test suite on there + it'll automatically build a new preview docker image:

https://hub.docker.com/r/redashcommunity/redash/tags

You might need to give it an hour or so to finish the tests and building, then you could try that docker build out with your Mongo version 6 data source and see if it works ok. 😄

[justinclift]

It looks like the updated docker hub image has already built, so you should be ok to try it out. 😄

[Adityak4201]

Hey @justinclift. Thanks for the quick response. I will test it and let you know

[justinclift]

Thanks, please do. If the updated PyMongo library doesn't work properly, we'll probably need to change the version back to the older one until we can look at it properly.

[Adityak4201]

I have been trying to run the https://hub.docker.com/r/redashcommunity/redash/tags with docker. I have been following https://redash.io/help/open-source/dev-guide/docker for running the application on docker.

But, when I am running the command docker-compose run --rm server create_db, I am getting the following error:

Starting redash_redis_1 ... done
Starting redash_postgres_1 ... done
Traceback (most recent call last):
File "/app/manage.py", line 6, in
from redash.cli import manager
File "/app/redash/init.py", line 56, in
import_query_runners(settings.QUERY_RUNNERS)
File "/app/redash/query_runner/init.py", line 445, in import_query_runners
import(runner_import)
File "/app/redash/query_runner/dynamodb_sql.py", line 10, in
from dql import Engine, FragmentEngine
File "/usr/local/lib/python3.7/site-packages/dql/init.py", line 6, in
from .cli import DQLClient
File "/usr/local/lib/python3.7/site-packages/dql/cli.py", line 21, in
from .engine import FragmentEngine
File "/usr/local/lib/python3.7/site-packages/dql/engine.py", line 47, in
from .grammar import line_parser, parser
File "/usr/local/lib/python3.7/site-packages/dql/grammar/init.py", line 37, in
from .query import (
File "/usr/local/lib/python3.7/site-packages/dql/grammar/query.py", line 45, in
from .parsed_primitives import primitive as parsed_primitive
File "/usr/local/lib/python3.7/site-packages/dql/grammar/parsed_primitives.py", line 30, in
integer = pyparsing_common.signedInteger
AttributeError: type object 'pyparsing_common' has no attribute 'signedInteger'

[justinclift]

Interesting. It looks like an error happening in the DynamoDB connection library. Looking at that library on GitHub (https://github.com/stevearc/dql), it seems pretty much dead or close to it.

My first thought is we could just remove the DynamoDB library support for now, in order to get things working.

But it might actually be something else (simple) instead.

@gaecoli Would you have a few minutes to take a look at this?

[gaecoli]

Interesting. It looks like an error happening in the DynamoDB connection library. Looking at that library on GitHub (https://github.com/stevearc/dql), it seems pretty much dead or close to it.

My first thought is we could just remove the DynamoDB library support for now, in order to get things working.

But it might actually be something else (simple) instead.

@gaecoli Would you have a few minutes to take a look at this?

Tonight, i will look at this. I'm busy right now. Sorry.

[justinclift]

Tonight is completely fine, thank you. 😄

[gaecoli]

I have been trying to run the https://hub.docker.com/r/redashcommunity/redash/tags with docker. I have been following https://redash.io/help/open-source/dev-guide/docker for running the application on docker.

But, when I am running the command docker-compose run --rm server create_db, I am getting the following error:

Starting redash_redis_1 ... done Starting redash_postgres_1 ... done Traceback (most recent call last): File "/app/manage.py", line 6, in from redash.cli import manager File "/app/redash/init.py", line 56, in import_query_runners(settings.QUERY_RUNNERS) File "/app/redash/query_runner/init.py", line 445, in import_query_runners import(runner_import) File "/app/redash/query_runner/dynamodb_sql.py", line 10, in from dql import Engine, FragmentEngine File "/usr/local/lib/python3.7/site-packages/dql/init.py", line 6, in from .cli import DQLClient File "/usr/local/lib/python3.7/site-packages/dql/cli.py", line 21, in from .engine import FragmentEngine File "/usr/local/lib/python3.7/site-packages/dql/engine.py", line 47, in from .grammar import line_parser, parser File "/usr/local/lib/python3.7/site-packages/dql/grammar/init.py", line 37, in from .query import ( File "/usr/local/lib/python3.7/site-packages/dql/grammar/query.py", line 45, in from .parsed_primitives import primitive as parsed_primitive File "/usr/local/lib/python3.7/site-packages/dql/grammar/parsed_primitives.py", line 30, in integer = pyparsing_common.signedInteger AttributeError: type object 'pyparsing_common' has no attribute 'signedInteger'

Please show me how do you connect it.

[gaecoli]

Interesting. It looks like an error happening in the DynamoDB connection library. Looking at that library on GitHub (https://github.com/stevearc/dql), it seems pretty much dead or close to it.

My first thought is we could just remove the DynamoDB library support for now, in order to get things working.

But it might actually be something else (simple) instead.

@gaecoli Would you have a few minutes to take a look at this?

I think our docker image not build dql==0.6.2 in it. Thus causing the situation. @justinclift

[justinclift]

Oh. I wonder if our docker image build is not using the requirements_all_ds.txt during build then...

[gaecoli]

Oh. I wonder if our docker image build is not using the requirements_all_ds.txt during build then...

Yes, you understand me.

[justinclift]

Cool. We need to fix that. 😄

@junnplus @hazmeister It looks like our preview images on Docker Hub aren't being built with all of the libraries from requirements_all_ds.txt. Any idea how to get them included?

[justinclift]

Hmmm, it looks like GitHub now enforces a very narrow "maximum width" for wiki pages, making them useless for tables. That's not good. 😦

[justinclift]

I think there's a cut-n-paste mistake here too:

redash/redash/settings/__init__.py

Lines 167 to 174 in d2ac245

	REFERRER_POLICY = os.environ.get(
	"REDASH_REFERRER_POLICY", "strict-origin-when-cross-origin"
	)
	# Whether and how to send Feature-Policy response headers. Defaults to
	# an empty value.
	# See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Feature-Policy
	# for more information.
	FEATURE_POLICY = os.environ.get("REDASH_REFERRER_POLICY", "")

Both REFERRER_POLICY and FEATURE_POLICY are being loaded from the environment variable REDASH_REFERRER_POLICY, even though they have different uses and take different values. It probably hasn't been spotted before due to the default value for each of them being ok.

That FEATURE_POLICY one should probably be using a REDASH_FEATURE_POLICY environment variable instead.

[gaecoli]

Yes. It will be fixed.

…

On Jun 20, 2023, at 18:47, Aditya Khandelwal ***@***.***> wrote: Also, while I have been trying to run docker image for redashcommunity/redash:preview, I am getting error: 2023-06-20 10:43:44.157 UTC [1] FATAL: database files are incompatible with server 2023-06-20 10:43:44.157 UTC [1] DETAIL: The data directory was initialized by PostgreSQL version 12, which is not compatible with this version 15.3. I think postgresql version needs to be cross checked again. — Reply to this email directly, view it on GitHub <#59 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALBBQNGDJG4OJRDOM3IOOBTXMF5UBANCNFSM6AAAAAAZLXQQD4>. You are receiving this because you were mentioned.

[justinclift]

@jezdez Back in the day (~2019) you added a commit for Redash with this code fragment in it:

redash/redash/settings/__init__.py

Lines 167 to 174 in d2ac245

	REFERRER_POLICY = os.environ.get(
	"REDASH_REFERRER_POLICY", "strict-origin-when-cross-origin"
	)
	# Whether and how to send Feature-Policy response headers. Defaults to
	# an empty value.
	# See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Feature-Policy
	# for more information.
	FEATURE_POLICY = os.environ.get("REDASH_REFERRER_POLICY", "")

That code fragment seems to have a cut-n-paste typo in it.

Any idea if FEATURE_POLICY should be getting picked up from REDASH_FEATURE_POLICY rather than from REDASH_REFERRER_POLICY?

Not sure if you remember this code in question, but figured it doesn't hurt to ask... 😄

[jezdez]

Yeah, definitely a typo, REDASH_FEATURE_POLICY is the correct one

[justinclift]

Awesome, thanks heaps @jezdez. 😄

It'll be fixed in #64, which is running through the CI tests now.

[hazmeister]

Cool. We need to fix that. 😄

@junnplus @hazmeister It looks like our preview images on Docker Hub aren't being built with all of the libraries from requirements_all_ds.txt. Any idea how to get them included?

Any ideas what ds stands for? I'm a few days without access to a proper machine- can you pip install it manually using docker exec? I seem to recall python needs updating- perhaps that version only installs on a newer python base?

[justinclift]

DS means "Data Source". 😄

I can probably take a look at it tomorrow. Badly need sleep at the moment after being up all night again. 🤦‍♂️

[gaecoli]

DS means "Data Source". 😄

I can probably take a look at it tomorrow. Badly need sleep at the moment after being up all night again. 🤦‍♂️

We can move ds lib to requirement.txt. Just a idea.

[gaecoli]

Cool. We need to fix that. 😄
@junnplus @hazmeister It looks like our preview images on Docker Hub aren't being built with all of the libraries from requirements_all_ds.txt. Any idea how to get them included?

Any ideas what ds stands for? I'm a few days without access to a proper machine- can you pip install it manually using docker exec? I seem to recall python needs updating- perhaps that version only installs on a newer python base?

When you build images with docker, i want to know what args are you used?

[justinclift]

In theory, it should be done by this:

https://github.com/RedashCommunity/redash/blob/master/.github/workflows/docker-preview.yml

That file seems to be mostly pre-defined GitHub Action er... "actions". I'm not sure if the docker image build is being done by docker/setup-buildx-action@v2 or docker/build-push-action@v4 though.

Those actions have docs:

• https://github.com/docker/setup-buildx-action
• https://github.com/docker/build-push-action/

But after skimming those, it's still unclear if the build action is just directly running through the Dockerfile to create the image, or perhaps it's using the Makefile.

If it's building the image directly using the Dockerfile, then I thought it would be automatically using requirements_all_ds.txt already:

redash/Dockerfile

Lines 93 to 94 in d844f20

	COPY requirements_all_ds.txt ./
	RUN if [ "x$skip_ds_deps" = "x" ] ; then pip install -r requirements_all_ds.txt ; else echo "Skipping pip install -r requirements_all_ds.txt" ; fi

That's supposed to install all of the data source dependencies unless the variable "skip_ds_deps" is set to any value and passed to it.

---

When I run make by itself in my local Redash repo, it executes:

COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose build

That's the compose_build target in the Makefile:

redash/Makefile

Lines 3 to 4 in d844f20

	compose_build:
	COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose build

Which will be using the docker-compose.yml file:

https://github.com/RedashCommunity/redash/blob/d844f2086663d75c53a204e7311417f2988f7e79/docker-compose.yml

Which in turn will build the docker image from the Dockerfile if it's not already created by the GitHub actions run this time around.

To me, I'd think this would also be including all of the data source library dependencies already as per the above. It uses the same Dockerfile for the build.

---

Alternatively, there's the build process available through yarn instead:

redash/package.json

Line 10 in d844f20

"build": "yarn clean && yarn build:viz && NODE_ENV=production webpack",

That seems to be for just the front end though, so is probably called by other stuff as part of the build.

---

We can move ds lib to requirement.txt.

Yeah, it's probably the fastest way to get to a working result, while we're still getting a better idea of how all the pieces fit together.

a) Make it work <-- we're still here
b) Optimise it

😉

[justinclift]

Initial draft PR created. No idea if it'll pass our CI though. We may need to drop some of the data sources which pip (in my local install) complained about.

Lets see how it goes... 😄

[justinclift]

@Adityak4201 In the main getredash/redash repo we have a potential update for this almost ready. But I'd like someone that uses MongoDB 6 to try it and make sure the data source is indeed working ok (once the update is merged).

Is that something you'd be ok doing?

[Adityak4201]

@justinclift I have checked the connection to mongoDB 6 from getredash/redash and it was successful. I have successfully built dashboard also.

Thanks for the quick update 😄