Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

failed to load plugin [ldap-auth] err: error loading module 'lualdap' from file '/usr/local/apisix/deps/lib/lua/5.1/lualdap.so':"failed to load plugin [ldap-auth] #218

Open
kenhys opened this issue Jul 1, 2022 · 6 comments

Comments

@kenhys
Copy link

kenhys commented Jul 1, 2022

Problem

When trying to test the existing plugin with apisix RPM package, it fails with loading module failure.
(Just to clarify packaged version of apisix works well with it)

Steps to reproduce

  1. Install apisix, apisix-base in RHEL compatible distribution
$ rpm -q apisix
apisix-2.14.1-0.el8.x86_64
$ rpm -q apisix-base
apisix-base-1.21.4.1.0-0.el8.x86_64
  1. Install perl-Test-Nginx from powertools repository
$ rpm -q perl-Test-Nginx
perl-Test-Nginx-0.29-2.el8.noarch
  1. clone apisix repository
  2. Modify t/APISIX.pm to allow additional lua_package_path lua_package_cpath to refer apisix RPM's bundled modules
  3. run prove to test plugin/example.t
TEST_NGINX_BINARY=/usr/bin/openresty prove -I. -I./t -r t/plugin/example.t
...
Test Summary Report
-------------------
t/plugin/example.t (Wstat: 0 Tests: 91 Failed: 30)
  Failed tests:  3-4, 10-11, 17-18, 24-25, 31-32, 42-49
                52-53, 59-60, 66-67, 73-74, 80-81, 87-88
  Parse errors: No plan found in TAP output
Files=1, Tests=91,  4 wallclock secs ( 0.02 usr  0.01 sys +  0.57 cusr  0.18 csys =  0.78 CPU)                                             
Result: FAIL

Expected

TEST_NGINX_BINARY=/usr/bin/openresty prove -I. -I./t -r t/plugin/example.t succeeds.

Additional Information

t/servroot/logs/error.log indicates a loading error like this:

2022/07/01 14:30:40 [error] 81507#81507: *2 [lua] plugin.lua:110: load_plugin(): failed to load plugin [ldap-auth] err: error loading module
 'lualdap' from file '/usr/local/apisix/deps/lib/lua/5.1/lualdap.so':
        /usr/lib64/libk5crypto.so.3: undefined symbol: EVP_KDF_ctrl, version OPENSSL_1_1_1b, context: init_worker_by_lua*
2022/07/01 14:30:40 [error] 81506#81506: *1 [lua] plugin.lua:110: load_plugin(): failed to load plugin [ldap-auth] err: error loading module
 'lualdap' from file '/usr/local/apisix/deps/lib/lua/5.1/lualdap.so':
        /usr/lib64/libk5crypto.so.3: undefined symbol: EVP_KDF_ctrl, version OPENSSL_1_1_1b, context: init_worker_by_lua*

Then, disabled ldap-auth from conf/config-default.yaml plugins: section, it succeeds, so it can be a temporary workaround to test it.

NOTE: I expected that /usr/lib64/* is used, but it seems that it is not true in this case (undefined symbol).

$ ldd /usr/local/apisix/deps/lib/lua/5.1/lualdap.so  |grep crypto
        libcrypto.so.1.1 => /usr/lib64/libcrypto.so.1.1 (0x00007fdbf489e000)
        libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00007fdbf3d1b000)
$ objdump -TC /usr/lib64/libcrypt.so.1.1.0 |grep EVP_KDF_
$ objdump -TC /usr/lib64/libk5crypto.so.3 | grep EVP_KDF_
0000000000000000      DF *UND*  0000000000000000  OPENSSL_1_1_1b EVP_KDF_ctrl
0000000000000000      DF *UND*  0000000000000000  OPENSSL_1_1_1b EVP_KDF_CTX_new_id
0000000000000000      DF *UND*  0000000000000000  OPENSSL_1_1_1b EVP_KDF_CTX_free
0000000000000000      DF *UND*  0000000000000000  OPENSSL_1_1_1b EVP_KDF_derive
$ objdump -TC /usr/local/openresty/openssl111/lib/libcrypto.so.1.1 | grep EVP_KDF

It seems that there is a mismatch between build time dependency and runtime dependency or something else.

@tzssangglass
Copy link
Contributor

maybe you need to install openldap-devel, like

yum install -y openldap-devel

@tzssangglass
Copy link
Contributor

openldap-devel is not provided with apisix, it needs to be installed by the user.

@spacewander
Copy link
Contributor

openldap-devel is expected to be installed if apisix is installed via yum, see

dep_ldap="openldap-devel"

@spacewander
Copy link
Contributor

The EVP_KDF_ctrl is from libk5crypto which is Red Hat specific, not the libcrypto we finally use. Maybe using the libcrypto from openresty-openssl111 when installing lua-ldap can fix this problem.

@kenhys
Copy link
Author

kenhys commented Jul 4, 2022

as @spacewander says, maybe the root of this issue is caused from:

luarocks make ./rockspec/apisix-master-${iteration}.rockspec --tree=/tmp/build/output/apisix/usr/local/apisix/deps --local

lualdap 1.2.6-1 depends on lua >= 5.1 (5.1-1 provided by VM)
gcc -O2 -fPIC -I/usr/local/openresty/luajit/include/luajit-2.1 -c src/lualdap.c -o src/lualdap.o -I/usr/include
gcc -shared -o lualdap.so src/lualdap.o -L/usr/lib64 -Wl,-rpath,/usr/lib64 -lldap -llber
Cloning into 'lua-resty-rocketmq'...
Note: switching to 'f5aca7d66b6323805aa13b26567c39042a837ed4'.

@soulbird
Copy link
Contributor

soulbird commented Jul 5, 2022

lualdap.so depends on libldap-2.4.so.2

ldd /usr/local/apisix/deps/lib/lua/5.1/lualdap.so | grep libldap
ibldap-2.4.so.2 => /usr/lib64/libldap-2.4.so.2 (0x00007fe82934a000)

The libldap-2.4.so.2 dynamic library was installed via openldap-devel when installing apisix. openldap-devel uses centos8's openssl, so the EVP_KDF* symbols are introduced.

apisix-base specifies the search path of the dynamic library through rpath at build time, via: https://github.com/api7/apisix-build-tools/blob/master/build-apisix-base.sh#L94. When apisix is ​​running, it will first search for this path when loading the dynamic library. This caused libldap-2.4.so.2 to be loaded with openresty-openssl instead of the system's openssl, thus resulting in the symbol not being found. For the search path of dynamic libraries, please refer to: https://man7.org/training/download/shlib_dynlinker_slides.pdf

Therefore, the solutions are as follows:

  1. Delete the installed openresty-openssl. When apisix is ​​running, it will continue to search for the system openssl. This method is rude
  2. Use source code to compile and install openldap, specify openssl as openresty-openssl when compiling. But unfortunately, openldap does not support specifying openssl directories. However I found a modified way here. This seems to work, and someone has tried it in the apisix issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants