failed to load plugin [ldap-auth] err: error loading module 'lualdap' from file '/usr/local/apisix/deps/lib/lua/5.1/lualdap.so':"failed to load plugin [ldap-auth]

[kenhys]

Problem

When trying to test the existing plugin with apisix RPM package, it fails with loading module failure.
(Just to clarify packaged version of apisix works well with it)

Steps to reproduce

1. Install apisix, apisix-base in RHEL compatible distribution

$ rpm -q apisix
apisix-2.14.1-0.el8.x86_64
$ rpm -q apisix-base
apisix-base-1.21.4.1.0-0.el8.x86_64

2. Install perl-Test-Nginx from powertools repository

$ rpm -q perl-Test-Nginx
perl-Test-Nginx-0.29-2.el8.noarch

4. clone apisix repository
5. Modify t/APISIX.pm to allow additional lua_package_path lua_package_cpath to refer apisix RPM's bundled modules
6. run prove to test plugin/example.t

TEST_NGINX_BINARY=/usr/bin/openresty prove -I. -I./t -r t/plugin/example.t
...
Test Summary Report
-------------------
t/plugin/example.t (Wstat: 0 Tests: 91 Failed: 30)
  Failed tests:  3-4, 10-11, 17-18, 24-25, 31-32, 42-49
                52-53, 59-60, 66-67, 73-74, 80-81, 87-88
  Parse errors: No plan found in TAP output
Files=1, Tests=91,  4 wallclock secs ( 0.02 usr  0.01 sys +  0.57 cusr  0.18 csys =  0.78 CPU)                                             
Result: FAIL

Expected

TEST_NGINX_BINARY=/usr/bin/openresty prove -I. -I./t -r t/plugin/example.t succeeds.

Additional Information

t/servroot/logs/error.log indicates a loading error like this:

2022/07/01 14:30:40 [error] 81507#81507: *2 [lua] plugin.lua:110: load_plugin(): failed to load plugin [ldap-auth] err: error loading module
 'lualdap' from file '/usr/local/apisix/deps/lib/lua/5.1/lualdap.so':
        /usr/lib64/libk5crypto.so.3: undefined symbol: EVP_KDF_ctrl, version OPENSSL_1_1_1b, context: init_worker_by_lua*
2022/07/01 14:30:40 [error] 81506#81506: *1 [lua] plugin.lua:110: load_plugin(): failed to load plugin [ldap-auth] err: error loading module
 'lualdap' from file '/usr/local/apisix/deps/lib/lua/5.1/lualdap.so':
        /usr/lib64/libk5crypto.so.3: undefined symbol: EVP_KDF_ctrl, version OPENSSL_1_1_1b, context: init_worker_by_lua*

Then, disabled ldap-auth from conf/config-default.yaml plugins: section, it succeeds, so it can be a temporary workaround to test it.

NOTE: I expected that /usr/lib64/* is used, but it seems that it is not true in this case (undefined symbol).

$ ldd /usr/local/apisix/deps/lib/lua/5.1/lualdap.so  |grep crypto
        libcrypto.so.1.1 => /usr/lib64/libcrypto.so.1.1 (0x00007fdbf489e000)
        libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00007fdbf3d1b000)

$ objdump -TC /usr/lib64/libcrypt.so.1.1.0 |grep EVP_KDF_
$ objdump -TC /usr/lib64/libk5crypto.so.3 | grep EVP_KDF_
0000000000000000      DF *UND*  0000000000000000  OPENSSL_1_1_1b EVP_KDF_ctrl
0000000000000000      DF *UND*  0000000000000000  OPENSSL_1_1_1b EVP_KDF_CTX_new_id
0000000000000000      DF *UND*  0000000000000000  OPENSSL_1_1_1b EVP_KDF_CTX_free
0000000000000000      DF *UND*  0000000000000000  OPENSSL_1_1_1b EVP_KDF_derive

$ objdump -TC /usr/local/openresty/openssl111/lib/libcrypto.so.1.1 | grep EVP_KDF

It seems that there is a mismatch between build time dependency and runtime dependency or something else.

[tzssangglass]

maybe you need to install openldap-devel, like

yum install -y openldap-devel

[tzssangglass]

openldap-devel is not provided with apisix, it needs to be installed by the user.

[spacewander]

openldap-devel is expected to be installed if apisix is installed via yum, see

apisix-build-tools/package-apisix.sh

Line 10 in 0d46617

dep_ldap="openldap-devel"

[spacewander]

The EVP_KDF_ctrl is from libk5crypto which is Red Hat specific, not the libcrypto we finally use. Maybe using the libcrypto from openresty-openssl111 when installing lua-ldap can fix this problem.

[kenhys]

as @spacewander says, maybe the root of this issue is caused from:

apisix-build-tools/utils/install-common.sh

Line 98 in 222ef2e

luarocks make ./rockspec/apisix-master-${iteration}.rockspec --tree=/tmp/build/output/apisix/usr/local/apisix/deps --local

lualdap 1.2.6-1 depends on lua >= 5.1 (5.1-1 provided by VM)
gcc -O2 -fPIC -I/usr/local/openresty/luajit/include/luajit-2.1 -c src/lualdap.c -o src/lualdap.o -I/usr/include
gcc -shared -o lualdap.so src/lualdap.o -L/usr/lib64 -Wl,-rpath,/usr/lib64 -lldap -llber
Cloning into 'lua-resty-rocketmq'...
Note: switching to 'f5aca7d66b6323805aa13b26567c39042a837ed4'.

[soulbird]

lualdap.so depends on libldap-2.4.so.2

ldd /usr/local/apisix/deps/lib/lua/5.1/lualdap.so | grep libldap
ibldap-2.4.so.2 => /usr/lib64/libldap-2.4.so.2 (0x00007fe82934a000)

The libldap-2.4.so.2 dynamic library was installed via openldap-devel when installing apisix. openldap-devel uses centos8's openssl, so the EVP_KDF* symbols are introduced.

apisix-base specifies the search path of the dynamic library through rpath at build time, via: https://github.com/api7/apisix-build-tools/blob/master/build-apisix-base.sh#L94. When apisix is ​​running, it will first search for this path when loading the dynamic library. This caused libldap-2.4.so.2 to be loaded with openresty-openssl instead of the system's openssl, thus resulting in the symbol not being found. For the search path of dynamic libraries, please refer to: https://man7.org/training/download/shlib_dynlinker_slides.pdf

Therefore, the solutions are as follows:

1. Delete the installed openresty-openssl. When apisix is ​​running, it will continue to search for the system openssl. This method is rude
2. Use source code to compile and install openldap, specify openssl as openresty-openssl when compiling. But unfortunately, openldap does not support specifying openssl directories. However I found a modified way here. This seems to work, and someone has tried it in the apisix issue.