Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support: stream, v12, -fvisibility, CI 5.0 6.0 6.2, make debug. Fix json enc #22

Merged
merged 21 commits into from
Oct 18, 2023

Conversation

moticless
Copy link
Collaborator

@moticless moticless commented Oct 8, 2023

  • Support stream
  • Extend readerResp to be able reading bulk array (for Stream).
  • RDB2JSON: Fix encoding from \x## to standard \u00##
  • Make "-fvisibility=hidden" and static link rdb-cli with librdb
  • Run CI against redis 5.0,6.0,6.2 and unstable
  • Fix Module RESTORE for earlier redis versions
  • Added makefile rule "debug" to build with -O0
  • Make iovec macros more clear.
  • Support latest rdbversion 12.

Stream support

git:(add-stream-support) rdb-cli ../redis/dump.rdb json               
[{
    "my_key":"Hello, Redis!",
    "my_set":["member1","member2"],
    "my_zset":{"Bob":"5","Alice":"10","Charlie":"15"},
    "my_hash":{"field1":"value1","field2":"value2"},
    "my_list":["item1","item2","item3"],
    "mystream":{
      "entries":[ 
        { "id":"1695649068107-0", "items":{"message":"Message1"} },
        { "id":"1695649068110-0", "items":{"message":"Message2"} },
        { "id":"1695649069139-0", "items":{"message":"Message3"} },
        { "id":"1695649446276-0", "items":{"message":"Message4"} },
        { "id":"1695649456516-0", "items":{"message":"Message5"} },
        { "id":"1695893015933-0", "items":{"field1":"value1", "field2":"value2", "field3":"value3"} }
   ]}
}]

git:(add-stream-support) rdb-cli ../redis/dump.rdb json -i stream-meta                                                                                                                   
[{                                                                                                                                                                                                         
    "my_key":"Hello, Redis!",                                                                                                                                                                              
    "my_set":["member1","member2"],                                                                                                                                                                        
    "my_zset":{"Bob":"5","Alice":"10","Charlie":"15"},
    "my_hash":{"field1":"value1","field2":"value2"},
    "my_list":["item1","item2","item3"],
    "mystream":{
      "entries":[ 
        { "id":"1695649068107-0", "items":{"message":"Message1"} },
        { "id":"1695649068110-0", "items":{"message":"Message2"} },
        { "id":"1695649069139-0", "items":{"message":"Message3"} },
        { "id":"1695649446276-0", "items":{"message":"Message4"} },
        { "id":"1695649456516-0", "items":{"message":"Message5"} },
        { "id":"1695893015933-0", "items":{"field1":"value1", "field2":"value2", "field3":"value3"} }],
      "length": 6, 
      "entriesAdded": 6, 
      "firstID": "1695649068107-0", 
      "lastID": "1695893015933-0", 
      "maxDelEntryID": "0-0",
      "groups": [
        {"name": "groupA", "lastid": "1695649446276-0", "entriesRead": 4,
         "pending": [ 
           { "sent": 1695649446276, "id":"1695649446276-0", "count": 1 }],
         "consumers": [
           { "name": "consumerA2", "activeTime": 1696708503339, "seenTime": 1696708503339,
             "pending": [
               {"id":"1695649446276-0"}]}]},
        {"name": "groupB", "lastid": "1695649069139-0", "entriesRead": 3,
         "pending": [ 
           { "sent": 1695649069139, "id":"1695649069139-0", "count": 1 }],
         "consumers": [
           { "name": "consumerB1", "activeTime": 1696708503339, "seenTime": 1696708503339,
             "pending": [
               {"id":"1695649069139-0"}]}]}]}
}]

git:(add-stream-support) rdb-cli ../redis/dump.rdb resp 
*2   
$6    
SELECT  
$1     
0              
*3     
$3     
SET   
$6      
my_key
$13   
Hello, Redis!
*3      
$4      
SADD  
$6             
my_set         
$7     
member1        
*3      
$4
SADD    
$6  
my_set
$7      
member2   
....

Make "-fvisibility=hidden

Set the default visibility of symbols to be hidden.
This raises an issue with deps/redis files because it is currently being
build internally and staticaly linked with librdb.so whereas librdb-ext.so need it
as well for few APIs, such as the use of the rax data structure or CRC.

Instead of refactoring and creating an additional shared library, or making
copies for each library (with all the associated implications) we shall expose
specific symbols from librdb.so that are needed by librdb-ext.so. These symbols
won't be documented in the API, but they will be available for use. Since we
compile by default with hidden visibility, we need to mark them explicitly as
visible using the following macro inside deps/redis:

#define _LIBRDB_HIDDEN_API __attribute__((visibility("default")))

While it might initially seem cumbersome to manually designate each function as
visible, this approach has its benefits. By carefully selecting the functions
to expose, we are encouraged to contemplate what we are exposing and why.
Moreover, it helps prevent the cluttering clients of librdb with unnecessary
symbols.

List of all symbols (All Undefined symbols were filtered-out. Functions of standard libraries):

➜  push-to-librdb git:(add-stream-support) nm -D ./lib/librdb.so | grep ' [TDW] ' 
000000000000a1e0 T RDB_alloc
000000000000a310 T RDB_bulkClone
00000000000033b0 T RDB_bulkCopyClone
0000000000003300 T RDB_bulkCopyFree
000000000000a210 T RDB_bulkLen
000000000000a9b0 T RDB_createHandlersData
000000000000a7c0 T RDB_createHandlersRaw
000000000000a8c0 T RDB_createHandlersStruct
00000000000091a0 T RDB_createParserRdb
000000000000a130 T RDB_createReaderRdb
0000000000009610 T RDB_deleteParser
000000000000a790 T RDB_free
000000000000a630 T RDB_getBytesProcessed
000000000000a6f0 T RDB_getErrorCode
000000000000a720 T RDB_getErrorMessage
000000000000ab70 T RDB_getLibVersion
000000000000abc0 T RDB_getMaxSuppportRdbVersion
000000000000a6c0 T RDB_getNumHandlers
000000000000a590 T RDB_getNumItemsHint
000000000000a660 T RDB_getRdbVersion
000000000000a690 T RDB_getState
000000000000aaa0 T RDB_handleByLevel
000000000000a430 T RDB_IgnoreChecksum
000000000000a290 T RDB_isRefBulk
000000000000a490 T RDB_log
00000000000096e0 T RDB_parse
0000000000009ee0 T RDB_parseBuff
000000000000a3c0 T RDB_pauseParser
000000000000a750 T RDB_realloc
0000000000009fd0 T RDB_reportError
000000000000a5f0 T RDB_setDeepIntegCheck
000000000000a400 T RDB_setLogger
000000000000a5c0 T RDB_setLogLevel
000000000000a460 T RDB_setMaxRawSize
000000000000a390 T RDB_setPauseInterval
➜  push-to-librdb git:(add-stream-support) nm -D ./lib/librdb-ext.so | grep ' [TDW] '
0000000000005970 T RDBX_attachRespWriter
00000000000069b0 T RDBX_createHandlersFilterDbNum
0000000000006670 T RDBX_createHandlersFilterKey
0000000000006960 T RDBX_createHandlersFilterType
00000000000071a0 T RDBX_createHandlersToJson
0000000000003880 T RDBX_createHandlersToResp
00000000000096d0 T RDBX_createReaderFile
00000000000095a0 T RDBX_createReaderFileDesc
0000000000003530 T RDBX_createRespToFileWriter
0000000000005c60 T RDBX_createRespToRedisFd
0000000000006380 T RDBX_createRespToRedisTcp
00000000000059d0 T RDBX_enumerateCmds
0000000000005c20 T RDBX_setPipelineDepth
0000000000005a00 T RDBX_writeFromCmdNumber

api/librdb-ext-api.h Outdated Show resolved Hide resolved
Makefile Show resolved Hide resolved
deps/redis/rax_malloc.h Outdated Show resolved Hide resolved
src/cli/rdb-cli.c Outdated Show resolved Hide resolved
src/ext/handlersToJson.c Outdated Show resolved Hide resolved
src/ext/handlersToJson.c Outdated Show resolved Hide resolved
src/ext/handlersToJson.c Outdated Show resolved Hide resolved
src/ext/handlersToJson.c Outdated Show resolved Hide resolved
src/ext/handlersToResp.c Outdated Show resolved Hide resolved
src/ext/respToRedisLoader.c Outdated Show resolved Hide resolved
src/ext/respToRedisLoader.c Outdated Show resolved Hide resolved
src/lib/parser.c Show resolved Hide resolved
src/lib/parser.c Outdated Show resolved Hide resolved
src/lib/parser.c Show resolved Hide resolved
src/lib/parser.h Outdated Show resolved Hide resolved
src/lib/parserRaw.c Outdated Show resolved Hide resolved
src/lib/parserRaw.c Outdated Show resolved Hide resolved
src/cli/rdb-cli.c Outdated Show resolved Hide resolved
Copy link
Member

@oranagra oranagra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

skimmed through it (skipped the tests as usual, other than a quick look at json and resp output).
took a few minor comments.

api/librdb-api.h Outdated Show resolved Hide resolved
src/cli/rdb-cli.c Outdated Show resolved Hide resolved
api/librdb-api.h Show resolved Hide resolved
src/ext/handlersToResp.c Outdated Show resolved Hide resolved
src/ext/handlersToResp.c Outdated Show resolved Hide resolved
src/ext/readerResp.c Show resolved Hide resolved
src/lib/parser.h Outdated Show resolved Hide resolved
src/ext/handlersToResp.c Outdated Show resolved Hide resolved
src/lib/Makefile Show resolved Hide resolved
@moticless moticless changed the title Stream support Stream support; hide symbols; Test vs. Redis 5.0,6.0,6.2; make debug Oct 15, 2023
src/lib/parser.c Outdated Show resolved Hide resolved
deps/redis/librdb-hidden-api.h Outdated Show resolved Hide resolved
@moticless moticless changed the title Stream support; hide symbols; Test vs. Redis 5.0,6.0,6.2; make debug Stream support; hide symbols; CI 5.0,6.0,6.2,unstable; make debug; Fix json enc Oct 15, 2023
@moticless moticless changed the title Stream support; hide symbols; CI 5.0,6.0,6.2,unstable; make debug; Fix json enc Stream support; hide symbols; CI 5.0,6.0,6.2,latest; make debug; Fix json enc Oct 15, 2023
@moticless moticless changed the title Stream support; hide symbols; CI 5.0,6.0,6.2,latest; make debug; Fix json enc Support: stream, v12, -fvisibility, CI 5.0 6.0 6.2, make debug. Fix json enc Oct 16, 2023
src/ext/handlersToResp.c Outdated Show resolved Hide resolved
src/ext/handlersToResp.c Outdated Show resolved Hide resolved
@moticless moticless merged commit c212c7f into redis:main Oct 18, 2023
2 checks passed
@moticless moticless deleted the add-stream-support branch October 18, 2023 06:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants