Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RANGER-4932:Handling Specified key was too long; max key length error #391

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

basapuram-kumar
Copy link
Contributor

What changes were proposed in this pull request?

Proposed changes are mentioned at RANGER-4932

How was this patch tested?

tested this patch on my local cluster during the migration of ranger from 2.3.0 to 2.5.0, and with proposed changes, able to fix this issue.

@basapuram-kumar basapuram-kumar changed the title RANGER-4931:Handling Specified key was too long; max key length error RANGER-4932:Handling Specified key was too long; max key length error Sep 12, 2024
@prabhjyotsingh
Copy link

LGTM!

@spolavarpau1 / @princeap173 can you please help review.

@bhavikpatel9977
Copy link
Contributor

Changing the DEFAULT CHARSET for just one table might cause issue if the DEFAULT CHARSET is set to something different.

@basapuram-kumar
Copy link
Contributor Author

basapuram-kumar commented Sep 13, 2024

Hello @bhavikpatel9977 ,
I have following approaches to discuss here.

  1. Define the DEFAULT CHARSET=utf8 has been successful.
  2. Reduce the size of indexed columns either with
    KEYx_trx_log_v2_trx_id (trx_id(512))) or
    KEY x_trx_log_v2_trx_id (trx_id(255)) also seems to work..

Could you please advise whether one of these approaches would be optimal, or if there are any better alternatives we should consider?
Thank you.

@basapuram-kumar
Copy link
Contributor Author

Hi team, I'm just checking in on the status of my pull request. Would you be able to review it soon?
Would appreciate any feedback or suggestions on this PR.
Thanks!
cc: @bhavikpatel9977 , @spolavarpau1 , @princeap173 , @kumaab , @prabhjyotsingh

@basapuram-kumar
Copy link
Contributor Author

Hi team,
I'm just checking in on the status of my pull request. Would you be able to review it soon?
Would appreciate any feedback or suggestions on this PR. Thanks

@vikaskr22
Copy link
Contributor

Hi @basapuram-kumar ,
Thanks for the work !
Today I tried to understand this, and sharing my input on this.
I see merit in Bhavik's comment, changing the CHARSET at table level may create many problems. Charset changes in just one table may break JOIN operation or may result wrong output.
Sometimes, we set the Charset at Connection level in code , like we set to read using UTF-8, there it would break.

My input is, it should be defined at schema/DB level ( may be using DB server configs), means same charset for entire Ranger.

Now coming to Latin1, it will limit the characters that Ranger supports or will support in future.

On the second approach, where you suggested to reduce the length of trx_id(255), it may work. But if intention is to keep exact transactionId in the index, then it would be an issue.

Few questions:
What is the MySql version where you got this error ? What is default "storage_engine"? It should be 'InnoDb'.

I am also exploring this and will add my input here.

@basapuram-kumar
Copy link
Contributor Author

Hello @vikaskr22 ,
Thanks for the details explaination on the pros and cons of the suggested settings.

yes, absolutely agreed on the pointers.

For the asked info,

What is the MySql version where you got this error ? What is default "storage_engine"? It should be 'InnoDb'.

I am using MySQL-8.0.36, and the detault storage_engine is InnoDB

mysql > show create table x_trx_log_v2;

x_trx_log_v2 | CREATE TABLE x_trx_log_v2 (
id bigint NOT NULL AUTO_INCREMENT,
create_time datetime DEFAULT NULL,
added_by_id bigint DEFAULT NULL,
class_type int NOT NULL DEFAULT '0',
object_id bigint DEFAULT NULL,
parent_object_id bigint DEFAULT NULL,
parent_object_class_type int NOT NULL DEFAULT '0',
parent_object_name varchar(1024) DEFAULT NULL,
object_name varchar(1024) DEFAULT NULL,
change_info mediumtext,
trx_id varchar(1024) DEFAULT NULL,
action varchar(255) DEFAULT NULL,
sess_id varchar(512) DEFAULT NULL,
req_id varchar(30) DEFAULT NULL,
sess_type varchar(30) DEFAULT NULL,
PRIMARY KEY (id),
KEY x_trx_log_v2_FK_added_by_id (added_by_id),
KEY x_trx_log_v2_cr_time (create_time),
KEY x_trx_log_v2_trx_id (trx_id)
) ENGINE=InnoDB AUTO_INCREMENT=19 DEFAULT CHARSET=latin1 ROW_FORMAT=DYNAMIC |

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants