Skip to content

Conversation

WangMinan
Copy link
Contributor

@WangMinan WangMinan commented Dec 30, 2023

修复了binlog_row_value_options=partial_json情况下的解析问题

#5017 当mysql的binlog_row_value_options=partial_json时binlog解析失败

原因

MySQL源码中sql\log_event.cc\print_json_diff与canal中com.taobao.tddl.dbsync.binlog.JsonDiffConversion#print_json_diff(com.taobao.tddl.dbsync.binlog.LogBuffer, long, java.lang.String, int, java.nio.charset.Charset)这两个函数对于解析的边界取值上存在偏差

MySQL源码中的length仅为JSON部分更新内容的长度

image-20231230112136655

但是canal使用buffer.hasRemaining进行判断,会导致解析到溢出的部分,也就是不属于JSON部分更新的内容的部分。超解析。

image-20231230112302620

在本来循环应该结束的位置没有停止,buffer过消费。实际上应该对buffer解析到len长度就停止解析。

修复

在循环结尾处引入长度消费检测,如果消费长度超出上限就直接退出循环

image-20231230112453034

image-20231230112522364

同时在入口处进行捕捉。

在某些情况下,虽然开启了binlog_row_value_options=partial_json,但是after-image中的json字段的binlog记录仍然是完整的(主要出现在全量替换等情况下),不使用json函数进行表达,因此需要返回原有的完整解析的逻辑。

image-20231230114148630

需要注意重新解析时需要做回退位点

正常解析结果

情况如下

image-20231230113512497

执行SQL

update test_json_table set c1 = json_set(c1, '$.id', 1) where id = 1;

解析情况

image-20231230113620821

执行SQL

update test_json_table set c1 = json_set(c1 , '$.name', 'go', '$.website', 'golang.com') where id = 1;

解析情况

image-20231230113724425

执行SQL

update test_json_table set c2 = json_array_insert(c2, '$[1]', cast('{"id": 1}' as json)) where id = 1;

执行情况

image-20231230114249290

上述即为开启partial_json情况下after-image完整的情况

执行SQL

update test_json_table set c2 = json_set(c2, '$[1].name', 'wangminan') where id = 1;

执行情况

image-20231230114446790

执行SQL

update test_json_table set c2 = json_set(c2, '$[1].id', 2, '$[100]', 'y') where id = 1;

执行结果

image-20231230114559640

嵌套解析正常,能够成功获取多层函数

执行SQL

update test_json_table set c1 = json_remove(c1, '$.website') where id = 1;

执行结果

image-20231230141451990

上述用例包含了JSON_INSERT,JSON_REPLACE,JSON_ARRAY_INSERT,JSON_REMOVE以及partial_json下完整after-image,即可能出现的所有情况。

@CLAassistant
Copy link

CLAassistant commented Dec 30, 2023

CLA assistant check
All committers have signed the CLA.

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@agapple agapple merged commit 54145ce into alibaba:master Jan 2, 2024
@agapple
Copy link
Member

agapple commented Jan 2, 2024

tks

zoemak pushed a commit to zoemak/canal that referenced this pull request Jan 30, 2024
…#5018)

* 修改json部分更新的问题

* 保证after-image的情况下位点回退完整
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants