mysql做为数据源批采集无法正常checkpoint #5924
Unanswered
suifeng08371
asked this question in
Q&A
Replies: 1 comment
-
Hello, is this what happens when you try to remove the speed limit? mysql data volume How much DDL? Can you provide it? |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
通过seatunnel进行mysql到hive的数据同步,mysql作为数据源,通过jdbc的方式进行连接。指定了条数的限流"read_limit.rows_per_second" : 100000,导致checkpoint没法正常成功。导到checkpoint超时时间后,checkpoint直接报错。
批cai采集不支持checkpoint,还是我配置的问题?请帮忙解答。
对应的报错截图信息如下:
checpoint出错:
Exceptions信息:
对应的seatunnel配置如下:
{
"env" : {
"job.name" : "sigle_test_diaodu",
"job.mode" : "BATCH",
"execution.checkpoint.cleanup-mode" : false,
"execution.checkpoint.interval" : 10000,
"execution.checkpoint.data-uri" : "hdfs://ns1/seatunnel/checkpoint/xxxx",
"parallelism" : 1,
"read_limit.rows_per_second" : 100000
},
"source" : [ {
"url" : "jdbc:mysql://10.10.10.10:3306/dbName",
"driver" : "com.mysql.jdbc.Driver",
"user" : "username",
"password" : "password",
"query" : "SELECT DETAIL_ADRESS,MSISDN,USER_ID,USER_NAME,STATUS,ID FROM dbName.table_name",
"plugin_name" : "Jdbc",
"partition_column" : "ID",
"result_table_name" : "_seatunnel_table_5678"
} ],
"transform" : [ {
"query" : "SELECT DETAIL_ADRESS AS TEST1,MSISDN AS TEST4,USER_ID AS TEST3,USER_NAME AS TEST5,STATUS AS TEST6,'1' AS TEST2 FROM _seatunnel_table_5678",
"field_name" : "TEST1,TEST4,TEST3,TEST5,TEST6,TEST2",
"plugin_name" : "Sql",
"source_table_name" : "_seatunnel_table_5678",
"result_table_name" : "_seatunnel_table_5680"
} ],
"sink" : [ {
"table_name" : "prod.hive_table_name",
"hive_site_path" : "/hive/conf/hive-site.xml",
"metastore_uri" : "thrift://11.1.1.1:9083,thrift://11.1.1.2:9083",
"plugin_name" : "Hive",
"source_table_name" : "_seatunnel_table_5680"
} ]
}
Beta Was this translation helpful? Give feedback.
All reactions