Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] [Postgre CDC] Unsupported type: ARRAY<INT> #8466

Open
3 tasks done
Quilian opened this issue Jan 6, 2025 · 3 comments
Open
3 tasks done

[Feature] [Postgre CDC] Unsupported type: ARRAY<INT> #8466

Quilian opened this issue Jan 6, 2025 · 3 comments
Labels

Comments

@Quilian
Copy link

Quilian commented Jan 6, 2025

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

使用Zeta引擎,使用Postgre CDC创建连接报错不支持类型,看官方稳定是支持对应的类型的。

SeaTunnel Version

2.3.4

SeaTunnel Config

env {
  execution.parallelism = 1
  job.mode = "STREAMING"
  checkpoint.interval = 300
  read_limit.bytes_per_second=50000000
  read_limit.rows_per_second=10000
}
source {
 Postgres-CDC {
    #result_table_name = "app_task_info"
    username = "sjxx"
    password = "Sjxxxx"
    database-names = ["ccm_xx"]
    schema-names = ["public"]
    table-names = ["ccm_xx.public.app_task_info"]
    base-url = "jdbc:postgresql://xx.xx.xxx.xx:5436/ccm_xx?loggerLevel=OFF"
  }
}

sink {
    Console {
    }
}

Running Command

../bin/seatunnel.sh -c pg_source2.template

Error Exception

Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:199)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.api.table.factory.FactoryException: ErrorCode:[API-06], ErrorDescription:[Factory initialize failed] - Unable to create a source for identifier 'Postgres-CDC'.
        at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:100)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:320)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:181)
        at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:88)
        at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:161)
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:146)
        ... 2 more
Caused by: java.lang.UnsupportedOperationException: Unsupported type: ARRAY<INT>
        at org.apache.seatunnel.connectors.cdc.debezium.row.SeaTunnelRowDebeziumDeserializationConverters.createNotNullConverter(SeaTunnelRowDebeziumDeserializationConverters.java:178)
        at org.apache.seatunnel.connectors.cdc.debezium.row.SeaTunnelRowDebeziumDeserializationConverters.createConverter(SeaTunnelRowDebeziumDeserializationConverters.java:113)
        at org.apache.seatunnel.connectors.cdc.debezium.row.SeaTunnelRowDebeziumDeserializationConverters.lambda$new$0(SeaTunnelRowDebeziumDeserializationConverters.java:71)
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:546)
        at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
        at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:505)
        at org.apache.seatunnel.connectors.cdc.debezium.row.SeaTunnelRowDebeziumDeserializationConverters.<init>(SeaTunnelRowDebeziumDeserializationConverters.java:73)
        at org.apache.seatunnel.connectors.cdc.debezium.row.SeaTunnelRowDebeziumDeserializeSchema.createTableRowConverters(SeaTunnelRowDebeziumDeserializeSchema.java:295)
        at org.apache.seatunnel.connectors.cdc.debezium.row.SeaTunnelRowDebeziumDeserializeSchema.<init>(SeaTunnelRowDebeziumDeserializeSchema.java:87)
        at org.apache.seatunnel.connectors.cdc.debezium.row.SeaTunnelRowDebeziumDeserializeSchema$Builder.build(SeaTunnelRowDebeziumDeserializeSchema.java:331)
        at org.apache.seatunnel.connectors.seatunnel.cdc.postgres.source.PostgresIncrementalSource.createDebeziumDeserializationSchema(PostgresIncrementalSource.java:112)
        at org.apache.seatunnel.connectors.cdc.base.source.IncrementalSource.<init>(IncrementalSource.java:114)
        at org.apache.seatunnel.connectors.seatunnel.cdc.postgres.source.PostgresIncrementalSource.<init>(PostgresIncrementalSource.java:69)
        at org.apache.seatunnel.connectors.seatunnel.cdc.postgres.source.PostgresIncrementalSourceFactory.lambda$createSource$1(PostgresIncrementalSourceFactory.java:103)
        at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:112)
        at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:73)
        ... 7 more

Zeta or Flink or Spark Version

Zeta

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@Quilian Quilian added the bug label Jan 6, 2025
@hailin0 hailin0 added help wanted feature New feature and removed bug labels Jan 7, 2025
@davidzollo davidzollo changed the title [Bug] [Postgre CDC] Unsupported type: ARRAY<INT> [Feature] [Postgre CDC] Unsupported type: ARRAY<INT> Jan 7, 2025
@Quilian
Copy link
Author

Quilian commented Jan 8, 2025

补充,应该不支持全部ARRAY<>,因为多尝试了几个表,报错不支持ARRAY

@Quilian
Copy link
Author

Quilian commented Jan 8, 2025

也会报错不支持ARRAY

@Quilian
Copy link
Author

Quilian commented Jan 8, 2025

string

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants