Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
add example for big lake metastore
  • Loading branch information
sagarlakshmipathy authored Nov 8, 2023
1 parent 2f22b7f commit 141be61
Showing 1 changed file with 25 additions and 1 deletion.
26 changes: 25 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,30 @@ catalogOptions: # all other options are passed through in a map
key1: value1
key2: value2
```
Here's an example `my_config.yaml` and `catalog.yaml` files to sync with BigLake Metastore:

1. my_config.yaml
```yaml
sourceFormat: HUDI
targetFormats:
- ICEBERG
datasets:
-
tableBasePath: gs://path/to/source/data
tableName: table_name
namespace: database_name
```

2. catalog.yaml
```yaml
catalogImpl: org.apache.iceberg.gcp.biglake.BigLakeCatalog
catalogName: catalog_name
catalogOptions:
gcp_project: gcp_project_name
gcp_location: location # example us-west1
warehouse: gs://path/to/warehouse
```

5. run with `java -jar utilities/target/utilities-0.1.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml [--hadoopConfig hdfs-site.xml] [--clientsConfig clients.yaml] [--icebergCatalogConfig catalog.yaml]`
The bundled jar includes hadoop dependencies for AWS, Azure, and GCP. Authentication for AWS is done with
`com.amazonaws.auth.DefaultAWSCredentialsProviderChain`. To override this setting, specify a different implementation
Expand All @@ -102,4 +126,4 @@ You have found a bug, or have a cool idea you that want to contribute to the pro
Adding a new target format requires a developer implement [TargetClient](./api/src/main/java/io/onetable/spi/sync/TargetClient.java). Once you have implemented that interface, you can integrate it into the [OneTableClient](./core/src/main/java/io/onetable/client/OneTableClient.java). If you think others may find that target useful, please raise a Pull Request to add it to the project.

## Overview of the sync process
![img.png](docs/images/sync_flow.jpg)
![img.png](docs/images/sync_flow.jpg)

0 comments on commit 141be61

Please sign in to comment.