Limit number of records sampled and filter tables

[jayadevanm]

Thank you for this great product. It will add a lot of value to our organisation where we are trying to set up a glossary, tag tables and columns etc. One doubt - is is possible to limit the number of records scanned for profiling, or turn it off for specific tables? In our tables with millions of records, the ingestion job just keeps running.

[pmbrull]

Thanks for reaching out and bigger thanks for your words.

We currently have two ways of computing the data profiling:

1. Enable it via the metadata ingestion
2. Run it on its own Profiler Workflow (available in the UI, Airflow SDK, and CLI)

What could be done here is to disable it during the metadata ingestion (we are thinking about removing it from there completely exactly for the same reasons you're giving) and just have separated Profiler Workflows handling that.

The interesting part about the Profiler Workflows is that we're not limited to a single workflow per service. You could deploy multiple workflows, filtering by different sets of tables, and run each of them at a different schedule.

Regarding limiting the number of records, we have implemented the first version of a data sampling in the profiler backend. It allows running the workflow on a specific % of a random sample for each table. In the next releases, we'll update the UI to allow select that % on each table.

[pmbrull]

If you'd need to play with setting data sampling percentages before that is available, we can connect and help you set that either via the API or running the profiler workflow through the CLI. Setting the % once will then make it available for all next runs.

[mesmith027]

Hey @jayadevanm,

Let us know how using Openmetadata in your company is going, or if we can help you with anything else!