Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Generate a json with extended synonym as we build taxonomies #10742

Open
alexgarel opened this issue Aug 28, 2024 · 1 comment · Fixed by #10744
Open

Generate a json with extended synonym as we build taxonomies #10742

alexgarel opened this issue Aug 28, 2024 · 1 comment · Fixed by #10744
Assignees
@stephanegigandet
Labels
API READ All READ APIs include Product, Search… API Refactor API Issues related to the Open Food Facts API. More specific labels exist & should be used (API WRITE…) Data export We export data nightly as CSV, MongoDB… See: https://world.openfoodfacts.org/data 🧬 Taxonomies https://wiki.openfoodfacts.org/Global_taxonomies

Comments

@alexgarel
Copy link
Member

Problem

As we build taxonomies,
we export a json with taxonomy. This is very useful for third party applications that deals with open food facts data, and wants to run some analysis based on taxonomies.
For example, this is used by robotoff and search-a-licious.

As reported on https://wiki.openfoodfacts.org/Taxonomy_access, we currently have two versions: one with only synonyms, and one with additional properties.
But there is no export with extended synonyms (were we replace synonyms by synonyms), whereas it would be very useful to search-a-licious.

Proposed solution

Export a <taxonomy_name>.extended.json, which contains a extended_synonyms property with extended synonyms.

My guess is to avoid putting properties in it (to avoid file being too massive), one can download .extended and .full version if needed, the merge is easy to do.

Code pointers

It happens in Tags.pm, in build_taxonomies build_tags_taxonomy (at the end)
@github-project-automation github-project-automation bot moved this to To discuss and validate in 🍊 Open Food Facts Server issues Aug 28, 2024
@stephanegigandet stephanegigandet self-assigned this Aug 28, 2024
@stephanegigandet stephanegigandet mentioned this issue Aug 28, 2024
Merged
@github-project-automation github-project-automation bot moved this from To discuss and validate to Done in 🍊 Open Food Facts Server issues Aug 28, 2024
@alexgarel
Copy link
Member Author

https://static.openfoodfacts.org/data/taxonomies/categories.extended.json is not there @stephanegigandet even if the other files were generated Oct 3 in /srv/off/html/data/taxonomies

@alexgarel alexgarel reopened this Oct 9, 2024
@teolemon teolemon added 🧬 Taxonomies https://wiki.openfoodfacts.org/Global_taxonomies Data export We export data nightly as CSV, MongoDB… See: https://world.openfoodfacts.org/data API Issues related to the Open Food Facts API. More specific labels exist & should be used (API WRITE…) API READ All READ APIs include Product, Search… API Refactor labels Oct 12, 2024
@McCio McCio mentioned this issue Oct 29, 2024
Merged
stephanegigandet pushed a commit that referenced this issue Oct 30, 2024
@McCio
taxonomy: synonyms, stop-words, extended json exports (#10951)
36516d8 
### Taxonomies improvements

* Include {tag}.extended.json files in exported taxonomies (#10742)
* Some more synonym and stop-words in Italian translations
* Align github PR script to fetch origin/main (for linting of
taxonomies)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stephanegigandet stephanegigandet

Labels
API READ All READ APIs include Product, Search… API Refactor API Issues related to the Open Food Facts API. More specific labels exist & should be used (API WRITE…) Data export We export data nightly as CSV, MongoDB… See: https://world.openfoodfacts.org/data 🧬 Taxonomies https://wiki.openfoodfacts.org/Global_taxonomies
Projects
🍊 Open Food Facts Server issues
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: new [taxonomy].extended.json with extended synonyms openfoodfacts/openfoodfacts-server
3 participants
@alexgarel @teolemon @stephanegigandet