Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Store historical datasets on Amazon Datasets #9355

Open
teolemon opened this issue Nov 21, 2023 · 2 comments
Open

Store historical datasets on Amazon Datasets #9355

teolemon opened this issue Nov 21, 2023 · 2 comments
Assignees
@raphael0202
Labels
Data export We export data nightly as CSV, MongoDB… See: https://world.openfoodfacts.org/data 🎯 P1 product reformulation

Comments

@teolemon
Copy link
Member

What
@teolemon teolemon added 🎯 P1 Data export We export data nightly as CSV, MongoDB… See: https://world.openfoodfacts.org/data labels Nov 21, 2023
@CharlesNepote CharlesNepote mentioned this issue Dec 18, 2023
Closed
@teolemon teolemon moved this to To discuss and validate in 🍊 Open Food Facts Server issues Apr 23, 2024
@teolemon
Copy link
Member Author

We might be doing that on Hugging Faces instead, right @raphael0202 ?

@raphael0202
Copy link
Contributor

We might be doing that on Hugging Faces instead, right @raphael0202 ?

Hugging Face is not the best place to store historical data (as long as the parquet deduped algorithm by Hugging Face is not launched).
We could store old versions on AWS Open Datasets indeed (in a separate folder).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@raphael0202 raphael0202

Labels
Data export We export data nightly as CSV, MongoDB… See: https://world.openfoodfacts.org/data 🎯 P1 product reformulation
Projects
🍊 Open Food Facts Server issues
Status: To discuss and validate
🎯 Very high impact issues
Status: To do
Milestone
No milestone
Development

No branches or pull requests
2 participants
@teolemon @raphael0202