Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't find the dataset lighteval/MATH-Hard in the huggingface #2618

Open
Chenxi622 opened this issue Jan 10, 2025 · 6 comments
Open

Can't find the dataset lighteval/MATH-Hard in the huggingface #2618

Chenxi622 opened this issue Jan 10, 2025 · 6 comments
Labels
bug Something isn't working.

Comments

@Chenxi622
Copy link

when I run the code to get the results of the tasks in the open llm leaderboard v2, I can't find lighteval/MATH-Hard in the huggingface, how should I solve this problem?
Thanks!

@Chenxi622 Chenxi622 changed the title About the dataset lighteval/MATH-Hard used in the leaderboard Can't find the dataset lighteval/MATH-Hard in the huggingface Jan 10, 2025
@zxia545
Copy link

zxia545 commented Jan 10, 2025

Have the same issue

@baberabb
Copy link
Contributor

Hi! yeah, it doesn't seem to be up anymore. cc: @clefourrier @NathanHB

@baberabb baberabb added the bug Something isn't working. label Jan 10, 2025
@stijn-zyphra
Copy link

Did one find a temporary solution for this with an exactly similar dataset? I reckon this dataset would probably be uploaded to HF multiple times, so forking lm_eval or monkey patching with the reference to that one would be feasible for now. How long does it usually take before these kind of problems are resolved? Thanks in advance.

@baberabb
Copy link
Contributor

baberabb commented Jan 10, 2025

As it's the same dataset as MATH, but they only use level-5 problems, you could change the dataset_path here to EleutherAI/hendrycks_math and pipe a filter in the process_docs function (return dataset.filter(lambda x: x["level"] == "Level 5").map(...)), in the utils script (same folder).

Also add "level": "Level 5", field to all the samples in list_fewshot_samples

@stijn-zyphra
Copy link

Thank you very much, will fork it and implement a custom solution!

@Chenxi622
Copy link
Author

Thank you for the answer!I will try it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working.
Projects
None yet
Development

No branches or pull requests

4 participants