I get slightly Lower Rouge score for the same code

[aadithyamd]

I got Rouge following Rouge Scores:

For classifier
When I ran : ROUGE-F(1/2/3/l): 42.99/20.15/39.41
On Paper : ROUGE-F(1/2/3/l): 43.23/20.22/39.60

For Transformer

When I ran : ROUGE-F(1/2/3/l): 43.02/20.18/39.45
On Paper : ROUGE-F(1/2/3/l): 43.25/20.24/39.63

The values written here are the best scores I got from the model following the instructions and paper shows average of 3, which would be much lower in my case.

Model is evaluated as per the instruction in the Readme,
python train.py -mode validate -bert_data_path ../bert_data/cnndm -model_path MODEL_PATH -visible_gpus 0 -gpu_ranks 0 -batch_size 30000 -log_file LOG_FILE -result_path RESULT_PATH -test_all -block_trigram true

Is interval segments enabled by default? or should we add some parameter to enable it?

I have used one Tesla V100 GPU & library versions are probably updated.

Thanks

[lmq990417]

@aadithyamd Hello, I see you have successfully run this project, so I have two questions to ask you. The first problem is Format to Simpler json files, which is the fourth step of data processing. I do not export this part according to the tutorial and I do not know what is wrong with it. Then I found the processed data set and trained it. When I tested it, there was a problem with Pyrouge : no. Pyrouge /settings.ini. But with the problem :
[the 2021-01-26 13:11:00, 751 INFO] Running ROUGE with command/home/BertSum/pyrouge/ROUGE/tools/ROUGE - 1.5.5 / ROUGE - 1.5.5. Pl - e/home/BertSum pyrouge/ROUGE/tools/ROUGE - 1.5.5 / data - c. 95-1000 - m - r n 2 -a .. /temp/tmpqutlh7il/rouge_conf.xml

Traceback (most recent call last):

File "train.py", line 353, in

test(args, device_id, cp, step)

File "train.py", line 216, in test

trainer.test(test_iter,step)

File "/home/BertSum/src/models/trainer.py", line 297, in test

rouges = test_rouge(self.args.temp_dir, can_path, gold_path)

File "/home/BertSum/src/others/utils.py", line 84, in test_rouge

rouge_results = r.convert_and_evaluate()

File "/home/BertSum/src/others/pyrouge.py", line 398, in convert_and_evaluate

rouge_output = self.evaluate(system_id, rouge_args)

File "/home/BertSum/src/others/pyrouge.py", line 373, in evaluate

rouge_output = check_output(command).decode("UTF-8")

The File "/ usr/lib/python3.5 subprocess. Py", line 626, in check_output

**kwargs).stdout

The File "/ usr/lib/python3.5 subprocess. Py", line 693, in the run

with Popen(*popenargs, **kwargs) as process:

The File "/ usr/lib/python3.5 subprocess. Py", line 947, in init

restore_signals, start_new_session)

The File "/ usr/lib/python3.5 subprocess. Py", line 1551, in _execute_child

raise child_exception_type(errno_num, err_msg)

PermissionError: [Errno 13] Permission denied
Now I urgently need to realize this project, and I hope to get your answer and help! Thank you very much indeed!

[sarath-nookaraaju]

hi @lmq990417

it sounds like an issue with rouge installation. did you get OK from tests of Rouge?

[lmq990417]

@aadithyamd I followed the steps on Stack Overflow:

Step 1 : Install Pyrouge from source (not from pip)

git clone https://github.com/bheinzerling/pyrouge

cd pyrouge

pip install -e .

Step 2 : Install official ROUGE script

git clone https://github.com/andersjo/pyrouge.git rouge

Step 3 : Point Pyrouge to official rouge script

Pyrouge_set_rouge_path ~ / pyrouge rouge/tools/rouge - 1.5.5 /

The path given to pyrouge should be absolute path !

Step 4 : Install libxml parser

As mentioned in this issue, you need to install libxml parser :

sudo apt-get install libxml-parser-perl

Step 5 : Regenerate the Exceptions DB

As mentioned in this issue, you need to regenerate the Exceptions DB :

CD rouge/tools/rouge - 1.5.5 / data

The rm WordNet 2.0. Exc. Db

. / WordNet 2.0 - Exceptions/buildExeptionDB. Pl. / WordNet - 2.0 - Exceptions. / smart_common_words. TXT. / WordNet - 2.0. Exc. Db

Step 6 : Run the tests

python -m pyrouge.test

You should see :

RAN 11 tests in 6.322s

OK.

But since I didn't make the mistakes mentioned in steps 4 and 5:

The 2021-01-26 13:58:30, 527 [MainThread] [INFO] Set ROUGE home directory to/home/chengrong/BertSum/pyrouge/ROUGE/tools/ROUGE - 1.5.5.

I didn't follow these two steps, and there were three errors:

ERROR: test_evaluation (pyrouge.tests.Rouge155_test.PyrougeTest)

---

Traceback (most recent call last):

File "/home/BertSum/pyrouge/pyrouge/tests/Rouge155_test.py", line 156, in test_evaluation

pyrouge_output = rouge.evaluate(system_id=11).strip()

File "/home/BertSum/pyrouge/pyrouge/Rouge155.py", line 342, in evaluate

rouge_output = check_output(command, env=env).decode("UTF-8")

The File "/ usr/lib/python3.5 subprocess. Py", line 626, in check_output

**kwargs).stdout

The File "/ usr/lib/python3.5 subprocess. Py", line 693, in the run

with Popen(*popenargs, **kwargs) as process:

The File "/ usr/lib/python3.5 subprocess. Py", line 947, in init

restore_signals, start_new_session)

The File "/ usr/lib/python3.5 subprocess. Py", line 1551, in _execute_child

raise child_exception_type(errno_num, err_msg)

PermissionError: [Errno 13] Permission denied

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =

ERROR: test_options (pyrouge.tests.Rouge155_test.PyrougeTest)

---

Traceback (most recent call last):

File "/home/BertSum/pyrouge/pyrouge/tests/Rouge155_test.py", line 218, in test_options

pyrouge_output = check_output_clean(pyrouge_command)

File "/home/BertSum/pyrouge/pyrouge/tests/Rouge155_test.py", line 17, in

check_output_clean = lambda c: check_output(c).decode("UTF-8").strip()

The File "/ usr/lib/python3.5 subprocess. Py", line 626, in check_output

**kwargs).stdout

The File "/ usr/lib/python3.5 subprocess. Py", line 708, in the run

output=stdout, stderr=stderr)

subprocess.CalledProcessError: Command '['pyrouge_evaluate_plain_text_files', '-m', 'data/models_plain', '-s', 'data/systems_plain', '-sfp', 'D(\d+).M.100.T.A', '-mfp', 'D#ID#.M.100.T.[A-Z]', '-id', '1', '-rargs', '" - e/home/BertSum pyrouge/rouge/tools/rouge - 1.5.5 / data - c 90-2-1-1000 - U - r n 1.2 - a - 2 - w m data/config_test2 XML "'] 'returned non - zero exit status 1

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =

ERROR: test_rouge_for_plain_text (pyrouge.tests.Rouge155_test.PyrougeTest)

---

Traceback (most recent call last):

File "/home/BertSum/pyrouge/pyrouge/tests/Rouge155_test.py", line 173, in test_rouge_for_plain_text

pyrouge_output = check_output_clean(pyrouge_command.split())

File "/home/BertSum/pyrouge/pyrouge/tests/Rouge155_test.py", line 17, in

check_output_clean = lambda c: check_output(c).decode("UTF-8").strip()

The File "/ usr/lib/python3.5 subprocess. Py", line 626, in check_output

**kwargs).stdout

The File "/ usr/lib/python3.5 subprocess. Py", line 708, in the run

output=stdout, stderr=stderr)

subprocess.CalledProcessError: Command '['pyrouge_evaluate_plain_text_files', '-m', 'data/models_plain', '-s', 'data/systems_plain', '-sfp', 'D(\d+).M.100.T.A', '-mfp', 'D#ID#.M.100.T.[A-Z]', '-id', '1']' returned non-zero exit status 1

---

RAN 10 tests in 1.266s

FAILED (errors=3)

I also do not know why, excuse me you have what method, or tell me you are how to install!

[sarath-nookaraaju]

While a simpler method worked for me on my Mac, it's the above method that worked on Colab and Nimblebox platforms when running MatchSum; code I used given below - same as above, all together

!git clone https://github.com/bheinzerling/pyrouge
import os
os.chdir('/content/pyrouge')
print(os.getcwd())
!pip install -e .
!git clone https://github.com/andersjo/pyrouge.git rouge
!pyrouge_set_rouge_path '/content/pyrouge/rouge/tools/ROUGE-1.5.5'
!sudo apt-get install libxml-parser-perl
os.chdir('/content/pyrouge/rouge/tools/ROUGE-1.5.5/data')
!rm WordNet-2.0.exc.db
!sudo ./WordNet-2.0-Exceptions/buildExeptionDB.pl ./WordNet-2.0-Exceptions ./smart_common_words.txt ./WordNet-2.0.exc.db
!python -m pyrouge.test

[lmq990417]

@aadithyamd
It seems that we have to implement step four and step five, I will try again, and then trouble you if there is any problem! Thank you very much!