Wrong call to phpunit when asking for test groups #8082

tenzap · 2024-12-22T10:17:05Z

Apache NetBeans version

Apache NetBeans 24

What happened

The call to phpunit when asking for test groups isn't correct so that it reports that there is no group.

Updating the call would fix the issue.

Language / Project Type / NetBeans Component

PHP, testing with phpunit

How to reproduce

I setup a PHP project to run tests with PHPUnit
In the phpunit config (project properties / Testing / PHPUnit) I check "Ask for test groups before running tests".
I also have "test project using phpunit command only" set. As well as the XML file, and path to phpunit.

netbeans will invoke phpunit with

"/usr/bin/php" ".../vendor/bin/phpunit" "--colors" "--configuration" ".../application/tests/phpunit.xml" "--list-groups" "."

Output it

PHPUnit 10.5.40 by Sebastian Bergmann and contributors.

Available test group(s):
Done.

The part "--list-groups" "." should actually be "--list-groups" (ie. without the "." part).

When I run the command myself without the "." part), phpunit returns the list of groups, otherwise it doesn't.

"/usr/bin/php" ".../vendor/bin/phpunit" "--colors" "--configuration" ".../application/tests/phpunit.xml" "--list-groups"

PHPUnit 10.5.40 by Sebastian Bergmann and contributors.

Available test group(s):
 - default
 - html5

Did this work correctly in an earlier version?

No / Don't know

Operating System

Debian 12

JDK

Java: 17.0.13; OpenJDK 64-Bit Server VM 17.0.13+11-Debian-2deb12u1

Apache NetBeans packaging

Apache NetBeans provided installer

Anything else

No response

Are you willing to submit a pull request?

No

The text was updated successfully, but these errors were encountered:

junichi11 · 2024-12-23T00:24:00Z

@tenzap Can you provide an example project as a zip file to reproduce it?

tenzap · 2024-12-23T07:26:37Z

See: https://github.com/tenzap/phpunit-simple-example/tree/nb24-issue8082
(branch nb24-issue8082)

Launching composer install should be sufficient to install phpunit 10.5

For the configuration:

It looks like if I tick "test project using just phpunit", it shouldn't add the . to the phpunit command.

Actually the . refers to the directory in which phpunit searches test files. For phpunit, if --test-suffix is not set, it defaults searching only for files having suffix Test.php,.phpt. However, in this project the suffix is _test.php, so the . wouldn't find the test files, and that would explain why no groups are found.

BTW, if I tick "use XML configuration", maybe netbeans shouldn't even append the test dir to any phpunit call? I don't know if this makes sense.

junichi11 · 2024-12-24T02:17:40Z

@tenzap Thanks for the example.

junichi11 · 2024-12-24T02:21:12Z

@tmysik Can we remove .? (Is there a problem?)

netbeans/php/php.phpunit/src/org/netbeans/modules/php/phpunit/commands/PhpUnit.java

Lines 348 to 354 in be051eb

    
           List<String> params = createParams(true); 
        
           addBootstrap(phpModule, params); 
        
           addConfiguration(phpModule, params); 
        
           params.add(LIST_GROUPS_PARAM); 
        
           // list test groups from the current workdir 
        
           params.add("."); // NOI18N 
        
           phpUnit.additionalParameters(params);

tmysik · 2024-12-24T07:35:16Z

@tmysik Can we remove .? (Is there a problem?)

netbeans/php/php.phpunit/src/org/netbeans/modules/php/phpunit/commands/PhpUnit.java

Lines 348 to 354 in be051eb

List<String> params = createParams(true);

addBootstrap(phpModule, params);

addConfiguration(phpModule, params);

params.add(LIST_GROUPS_PARAM);

// list test groups from the current workdir

params.add("."); // NOI18N

phpUnit.additionalParameters(params);

@junichi11 I would guess so, if it works correctly without it. You know, this was implemented a very long time ago, so PHPUnit CLI could have changed several times already 🙂

junichi11 · 2024-12-24T07:59:50Z

@tmysik OK, I thought it may have a specific reason :)
Thank you, Tomas!

tmysik · 2024-12-24T08:04:15Z

It was not working without it, definitely 😁 In ~2010 or so 😁

tenzap · 2024-12-24T08:17:03Z

I think that if a XML config file is set or if we tick to use just phpunit, the dir should not be set,
Otherwise it should be set to the same dir as the one used when not ticking to use just phpunit.
The '.' just reflects the dir in which to search for tests (see phpunit command line options). If not set it would work according to XML config (either the default one, or the one set by '-c' argument) I suppose.

junichi11 · 2024-12-25T02:48:09Z

It doesn't work without . if a config file is not set. So, I will fix it to not add . if a config file is set. Thanks!

tenzap · 2024-12-25T06:52:36Z

Why do you use '.' instead of the test dirs? Wouldn't it be more correct to run the command with the test dir as set in the 'properties/testing' window of the project (which is the value used in the phounit command when not ticking 'use just phpunit' option)?

tmysik · 2024-12-25T08:36:24Z

Maybe, because you can run it on a directory underneath the test one? In such a case, you need the actual directory. Just guessing, I can be wrong, of course.

tenzap · 2024-12-25T15:31:24Z

For example, with phpunit 10.5

$ vendor/bin/phpunit --help
PHPUnit 10.5.40 by Sebastian Bergmann and contributors.

Usage:
  phpunit [options] <directory|file> ...

From what I understand in source code of phpunit 10.5

[options] from cli will override the one set in xml config file if it exists (as per the default search location of that file) or if its location is set by the corresponding command line argument.

See: Registry::init & (new Merger)->merge($cliConfiguration, $xmlConfiguration); which are initially called from Application::run()
https://github.com/sebastianbergmann/phpunit/blob/10.5.40/src/TextUI/Application.php#L100
https://github.com/sebastianbergmann/phpunit/blob/main/src/TextUI/Configuration/Registry.php#L109
<directory|file> (ie. like the . presently used by netbeans when querying groups, or like the test dir that is set when phpunit is invoked without ticking Test project using just phpunit command), if set, will also superseed the one in the xml config file. If not set, it will search for files this the --test-suffix <suffixes> (ie. Only search for test in files with specified suffix(es). Default: Test.php,.phpt) in the current working dir.

See TestSuiteBuilder::build()
https://github.com/sebastianbergmann/phpunit/blob/10.5.40/src/TextUI/Application.php#L397
https://github.com/sebastianbergmann/phpunit/blob/main/src/Framework/TestBuilder.php#L40
https://github.com/sebastianbergmann/phpunit/blob/main/src/TextUI/Configuration/TestSuiteBuilder.php#L42

So netbeans could work like this:

if user checks/ticks Test project using just phpunit command, don't append the <directory|file>, the inclusions of the dirs would be determined as per the XML config file if set, or as per the phpunit default. Otherwise, <directory|file> should be the director(y/ies) set in the "project properties / testing / test dirs" window of netbeans.
if user checks/ticks Ask for test groups before running tests, it should append the <directory|file> depending on whether Test project using just phpunit command is ticked. That's because one would want the list of groups for the tests that phpunit would run based on the way it is configured in netbeans.

junichi11 · 2024-12-25T15:49:16Z

if user checks/ticks Test project using just phpunit command, don't append the <directory|file>, the inclusions of the dirs would be determined as per the XML config file if set, or as per the phpunit default. Otherwise, <directory|file> should be the director(y/ies) set in the "project properties / testing / test dirs" window of netbeans.

if user checks/ticks Ask for test groups before running tests, it should append the <directory|file> depending on whether Test project using just phpunit command is ticked. That's because one would want the list of groups for the tests that phpunit would run based on the way it is configured in netbeans.

Can you provide examples(work/don't work) with exact steps to reproduce them?

tenzap · 2024-12-25T15:49:49Z

isn't the sample I provided sufficient?
It's just about checking/unchecking the boxes in netbeans.
BTW I'm new to phpunit, so that's just supposition and would probably be above my skills to do this in a time-efficient manner.

junichi11 · 2024-12-25T16:05:09Z

Well, Misunderstandings often occur because I'm not you...(We can't often reproduce issues.)
So, please write exact steps.
e.g.

Works fine:

check ...
...
Run the test file
...

Don't work:

check ...
...
...

tenzap · 2024-12-25T17:41:32Z

Sorry for misunderstanding.

For all use cases below, in the project properties / testing / testdirs, set the path to "/absolute/path/to/project/tests" dir.

Group A

This is the expected result for the branch nb24-issue8082 of https://github.com/tenzap/phpunit-simple-example/tree/nb24-issue8082

Case A1

Check only

use Custom PHPUnit Script (/absolute/path/to/project/vendor/bin/phpunit)
Test project using just phpunit command

Run tests (ALT+F6)

Doesn't work as expected: ie. phpunit is invoked without the testdir argument which is correct and since there is no check for 'use XML configuration', no configuration is added to the command line which is correct. However, phpunit prints the usage. This is because it is likely that it is not invoked from the root dir of the project.

Expected behavior: phpunit should be invoked from the root dir of the project. With this changed, phpunit will be able to run the tests.

Case A2

Check only

use XML configuration (/absolute/path/to/project/phpunit.xml)
use Custom PHPUnit Script (/absolute/path/to/project/vendor/bin/phpunit)
Test project using just phpunit command

Run tests (ALT+F6)

Works as expected: ie. phpunit is invoked without with the testdir argument and phpunit runs the test according to xml file.

Case A3

Check only

use XML configuration (/absolute/path/to/project/phpunit.xml)
use Custom PHPUnit Script (/absolute/path/to/project/vendor/bin/phpunit)
Test project using just phpunit command
Ask for test groups before running tests

Run tests (ALT+F6)

Doesn't work as expected: ie. netbeans invokes phpunit with the "." argument.

Expected behavior: It should invoke phpunit without the "." dir

Case A4

Check only

use Custom PHPUnit Script (/absolute/path/to/project/vendor/bin/phpunit)
Test project using just phpunit command
Ask for test groups before running tests

Run tests (ALT+F6)

Doesn't work as expected: ie. netbeans invokes phpunit with the "." argument.

Expected behavior: It should invoke phpunit without the testdir argument. And since there is not check for 'use XML configuration', no configuration is added to the command line which is correct. Moreover, phpunit should be invoked from the root dir of the project (I can't check if this is the case, but I think it is already the case)

Group B

This is the expected result for the branch nb24-issue8082-caseB of https://github.com/tenzap/phpunit-simple-example/tree/nb24-issue8082-caseB. The checkboxes to tick are like the 4 use cases above, except that we always keep Test project using just phpunit command unchecked.

Case B1

Check only

use Custom PHPUnit Script (/absolute/path/to/project/vendor/bin/phpunit)

Run tests (ALT+F6)

Works as expected: ie. phpunit is invoked with the testdir argument which is correct and since there is no check for 'use XML configuration', no configuration is added to the command line which is correct.

Case B2

Check only

use XML configuration (/absolute/path/to/project/phpunit.xml)
use Custom PHPUnit Script (/absolute/path/to/project/vendor/bin/phpunit)

Run tests (ALT+F6)

Works as expected: ie. phpunit is invoked with the testdir argument and phpunit runs the test according to xml file.

Case B3

Check only

use XML configuration (/absolute/path/to/project/phpunit.xml)
use Custom PHPUnit Script (/absolute/path/to/project/vendor/bin/phpunit)
Ask for test groups before running tests

Run tests (ALT+F6)

Doesn't work as expected: ie. netbeans invokes phpunit with the "." argument.

Expected behavior: It should invoke phpunit with the testdir argument as set in the project properties / testing / testdirs, ie "/absolute/path/to/project/tests" dir.

Case B4

Check only

use Custom PHPUnit Script (/absolute/path/to/project/vendor/bin/phpunit)
Ask for test groups before running tests

Run tests (ALT+F6)

Doesn't work as expected: ie. netbeans invokes phpunit with the "." argument. While it lists the groups in netbeans, I think the behavior could be different to be more consistent with the behavior when "use xml configuration" is set.

Expected behavior: It should invoke phpunit with the testdir argument. And since there is no check of 'use XML configuration', no configuration is added to the command line which is correct. Moreover, phpunit should be invoked from the root dir of the project (I can't check if this is the case, but I think it is already the case).

Group A & B

So in all cases:

if "Test project using just phpunit command" is ticked, the testdir or "." (in case of --list-groups) should never be appended. If it is unticked, it should always add the testdir (also in case of --list-group) to the phpunit invocation.
phpunit should be invoked from the root dir of the project in all cases

Hopefully I haven't made mistakes or misunderstood the way phpunit works.

junichi11 · 2024-12-26T03:21:01Z

@tenzap Thanks!

tenzap added kind:bug Bug report or fix needs:triage Requires attention from one of the committers labels Dec 22, 2024

KacerCZ added the PHP [ci] enable extra PHP tests (php/php.editor) label Dec 22, 2024

junichi11 removed the needs:triage Requires attention from one of the committers label Dec 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wrong call to phpunit when asking for test groups #8082

Wrong call to phpunit when asking for test groups #8082

tenzap commented Dec 22, 2024

junichi11 commented Dec 23, 2024

tenzap commented Dec 23, 2024 •

edited

Loading

junichi11 commented Dec 24, 2024

junichi11 commented Dec 24, 2024

tmysik commented Dec 24, 2024

junichi11 commented Dec 24, 2024

tmysik commented Dec 24, 2024

tenzap commented Dec 24, 2024

junichi11 commented Dec 25, 2024 •

edited

Loading

tenzap commented Dec 25, 2024

tmysik commented Dec 25, 2024

tenzap commented Dec 25, 2024 •

edited

Loading

junichi11 commented Dec 25, 2024

tenzap commented Dec 25, 2024 •

edited

Loading

junichi11 commented Dec 25, 2024

tenzap commented Dec 25, 2024

junichi11 commented Dec 26, 2024

Wrong call to phpunit when asking for test groups #8082

Wrong call to phpunit when asking for test groups #8082

Comments

tenzap commented Dec 22, 2024

Apache NetBeans version

What happened

Language / Project Type / NetBeans Component

How to reproduce

Did this work correctly in an earlier version?

Operating System

JDK

Apache NetBeans packaging

Anything else

Are you willing to submit a pull request?

junichi11 commented Dec 23, 2024

tenzap commented Dec 23, 2024 • edited Loading

junichi11 commented Dec 24, 2024

junichi11 commented Dec 24, 2024

tmysik commented Dec 24, 2024

junichi11 commented Dec 24, 2024

tmysik commented Dec 24, 2024

tenzap commented Dec 24, 2024

junichi11 commented Dec 25, 2024 • edited Loading

tenzap commented Dec 25, 2024

tmysik commented Dec 25, 2024

tenzap commented Dec 25, 2024 • edited Loading

junichi11 commented Dec 25, 2024

tenzap commented Dec 25, 2024 • edited Loading

junichi11 commented Dec 25, 2024

tenzap commented Dec 25, 2024

Group A

Case A1

Case A2

Case A3

Case A4

Group B

Case B1

Case B2

Case B3

Case B4

Group A & B

junichi11 commented Dec 26, 2024

tenzap commented Dec 23, 2024 •

edited

Loading

junichi11 commented Dec 25, 2024 •

edited

Loading

tenzap commented Dec 25, 2024 •

edited

Loading

tenzap commented Dec 25, 2024 •

edited

Loading