RFC: Unique testnames

[Nick Clifton]

Hi Guys,

I am planning on committing the attached patch, but I thought that it
would be a good idea to check with the community first to see if there
are any comments.

The patch fixes an issue with the binutils testsuites where some tests
have identical names. This is a problem when in comes to reviewing
failures and trying to identify exactly which test has failed, and it
is an issue for automatic testing frameworks which need unique names
for each test.

In most cases I have just added a number or string to the end of tests
which have identical names, but in one case I made a bigger change:
The "objcopy zlib-gnu compress debug sections 3" test in the binutils'
compress.exp file appears twice, and the two tests appear to be
completely identical. I have assumed that the intent was that the
second version of the test would check the copying of an object file
containing sections compressed using the zlib-gabi method. So I have
tweaked the test to do this.

One other change that might also affect people who run regression
tests is that there was a bug in ld-size/size.exp where the size-3e
test was not properly escaped, meaning that it was not run... So this
patch fixes that too.

Any comments before I check this patch in ?

Cheers
Nick

binutils/ChangeLog
2018-09-13 Nick Clifton <***@redhat.com>

* testsuite/binutils-all/compress.exp: Rename second "objcopy
zlib-gnu compress debug sections 3" test to "objcopy zlib-gabi
compress debug sections 3" and use gabi object files instead
of gnu object files.
* testsuite/binutils-all/objcopy.exp: Add suffix to the names
of the "ELF group" tests.
* testsuite/binutils-all/readelf.exp (proc readelf_find_size):
Add an iteration parameter and include it in the name of the
test. Update callers to include an iteration count.

gas/ChangeLog
2018-09-13 Nick Clifton <***@redhat.com>

* testuite/gas/elf/group0a.d: Add extra details to the test
name.
* testuite/gas/elf/group0b.d: Likewise.
* testuite/gas/elf/group1a.d: Likewise.
* testuite/gas/elf/group1b.d: Likewise.
* testuite/gas/elf/group0b.d: Likewise.
* testuite/gas/elf/section9.d: Likewise.
* testuite/gas/i386/ilp32/lns/lns-common-1.d: Likewise.
* testuite/gas/i386/ilp32/lns/lns-duplicate-1.d: Likewise.

gas/ChangeLog
2018-09-13 Nick Clifton <***@redhat.com>

* testuite/ld/ld-elf/audit.exp: Differentiate the names of the
two "Run with shared with --audit" tests.
* testuite/ld/ld-elf/compress.exp: Differentiate the zlib
compressed debug output test names.
* testuite/ld/ld-i386/tlspie1.d: Add extra details to the test
name.
* testuite/ld/ld-i386/tlspie2.d: Likewise.
* testuite/ld/ld-size/size.exp: Add missing escapes to the end
of lines in the size-3e test.
* testuite/ld/ld-unique/unique.exp: Differentiate the names of
the two "Checking unique PIC object" tests.
* testuite/ld/ld-x86-64/tlspie1.d: Add extra details to the test
name.

[Jeff Law]

Post by Nick Clifton
Hi Guys,
I am planning on committing the attached patch, but I thought that it
would be a good idea to check with the community first to see if there
are any comments.
The patch fixes an issue with the binutils testsuites where some tests
have identical names. This is a problem when in comes to reviewing
failures and trying to identify exactly which test has failed, and it
is an issue for automatic testing frameworks which need unique names
for each test.
In most cases I have just added a number or string to the end of tests
The "objcopy zlib-gnu compress debug sections 3" test in the binutils'
compress.exp file appears twice, and the two tests appear to be
completely identical. I have assumed that the intent was that the
second version of the test would check the copying of an object file
containing sections compressed using the zlib-gabi method. So I have
tweaked the test to do this.
One other change that might also affect people who run regression
tests is that there was a bug in ld-size/size.exp where the size-3e
test was not properly escaped, meaning that it was not run... So this
patch fixes that too.
Any comments before I check this patch in ?

So to give wider background to the community. It's the .sum file
comparison tool provided by GCC (contrib/compare_tests) that doesn't
handle this case gracefully.

If the tests have different PASS/FAIL states, the tool will report that
there were regressions/errors in the new testrun. Uniqueness of the
testnames avoids this problem.

The irony is that the Codesourcery guys were raising the uniqueness of
testnames issues for GCC eons ago (in the QMtest discussions). I
largely ignored it at the time since I've always done my comparisons by
hand (thankfully others took on the task of addressing this issue for
GCC). Of course this stuff matters with automatic regression testers :-)

jeff

[Pedro Alves]

Post by Jeff Law
So to give wider background to the community. It's the .sum file
comparison tool provided by GCC (contrib/compare_tests) that doesn't
handle this case gracefully.
If the tests have different PASS/FAIL states, the tool will report that
there were regressions/errors in the new testrun. Uniqueness of the
testnames avoids this problem.
The irony is that the Codesourcery guys were raising the uniqueness of
testnames issues for GCC eons ago (in the QMtest discussions). I
largely ignored it at the time since I've always done my comparisons by
hand (thankfully others took on the task of addressing this issue for
GCC). Of course this stuff matters with automatic regression testers :-)

We're pretty anal about this in GDB too, for the same reasons. :-) See:

https://sourceware.org/gdb/wiki/GDBTestcaseCookbook#Make_sure_test_messages_are_unique

Note GDB avoids using " (foo)" at end of names as a way to make test
names unique. Not sure whether that's relevant for binutils:

https://sourceware.org/gdb/wiki/GDBTestcaseCookbook#Do_not_use_.22tail_parentheses.22_on_test_messages

Thanks,
Pedro Alves

[Nick Clifton]

Hi Pedro,

Post by Pedro Alves
https://sourceware.org/gdb/wiki/GDBTestcaseCookbook#Make_sure_test_messages_are_unique

Thanks for the reference.

Post by Pedro Alves
Note GDB avoids using " (foo)" at end of names as a way to make test
https://sourceware.org/gdb/wiki/GDBTestcaseCookbook#Do_not_use_.22tail_parentheses.22_on_test_messages

Well the use of parenthesised comments at the end of test names is
failry well established in the binutils now, so I am not going to
change it with this patch. Maybe one day in the future.


Anyway - since there have been no objections, I am going to check
the patch in.

Cheers
Nick