Linker exposing private symbols

[Jeffrey Walton]

I'm trying to understand some behavior I am seeing when linking a
wrapper shared object to an archive. The archive is a C++ library and
has hundreds of members and thousands (maybe 10's of thousands) of
functions. The wrapper shared object exports 2 symbols using GCC
attributes.

With symbol visibility and without --exclude-libs:

$ g++ -o sha256_wrapper.so -g2 -O3 -shared -fPIC -ffunction-sections
-fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden
sha256_wrapper.o ./libcryptopp.a -Wl,--gc-sections

Results in:

$ nm -gCD sha256_wrapper.so | grep ' T ' | wc -l
816

With symbol visibility and with --exclude-libs,All:

$ g++ -o sha256_wrapper.so -DNDEBUG -g2 -O3 -shared -fPIC
-ffunction-sections -fdata-sections -fvisibility=hidden
-fvisibility-inlines-hidden sha256_wrapper.o ./libcryptopp.a
-Wl,--gc-sections -Wl,--exclude-libs,All

$ nm -gCD sha256_wrapper.so | grep ' T ' | wc -l
816

With symbol visibility and with --exclude-libs,ALL:

$ nm -gCD sha256_wrapper.so | grep ' T '
00000000000184e8 T _fini
00000000000069d0 T _init
00000000000095b0 T sha256_hash_message
0000000000009710 T sha256_verify_digest

The last one is the one I expect from all of them because we are using
symbol visibility.

I'm on Fedora 28 and GCC version is 8.1.1 and LD version is 2.29.1.

I have two questions. First, why doesn't the toolchain honor the
visibility request (why do I need --exclude-libs)? I uess another way
to put it is, how can symbols be re-exported if they are private?
Second, why is ALL case sensitive (and without a diagnostic)?

Jeff

[Alan Modra]

Post by Jeffrey Walton
I have two questions. First, why doesn't the toolchain honor the
visibility request (why do I need --exclude-libs)? I uess another way
to put it is, how can symbols be re-exported if they are private?

What makes you think the toolchain is misbehaving? Your visibility
command line options affect the code you are compiling, but you of
course are linking against other object files as well. Maybe all the
symbols that seem unexpected to you come from libcryptopp.a or libc?

Post by Jeffrey Walton
Second, why is ALL case sensitive (and without a diagnostic)?

Seems good to me. You asked ld to exclude All.a symbols.

[Jeffrey Walton]

Post by Alan Modra

Post by Jeffrey Walton
I have two questions. First, why doesn't the toolchain honor the
visibility request (why do I need --exclude-libs)? I uess another way
to put it is, how can symbols be re-exported if they are private?

What makes you think the toolchain is misbehaving? Your visibility
command line options affect the code you are compiling, but you of
course are linking against other object files as well.

Thanks Alan.

I feel like the toolchain is misbehaving because I asked for 2 symbols
to be exported but that did not happen.

There's nothing special about me. Others have encountered the same,
like https://stackoverflow.com/q/2222162/608639 and
https://stackoverflow.com/q/37934388/608639 .

I'll have to think about things some more. Maybe the complaint is, we
asked GCC to ensure 2 symbols were exported (and the rest private) but
GCC did not fulfill the request when it drove link. I may be splitting
too many hairs, though.

Post by Alan Modra
Maybe all the
symbols that seem unexpected to you come from libcryptopp.a or libc?

Yes, the symbols are from the archives.

Post by Alan Modra

Post by Jeffrey Walton
Second, why is ALL case sensitive (and without a diagnostic)?

Seems good to me. You asked ld to exclude All.a symbols.

Just to play devils advocate... I don't have a library ALL.a either.

Jeff

[Michael Matz]

Hi,

Post by Jeffrey Walton
I feel like the toolchain is misbehaving because I asked for 2 symbols
to be exported but that did not happen.
There's nothing special about me. Others have encountered the same,
like https://stackoverflow.com/q/2222162/608639 and

That one is confusing compiling and linking.

Post by Jeffrey Walton
https://stackoverflow.com/q/37934388/608639 .

And that one is misreading output of nm.

Post by Jeffrey Walton
I'll have to think about things some more. Maybe the complaint is, we
asked GCC to ensure 2 symbols were exported (and the rest private) but
GCC did not fulfill the request when it drove link. I may be splitting
too many hairs, though.

But you don't need to ask only the compiler, you also need to ask the link
editor if foreign object files are involved (where the compiler didn't
have a chance to mark symbols as hidden). What you effectively did was:

(assume no hidden attributes in sources)

% gcc -c file1.c
% gcc -c -fvisibility=hidden file2.c
% gcc -fvisibility=hidden -shared -o result.so file1.o file2.o

and now expect that the symbols from file1 are hidden even though you
didn't compile that file for this. (In your case the cryptopp.a file is a
collection of files similar to file1.o) Well, as -fvisibility=hidden is a
compile-only option it's ignored for the linking step.

Post by Jeffrey Walton

Post by Alan Modra

Post by Jeffrey Walton
Second, why is ALL case sensitive (and without a diagnostic)?

Command line options and file names are generally case sensitive.

Post by Jeffrey Walton

Post by Alan Modra
Seems good to me. You asked ld to exclude All.a symbols.

Just to play devils advocate... I don't have a library ALL.a either.

To cite:

--exclude-libs lib,lib,...
Specifies a list of archive libraries from which symbols should not
be automatically exported. The library names may be delimited by
commas or colons. Specifying "--exclude-libs ALL" excludes symbols
in all archive libraries from automatic export. This option is
available only for the i386 PE targeted port of the linker and for
ELF targeted ports. For i386 PE, symbols explicitly listed in a
.def file are still exported, regardless of this option. For ELF
targeted ports, symbols affected by this option will be treated as
hidden.

All working as documented and designed. (And yes, this means this won't
work if you have in fact a static archive named ALL.a, tough luck).


Ciao,
Michael.

[Jeffrey Walton]

Post by Michael Matz
...

Post by Jeffrey Walton
There's nothing special about me. Others have encountered the same,
like https://stackoverflow.com/q/2222162/608639 and

That one is confusing compiling and linking.

Yeah, we need to see the compile and link command. To hazard a guess,
he is adding -fvisibility=hidden but after the compiler drives link he
is getting unwanted symbols in the shared object. The toolchain is not
doing what the user expect.

Post by Michael Matz

Post by Jeffrey Walton
https://stackoverflow.com/q/37934388/608639 .

And that one is misreading output of nm.

Yeah, my bad. Here's another example:
https://stackoverflow.com/q/1601900/608639.

The common theme is the user explicitly requests all symbols to be
private with -fvisibility=hidden (sans what is explicitly exported)
but that's not happening once the toolchain works its magic.

Jeff

[Michael Matz]

Hi,

Post by Michael Matz
That one is confusing compiling and linking.

Yeah, we need to see the compile and link command. To hazard a guess, he
is adding -fvisibility=hidden but after the compiler drives link he is
getting unwanted symbols in the shared object. The toolchain is not
doing what the user expect.

Because the expectations are wrong. -fvisibility=hidden is a compile time
option, no link time option. It has no effect whatsoever while linking.

The common theme is the user explicitly requests all symbols to be
private with -fvisibility=hidden

The user might wish that that option is requesting this, but that doesn't
make it true. I'm not sure what you're trying to get at.


Ciao,
Michael.

[Jeffrey Walton]

Post by Michael Matz
Hi,

Post by Michael Matz
That one is confusing compiling and linking.

Yeah, we need to see the compile and link command. To hazard a guess, he
is adding -fvisibility=hidden but after the compiler drives link he is
getting unwanted symbols in the shared object. The toolchain is not
doing what the user expect.

Because the expectations are wrong. -fvisibility=hidden is a compile time
option, no link time option. It has no effect whatsoever while linking.

The common theme is the user explicitly requests all symbols to be
private with -fvisibility=hidden

The user might wish that that option is requesting this, but that doesn't
make it true. I'm not sure what you're trying to get at.

What I am stating is the shared object does not meet expectations.
When the compiler drives link we expect the everything to be hidden
except what is explicitly marked public/default.

Here is the GCC man page on it
(https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html):

-fvisibility=[default|internal|hidden|protected]

Set the default ELF image symbol visibility to the specified
option—all symbols are marked with this unless overridden
within the code...

The compiler docs clearly state all symbols are private when the
private option is used.

Jeff

[Alan Modra]

Post by Jeffrey Walton

Post by Michael Matz
Hi,

Post by Michael Matz
That one is confusing compiling and linking.

Yeah, we need to see the compile and link command. To hazard a guess, he
is adding -fvisibility=hidden but after the compiler drives link he is
getting unwanted symbols in the shared object. The toolchain is not
doing what the user expect.

Because the expectations are wrong. -fvisibility=hidden is a compile time
option, no link time option. It has no effect whatsoever while linking.

The common theme is the user explicitly requests all symbols to be
private with -fvisibility=hidden

The user might wish that that option is requesting this, but that doesn't
make it true. I'm not sure what you're trying to get at.

What I am stating is the shared object does not meet expectations.
When the compiler drives link we expect the everything to be hidden
except what is explicitly marked public/default.

You have been told once by me and twice by Michael that the visibility
options you are passing to gcc affect code generation, not linking.

We're telling you this in the hope that it might educate you, but if
you wish to hold on to your expectations you're free to do that. Just
don't complain when your belief that you can walk through walls
results in a sore nose.

Post by Jeffrey Walton
Here is the GCC man page on it

Take note of the section.

Post by Jeffrey Walton
-fvisibility=[default|internal|hidden|protected]
Set the default ELF image symbol visibility to the specified
option—all symbols are marked with this unless overridden
within the code...
The compiler docs clearly state all symbols are private when the
private option is used.
Jeff

[Jeffrey Walton]

Post by Alan Modra

...

You have been told once by me and twice by Michael that the visibility
options you are passing to gcc affect code generation, not linking.
We're telling you this in the hope that it might educate you, but if
you wish to hold on to your expectations you're free to do that. Just
don't complain when your belief that you can walk through walls
results in a sore nose.

Thanks everyone.

There's an open issue for GCC that hopes to close the documentation
gaps. https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87190 .

Jeff