PT_NOTE alignment, NT_GNU_PROPERTY_TYPE_0, glibc and gold (was: Re: [PATCH] Document GNU_PROPERTY_X86_ISA_1

The .note.gnu.property section with NT_GNU_PROPERTY_TYPE_0 has been
https://github.com/hjl-tools/linux-abi
GNU_PROPERTY_X86_ISA_1_USED and GNU_PROPERTY_X86_ISA_1_NEEDED are
processor-specific program property types for i386 and x86-64.

The specification is incomplete as far as alignment matters are concerned.

Is the link editor supposed to maintain separate segments for notes with
different alignments? Or is it possible to merge the notes into a
single segment, potentially after adjusting alignment?

Is the link editor *required* to produce 8-byte alignment for notes in
ELFCLASS64 objects?

Currently, we do not have agreement between binutils (particularly gold)
and the glibc dynamic loader when it comes to alignment of PT_NOTE
segments. glibc will disregard property notes in ELFCLASS64 objects
which have 4-byte alignment, but gold produces such notes. This needs
to be fixed.

Thanks,
Florian

H.J. Lu

2018-08-16 13:19:33 UTC

The specification is incomplete as far as alignment matters are concerned.

https://github.com/hjl-tools/linux-abi/wiki/linux-abi-draft.pdf

has

2.1.7 Alignment of Note Sections

All entries in a PT_NOTE segment have the same alignment which equals to the
p_align field in program header.
According to gABI, each note entry should be aligned to 4 bytes in 32-bit
objects or 8 bytes in 64-bit objects. But .note.ABI-tag section (see
Section 2.1.6) and .note.gnu.build-id section (see Section 2.1.4) are
aligned
to 4 bytes in both 32-bit and 64-bit objects. Note parser should use p_align for
note alignment, instead of assuming alignment based on ELF file class.

Post by Florian Weimer
Is the link editor supposed to maintain separate segments for notes with
different alignments? Or is it possible to merge the notes into a single
segment, potentially after adjusting alignment?

It is possible. We just need to place 4-byte aligned notes after 8-byte
aligned notes.

Post by Florian Weimer
Is the link editor *required* to produce 8-byte alignment for notes in
ELFCLASS64 objects?

It is decided by the alignment of NOTE section, not by linker.

Post by Florian Weimer
Currently, we do not have agreement between binutils (particularly gold) and
the glibc dynamic loader when it comes to alignment of PT_NOTE segments.
glibc will disregard property notes in ELFCLASS64 objects which have 4-byte
alignment, but gold produces such notes. This needs to be fixed.

I don't believe this is true. See above.

After this commit:

commit 8d81ce0c6d6ca923571e8b2bac132929f9a02973
Author: H.J. Lu <***@gmail.com>
Date: Tue Nov 28 09:56:47 2017 -0800

Properly compute offsets of note descriptor and next note [BZ #22370]

A note header has 3 4-bytes fields, followed by note name and note
descriptor. According to gABI, in a note entry, the note name field,
not note name size, is padded for the note descriptor. And the note
descriptor field, not note descriptor size, is padded for the next
note entry. Notes are aligned to 4 bytes in 32-bit objects and 8 bytes
in 64-bit objects.

For all GNU notes, the name is "GNU" which is 4 bytes. They have the
same format in the first 16 bytes in both 32-bit and 64-bit objects.
They differ by note descriptor size and note type. So far, .note.ABI-tag
and .note.gnu.build-id notes are always aligned to 4 bytes. The exsting
codes compute the note size by aligning the note name size and note
descriptor size to 4 bytes. It happens to produce the same value as
the actual note size by luck since the name size is 4 and offset of the
note descriptor is 16. But it will produce the wrong size when note
alignment is 8 bytes in 64-bit objects.

This patch defines ELF_NOTE_DESC_OFFSET and ELF_NOTE_NEXT_OFFSET to
properly compute offsets of note descriptor and next note. It uses
alignment of PT_NOTE segment to support both 4-byte and 8-byte note
alignments in 64-bit objects. To handle PT_NOTE segments with
incorrect alignment, which may lead to an infinite loop, if segment
alignment is less than 4, we treate alignment as 4 bytes since some
note segments have 0 or 1 byte alignment.

glibc can handle both 4 byte and 8 byte NOTE alignments.

--
H.J.

H.J. Lu

2018-08-16 13:29:34 UTC

The specification is incomplete as far as alignment matters are concerned.

https://github.com/hjl-tools/linux-abi/wiki/linux-abi-draft.pdf
has
2.1.7 Alignment of Note Sections
All entries in a PT_NOTE segment have the same alignment which equals to the
p_align field in program header.
According to gABI, each note entry should be aligned to 4 bytes in 32-bit
objects or 8 bytes in 64-bit objects. But .note.ABI-tag section (see
Section 2.1.6) and .note.gnu.build-id section (see Section 2.1.4) are
aligned
to 4 bytes in both 32-bit and 64-bit objects. Note parser should use p_align for
note alignment, instead of assuming alignment based on ELF file class.

It is possible. We just need to place 4-byte aligned notes after 8-byte
aligned notes.

Wait. We can't since the p_align field is used to compute the next note.
We need at least 2 NOTE segments where there are both 4-byte and
8-byte aligned notes.

Post by Florian Weimer
Is the link editor *required* to produce 8-byte alignment for notes in
ELFCLASS64 objects?

It is decided by the alignment of NOTE section, not by linker.

I don't believe this is true. See above.
commit 8d81ce0c6d6ca923571e8b2bac132929f9a02973
Date: Tue Nov 28 09:56:47 2017 -0800
Properly compute offsets of note descriptor and next note [BZ #22370]
A note header has 3 4-bytes fields, followed by note name and note
descriptor. According to gABI, in a note entry, the note name field,
not note name size, is padded for the note descriptor. And the note
descriptor field, not note descriptor size, is padded for the next
note entry. Notes are aligned to 4 bytes in 32-bit objects and 8 bytes
in 64-bit objects.
For all GNU notes, the name is "GNU" which is 4 bytes. They have the
same format in the first 16 bytes in both 32-bit and 64-bit objects.
They differ by note descriptor size and note type. So far, .note.ABI-tag
and .note.gnu.build-id notes are always aligned to 4 bytes. The exsting
codes compute the note size by aligning the note name size and note
descriptor size to 4 bytes. It happens to produce the same value as
the actual note size by luck since the name size is 4 and offset of the
note descriptor is 16. But it will produce the wrong size when note
alignment is 8 bytes in 64-bit objects.
This patch defines ELF_NOTE_DESC_OFFSET and ELF_NOTE_NEXT_OFFSET to
properly compute offsets of note descriptor and next note. It uses
alignment of PT_NOTE segment to support both 4-byte and 8-byte note
alignments in 64-bit objects. To handle PT_NOTE segments with
incorrect alignment, which may lead to an infinite loop, if segment
alignment is less than 4, we treate alignment as 4 bytes since some
note segments have 0 or 1 byte alignment.
glibc can handle both 4 byte and 8 byte NOTE alignments.
--
H.J.

--
H.J.

Florian Weimer

2018-08-16 13:31:07 UTC

The specification is incomplete as far as alignment matters are concerned.

https://github.com/hjl-tools/linux-abi/wiki/linux-abi-draft.pdf
has
2.1.7 Alignment of Note Sections
All entries in a PT_NOTE segment have the same alignment which equals to the
p_align field in program header.
According to gABI, each note entry should be aligned to 4 bytes in 32-bit
objects or 8 bytes in 64-bit objects. But .note.ABI-tag section (see
Section 2.1.6) and .note.gnu.build-id section (see Section 2.1.4) are
aligned
to 4 bytes in both 32-bit and 64-bit objects. Note parser should use p_align for
note alignment, instead of assuming alignment based on ELF file class.

This is still ambiguous, particularly based on your comments below.

It is possible. We just need to place 4-byte aligned notes after 8-byte
aligned notes.

Based on section 2.1.7, this would not be valid by itself because the
section needs to have 8-byte alignment (to satisfy the property notes
requirement). All notes in the segment need to have the same alignment
(because p_align is supposed to be used for parsing). So reordering
alone will not produce a valid segment.

Part of the problem is that the note header is 12 bytes (not a multiple
of 8), and that the name and descriptor lengths do not include the
padding (which makes sense), so you really need a correct source of
alignment.

If we want to generate a single segment (and I think we should), we need
to realign the notes to a common alignment, either 4 or 8 bytes. That's
what gold seems todo right now, with 4-byte alignment.

Post by Florian Weimer
Is the link editor *required* to produce 8-byte alignment for notes in
ELFCLASS64 objects?

It is decided by the alignment of NOTE section, not by linker.

I don't believe this is true. See above.

Which part? I see the 4-byte segment alignment with gold from
binutils-2.31.1-11.fc29.x86_64.

Post by H.J. Lu
commit 8d81ce0c6d6ca923571e8b2bac132929f9a02973
Date: Tue Nov 28 09:56:47 2017 -0800
Properly compute offsets of note descriptor and next note [BZ #22370]

…

Post by H.J. Lu
glibc can handle both 4 byte and 8 byte NOTE alignments.

There's still this code in glibc, in sysdeps/x86/dl-prop.h:

/* The NT_GNU_PROPERTY_TYPE_0 note must be aliged to 4 bytes in
32-bit objects and to 8 bytes in 64-bit objects. Skip notes
with incorrect alignment. */
if (align != (__ELF_NATIVE_CLASS / 8))
return;

Thanks,
Florian

H.J. Lu

2018-08-16 13:39:26 UTC

The specification is incomplete as far as alignment matters are concerned.

https://github.com/hjl-tools/linux-abi/wiki/linux-abi-draft.pdf
has
2.1.7 Alignment of Note Sections
All entries in a PT_NOTE segment have the same alignment which equals to the
p_align field in program header.
According to gABI, each note entry should be aligned to 4 bytes in 32-bit
objects or 8 bytes in 64-bit objects. But .note.ABI-tag section (see
Section 2.1.6) and .note.gnu.build-id section (see Section 2.1.4) are
aligned
to 4 bytes in both 32-bit and 64-bit objects. Note parser should use p_align for
note alignment, instead of assuming alignment based on ELF file class.

This is still ambiguous, particularly based on your comments below.

https://github.com/hjl-tools/linux-abi/wiki/linux-abi-draft.pdf

conforms to gABI unless stated otherwise.

It is possible. We just need to place 4-byte aligned notes after 8-byte
aligned notes.

Based on section 2.1.7, this would not be valid by itself because the
section needs to have 8-byte alignment (to satisfy the property notes
requirement). All notes in the segment need to have the same alignment
(because p_align is supposed to be used for parsing). So reordering alone
will not produce a valid segment.
Part of the problem is that the note header is 12 bytes (not a multiple of
8), and that the name and descriptor lengths do not include the padding
(which makes sense), so you really need a correct source of alignment.
If we want to generate a single segment (and I think we should), we need to
realign the notes to a common alignment, either 4 or 8 bytes. That's what
gold seems todo right now, with 4-byte alignment.

I was wrong. We need 2 NOTE segments one fore 8-byte alignment and
one for 4-byte alignment.

Post by Florian Weimer
Is the link editor *required* to produce 8-byte alignment for notes in
ELFCLASS64 objects?

It is decided by the alignment of NOTE section, not by linker.

I don't believe this is true. See above.

Which part? I see the 4-byte segment alignment with gold from
binutils-2.31.1-11.fc29.x86_64.

glibc only discards 4-byte aligned NT_GNU_PROPERTY_TYPE_0 note
since NT_GNU_PROPERTY_TYPE_0 note follows gABI. If gold
generates 4 byte alignment, it is a gold bug.

Post by H.J. Lu
commit 8d81ce0c6d6ca923571e8b2bac132929f9a02973
Date: Tue Nov 28 09:56:47 2017 -0800
Properly compute offsets of note descriptor and next note [BZ #22370]

…

Post by H.J. Lu
glibc can handle both 4 byte and 8 byte NOTE alignments.

/* The NT_GNU_PROPERTY_TYPE_0 note must be aliged to 4 bytes in
32-bit objects and to 8 bytes in 64-bit objects. Skip notes
with incorrect alignment. */
if (align != (__ELF_NATIVE_CLASS / 8))
return;

This code is correct. NT_GNU_PROPERTY_TYPE_0 follows gABI.

--
H.J.

Florian Weimer

2018-08-16 14:01:23 UTC

The specification is incomplete as far as alignment matters are concerned.

https://github.com/hjl-tools/linux-abi/wiki/linux-abi-draft.pdf
has
2.1.7 Alignment of Note Sections
All entries in a PT_NOTE segment have the same alignment which equals to the
p_align field in program header.
According to gABI, each note entry should be aligned to 4 bytes in 32-bit
objects or 8 bytes in 64-bit objects. But .note.ABI-tag section (see
Section 2.1.6) and .note.gnu.build-id section (see Section 2.1.4) are
aligned
to 4 bytes in both 32-bit and 64-bit objects. Note parser should use p_align for
note alignment, instead of assuming alignment based on ELF file class.

This is still ambiguous, particularly based on your comments below.

https://github.com/hjl-tools/linux-abi/wiki/linux-abi-draft.pdf
conforms to gABI unless stated otherwise.

I still think there is ambiguity.

Post by H.J. Lu
I was wrong. We need 2 NOTE segments one fore 8-byte alignment and
one for 4-byte alignment.

If one needs to generate two segments, the ABI documents should say so.
Otherwise, linker implementors will assume that there are other ways to
implement this.

Post by H.J. Lu
glibc only discards 4-byte aligned NT_GNU_PROPERTY_TYPE_0 note
since NT_GNU_PROPERTY_TYPE_0 note follows gABI. If gold
generates 4 byte alignment, it is a gold bug.

See above. I don't have a strong opinion how we make gold and glibc
interoperate, but if gold is wrong, the ABI documents should be made
more precise.

Thanks,
Florian

H.J. Lu

2018-08-16 14:43:25 UTC

The specification is incomplete as far as alignment matters are concerned.

https://github.com/hjl-tools/linux-abi/wiki/linux-abi-draft.pdf
has
2.1.7 Alignment of Note Sections
All entries in a PT_NOTE segment have the same alignment which equals to the
p_align field in program header.
According to gABI, each note entry should be aligned to 4 bytes in 32-bit
objects or 8 bytes in 64-bit objects. But .note.ABI-tag section (see
Section 2.1.6) and .note.gnu.build-id section (see Section 2.1.4) are
aligned
to 4 bytes in both 32-bit and 64-bit objects. Note parser should use p_align for
note alignment, instead of assuming alignment based on ELF file class.

This is still ambiguous, particularly based on your comments below.

https://github.com/hjl-tools/linux-abi/wiki/linux-abi-draft.pdf
conforms to gABI unless stated otherwise.

I still think there is ambiguity.

Can you check out

https://github.com/hjl-tools/linux-abi

and send me a patch to make note alignment less ambiguous?

--
H.J.

Florian Weimer

2018-08-16 14:21:05 UTC

Post by H.J. Lu
glibc only discards 4-byte aligned NT_GNU_PROPERTY_TYPE_0 note
since NT_GNU_PROPERTY_TYPE_0 note follows gABI. If gold
generates 4 byte alignment, it is a gold bug.

I filed: https://sourceware.org/bugzilla/show_bug.cgi?id=23535

Thanks,
Florian

H.J. Lu

2018-08-16 17:46:52 UTC

Post by H.J. Lu
glibc only discards 4-byte aligned NT_GNU_PROPERTY_TYPE_0 note
since NT_GNU_PROPERTY_TYPE_0 note follows gABI. If gold
generates 4 byte alignment, it is a gold bug.

I filed: https://sourceware.org/bugzilla/show_bug.cgi?id=23535

A patch is posted at:

https://sourceware.org/ml/binutils/2018-08/msg00306.html

--
H.J.

Mark Wielaard

2018-08-16 19:16:28 UTC

Post by H.J. Lu
glibc only discards 4-byte aligned NT_GNU_PROPERTY_TYPE_0 note
since NT_GNU_PROPERTY_TYPE_0 note follows gABI. If gold
generates 4 byte alignment, it is a gold bug.

I filed: https://sourceware.org/bugzilla/show_bug.cgi?id=23535

I don't think this is a bug in gold, but one in ld:
https://sourceware.org/bugzilla/show_bug.cgi?id=22749

In the GNU abi all ELF Notes are arrays of 32bit words
(and so 4-byte aligned). This is the same for most other
ELF systems. Making the ELF Notes fields 64bit words (and
so 8-byte aligned) in ELFCLASS64 would indeed be what gabi
literally says, but not what GNU systems, and others, follow.

Having a mix of 4-byte words and 8-byte words ELF Notes
in the same ELF file seems unnecessarily confusing and
introduces extra segments and sections.

Cheers,

Mark

H.J. Lu

2018-08-16 19:36:34 UTC

Post by H.J. Lu
glibc only discards 4-byte aligned NT_GNU_PROPERTY_TYPE_0 note
since NT_GNU_PROPERTY_TYPE_0 note follows gABI. If gold
generates 4 byte alignment, it is a gold bug.

I filed: https://sourceware.org/bugzilla/show_bug.cgi?id=23535

https://sourceware.org/bugzilla/show_bug.cgi?id=22749
In the GNU abi all ELF Notes are arrays of 32bit words
(and so 4-byte aligned). This is the same for most other

That is not true. NT_GNU_PROPERTY_TYPE_0 note is
8 byte aligned in 64-bit ELF.

Post by Mark Wielaard
ELF systems. Making the ELF Notes fields 64bit words (and
so 8-byte aligned) in ELFCLASS64 would indeed be what gabi
literally says, but not what GNU systems, and others, follow.
Having a mix of 4-byte words and 8-byte words ELF Notes
in the same ELF file seems unnecessarily confusing and
introduces extra segments and sections.
Cheers,
Mark

--
H.J.

Mark Wielaard

2018-08-17 06:04:30 UTC

Post by H.J. Lu
glibc only discards 4-byte aligned NT_GNU_PROPERTY_TYPE_0 note
since NT_GNU_PROPERTY_TYPE_0 note follows gABI. If gold
generates 4 byte alignment, it is a gold bug.

I filed: https://sourceware.org/bugzilla/show_bug.cgi?id=23535

https://sourceware.org/bugzilla/show_bug.cgi?id=22749
In the GNU abi all ELF Notes are arrays of 32bit words
(and so 4-byte aligned). This is the same for most other

That is not true. NT_GNU_PROPERTY_TYPE_0 note is
8 byte aligned in 64-bit ELF.

Which is the bug we are trying to fix, right?
The issue with that note is that ld generates it with 32bit words,
but with extra padding to make it 8-byte aligned. That makes it
not confirm to the GNU abi and also not to the gabi. It causes all
ELF Note parsers to have to be adjusted. And since there are also
normal GNU abi ELF Notes without the extra padding it causes extra
sections and segments to be created because the alignment differs.
It seems simpler to just fix ld so that glibc can use a normal ELF
note parser for all ELF Notes, than to propagate that bug to other
linkers.

--
H.J.

Florian Weimer

2018-08-17 06:20:53 UTC

Post by H.J. Lu
glibc only discards 4-byte aligned NT_GNU_PROPERTY_TYPE_0 note
since NT_GNU_PROPERTY_TYPE_0 note follows gABI. If gold
generates 4 byte alignment, it is a gold bug.

I filed: https://sourceware.org/bugzilla/show_bug.cgi?id=23535

https://sourceware.org/bugzilla/show_bug.cgi?id=22749
In the GNU abi all ELF Notes are arrays of 32bit words
(and so 4-byte aligned). This is the same for most other
ELF systems. Making the ELF Notes fields 64bit words (and
so 8-byte aligned) in ELFCLASS64 would indeed be what gabi
literally says, but not what GNU systems, and others, follow.

The binaries are already out there and will be around for a while, even
if we reduce the alignment now. So generic ELF parsers need to cope
with these binaries anyway.

Post by Mark Wielaard
Having a mix of 4-byte words and 8-byte words ELF Notes
in the same ELF file seems unnecessarily confusing and
introduces extra segments and sections.

There are no 8-byte word ELF notes on GNU systems, that's an HP-UX
feature. The new 8-byte-aligned notes still have a four-byte header.

But like you, I don't yet see the value of the 8-byte alignment. We
could decide that the current gold behavior is valid, fix glibc, and
move on.

Thanks,
Florian

Mark Wielaard

2018-08-17 06:41:46 UTC

Post by Mark Wielaard
Having a mix of 4-byte words and 8-byte words ELF Notes
in the same ELF file seems unnecessarily confusing and
introduces extra segments and sections.

There are no 8-byte word ELF notes on GNU systems, that's an HP-UX feature.
The new 8-byte-aligned notes still have a four-byte header.

Yes, you are right, HP-UX follows gabi and has 64-bit word notes,
with 8-byte alignment. I meant the different padding used with GNU
style 32-bit word ELF notes to get different alignment styles which
is confusing and none compliant to any standard.

But like you, I don't yet see the value of the 8-byte alignment. We could
decide that the current gold behavior is valid, fix glibc, and move on.

Right. gold seems to produce normal GNU abi ELF Notes, which should
be accepted as is.

Cheers,

Mark

H.J. Lu

2018-08-17 15:10:15 UTC

Post by Mark Wielaard
Having a mix of 4-byte words and 8-byte words ELF Notes
in the same ELF file seems unnecessarily confusing and
introduces extra segments and sections.

There are no 8-byte word ELF notes on GNU systems, that's an HP-UX feature.
The new 8-byte-aligned notes still have a four-byte header.

But like you, I don't yet see the value of the 8-byte alignment. We could
decide that the current gold behavior is valid, fix glibc, and move on.

Right. gold seems to produce normal GNU abi ELF Notes, which should
be accepted as is.

NT_GNU_PROPERTY_TYPE_0 should stay to follow gABI.

--
H.J.

Mark Wielaard

2018-08-17 21:05:24 UTC

But like you, I don't yet see the value of the 8-byte alignment. We could
decide that the current gold behavior is valid, fix glibc, and move on.

Right. gold seems to produce normal GNU abi ELF Notes, which should
be accepted as is.

NT_GNU_PROPERTY_TYPE_0 should stay to follow gABI.

That doesn't make any sense. gABI doesn't have a concept of 32bit word
ELF notes that are 8-byte aligned. That is just a bug in ld when it
generates the note. It also doesn't make sense to generate GNU ELF notes
with slightly different padding added depending on type in the same ELF
file. That just creates confusion, causes you to define different
alignments of notes resulting in extra PT_LOAD segments and results in
bugs like we are discussing now where ELF note parsers fail to parse
some (valid) notes. Lets just agree that the gold linker is correct and
produces consistent GNU abi ELF notes. And lets just fix ld to do the same.

Thanks,

Mark

Cary Coutant

2018-08-21 23:02:21 UTC

But like you, I don't yet see the value of the 8-byte alignment. We could
decide that the current gold behavior is valid, fix glibc, and move on.

Right. gold seems to produce normal GNU abi ELF Notes, which should
be accepted as is.

NT_GNU_PROPERTY_TYPE_0 should stay to follow gABI.

I thought the outcome of the old discussion on the gABI/psABI mailing
lists was that there was no value in breaking compatibility by
introducing an 8-byte aligned note section.

As it is, the psABI definition of the gnu properties note is a
definitional disaster: like a proper note, it's got three 4-byte words
(namesz, descsz, and type), followed by the name field (which happens
to be 4 bytes long in this case), then a desc field. For this note,
the desc field contains a properties array, where each property has
two 4-byte values (pr_type and pr_datasz) and a data field padded to a
multiple of 8 bytes. But it describes the properties array as an array
of 8-byte words, even though it's really composed of the two 4-byte
words and an arbitrary-length buffer (which according to the spec, is
always 8 bytes). Why an 8-byte alignment, though? It doesn't provide
any conceivable benefit.

I think gold is doing the right thing by placing the output in a
4-byte aligned note section, and combining it with other note sections
in a 4-byte aligned PT_NOTE segment.

As long as HJ is planning on breaking compatibility with his new
definition of the x86 properties [1], why not just throw it away and
do it right? Let's stop using NOTE sections for something they weren't
designed for, and instead use a special section type and segment type
(in keeping with existing models like Sun, HP, MIPS, and Arm). Why
should the loader have to go parsing everything in a generic PT_NOTE
segment anyway?

-cary

[1] https://groups.google.com/d/msg/x86-64-abi/-D05GQ3kWrA/stKtIPy8DQAJ

Florian Weimer

2018-08-22 09:39:02 UTC

Post by Cary Coutant

But like you, I don't yet see the value of the 8-byte alignment. We could
decide that the current gold behavior is valid, fix glibc, and move on.

Right. gold seems to produce normal GNU abi ELF Notes, which should
be accepted as is.

NT_GNU_PROPERTY_TYPE_0 should stay to follow gABI.

I thought the outcome of the old discussion on the gABI/psABI mailing
lists was that there was no value in breaking compatibility by
introducing an 8-byte aligned note section.

The code for generating them is still in BFD ld and has been there for
several binutils releases. <cet.h> in GCC 8 generates 8-byte-aligned
notes, too.

glibc 2.28 assumes that no 4-byte-aligned PT_NOTE segments contain GNU
property notes and will search for such notes in 8-byte aligned segments
only. The glibc changes were formally reviewed and checked for
interoperability with BFD ld. I think we have (had) community consensus
for them.

We have built and hundreds (if not thousands) of binaries with
8-byte-aligned notes and shipped them (in Fedora), assuming that there
was upstream consensus about this x86-64 psABI extension.

Post by Cary Coutant
As it is, the psABI definition of the gnu properties note is a
definitional disaster: like a proper note, it's got three 4-byte words
(namesz, descsz, and type), followed by the name field (which happens
to be 4 bytes long in this case), then a desc field. For this note,
the desc field contains a properties array, where each property has
two 4-byte values (pr_type and pr_datasz) and a data field padded to a
multiple of 8 bytes. But it describes the properties array as an array
of 8-byte words, even though it's really composed of the two 4-byte
words and an arbitrary-length buffer (which according to the spec, is
always 8 bytes). Why an 8-byte alignment, though? It doesn't provide
any conceivable benefit.
I think gold is doing the right thing by placing the output in a
4-byte aligned note section, and combining it with other note sections
in a 4-byte aligned PT_NOTE segment.

Based on what I've seen, this is a technically feasible option.

But we have existing binaries with 8-byte-aligned notes, and those won't
go away soon. There is a cost to having 8-byte-aligned notes (gold
fixes, elfutils fixes, plenty of other fixes), and to having
4-byte-aligned notes only (CET-compatible binaries will not be treated
as such because their 8-byte-aligned PT_NOTE section is invalid).

I don't know if there is anything else that is coming down the pipe
which would benefit from 8-byte-aligned notes. Nick has been working on
.gnu.build.attributes non-allocated note sections. They are nominally
4-byte-aligned, but contain 64-bit relocations, but so far, the linker
has not complained, so maybe this is okay.

Post by Cary Coutant
As long as HJ is planning on breaking compatibility with his new
definition of the x86 properties [1], why not just throw it away and
do it right? Let's stop using NOTE sections for something they weren't
designed for, and instead use a special section type and segment type
(in keeping with existing models like Sun, HP, MIPS, and Arm).

This, too, would invalidate existing binaries.

Post by Cary Coutant
Why should the loader have to go parsing everything in a generic PT_NOTE
segment anyway?

With 8-byte alignment, in practice, it is currently quite fast to
discover the new notes because the other notes have 4-byte alignment, so
the PT_NOTE segment with the property notes should be pleasantly short.

Again, maybe we shouldn't have built all those binaries, but we thought
the binutils community at least was in favor of the 8-byte-aligned
notes, This puts me in a difficult decision: The technical facts seem
to favor 4-byte-aligned notes (but again, there might be further
features that actually need 8-byte alignment). But the existing
binaries with 8-byte-aligned notes provide a strong business
justification for Red Hat to keep support for the 8-byte-aligned notes.

Thanks,
Florian

Mark Wielaard

2018-08-22 10:08:29 UTC

Post by Florian Weimer
glibc 2.28 assumes that no 4-byte-aligned PT_NOTE segments contain GNU
property notes and will search for such notes in 8-byte aligned segments
only. The glibc changes were formally reviewed and checked for
interoperability with BFD ld. I think we have (had) community consensus for
them.

I think there was actually consensus that they were a bad idea.
There is a bug and a fix for BFD ld:
https://sourceware.org/bugzilla/show_bug.cgi?id=22749

Unfortunately it didn't make it before the binutils release:
https://sourceware.org/ml/binutils/2018-01/msg00354.html
Which might have given the impression it wasn't a simple bug.
Sorry about that. I have raised the issue a couple of time,
but apparently not clearly enough.

I think the best thing forward is to actually apply and backport
the fix now (and fix cet.h) to generate proper GNU ELF notes.
It is unfortunate there are a few binaries out there with
these bad notes. But they haven't hit any long term distros
yet. And they should impact only CET, which isnt really
supported anywhere yet because there isn't any hardware
for it.

I understand the appeal of introducing a new section and
segment type for storing these properties, but I fear that
will be much more effort and coordination.

Cheers,

Mark

Cary Coutant

2018-08-22 23:36:28 UTC

Post by Cary Coutant
Why should the loader have to go parsing everything in a generic PT_NOTE
segment anyway?

I understand the benefit of having a PT_NOTE segment containing
nothing but the one note, and this is certainly expedient, but it's
quite ugly and, I think, fragile. This really underscores the value of
defining a bespoke segment type for this one purpose.

How difficult would it be for glibc if the linker would consume the
existing .note.gnu.property sections, then generate an output section
with the same format (maybe minus the SHT_NOTE overhead), but using a
new section type in its own new segment type? You could continue to
look for 8-byte aligned legacy PT_NOTE segments, but could also look
for the new PT_GNU_PROPERTY segment.

-cary

Florian Weimer

2018-08-24 18:39:16 UTC

Post by Cary Coutant

Post by Cary Coutant
Why should the loader have to go parsing everything in a generic PT_NOTE
segment anyway?

Yes, the existing note processing in linkers makes this fragile.

Post by Cary Coutant
How difficult would it be for glibc if the linker would consume the
existing .note.gnu.property sections, then generate an output section
with the same format (maybe minus the SHT_NOTE overhead), but using a
new section type in its own new segment type? You could continue to
look for 8-byte aligned legacy PT_NOTE segments, but could also look
for the new PT_GNU_PROPERTY segment.

It does not sound difficult to implement. This is based on the
assumption that the current dynamic loader ignores unknown segment types
(which I think is true).

But I don't view this a win because it doesn't take the 8-byte notes out
of circulation. We'd still have to support both indefinitely, and would
have update GCC as well. We can just keep using the existing code.

Thanks,
Florian

Michael Matz

2018-08-23 14:41:56 UTC

Hi,

Not really. You had one person favoring 8byte alignment (variously
because the gABI says so, even though that's irrelevant for .gnu.note,
and/or because then int64 accesses can be used without having to write a
read_int64 macro that deals with unalignment), and the rest either
ignoring this or saying "ugh, bad idea, breaks compatibility and breaks
forward compat because not self-describing anymore". The former put in
work to implement 8byte alignment, ignoring the compat issues. Now you
actually see those issues. Everybody looses. That's what you get if
people create facts before consensus.

Ciao,
Michael.

Florian Weimer

2018-08-23 14:43:54 UTC

Post by Michael Matz
Hi,

Not really.

Sorry for being ambiguous. I was talking about review of the glibc
changes, not the binutils changes in the paragraph above.

Thanks,
Florian

H.J. Lu

2018-08-17 15:13:22 UTC

Post by H.J. Lu
glibc only discards 4-byte aligned NT_GNU_PROPERTY_TYPE_0 note
since NT_GNU_PROPERTY_TYPE_0 note follows gABI. If gold
generates 4 byte alignment, it is a gold bug.

I filed: https://sourceware.org/bugzilla/show_bug.cgi?id=23535

The binaries are already out there and will be around for a while, even if
we reduce the alignment now. So generic ELF parsers need to cope with these
binaries anyway.

Post by Mark Wielaard
Having a mix of 4-byte words and 8-byte words ELF Notes
in the same ELF file seems unnecessarily confusing and
introduces extra segments and sections.

There are no 8-byte word ELF notes on GNU systems, that's an HP-UX feature.
The new 8-byte-aligned notes still have a four-byte header.
But like you, I don't yet see the value of the 8-byte alignment. We could
decide that the current gold behavior is valid, fix glibc, and move on.

Gold didn't follow the spec and should be fixed.

--
H.J.

Florian Weimer

2018-09-19 19:22:36 UTC

At the GNU Tools Cauldron in Manchester, we had an ad-hoc side meeting
to work out a consensus among many of those involved so far.

I'm including the minutes below. It is my hope that this can serve as a
guide for obtaining community consensus.

Cary, after reviewing the minutes, would you be willing to join the
emerging consensus, and re-review H.J.'s patch to align gold's behavior
with BFD ld?

Thanks,
Florian

# Background for the GNU property notes design

The notes are intended for subarchitecture markup, including the recent
Intel CET (CET). (CET is an upcoming Intel CPU feature which needs
forwards and backwards compatibility for binaries, across and with the
existing ecosystem. CPU features need to be enabled per process if all
loaded objects support it.)

Putting the markup in a new, separate program header segment (PT_*)
has more risk across the ecosystem than PT_NOTE. We know that PT_NOTE
works because different new notes have been added to the PT_NOTE
segment over time.

gABI does not completely and unambiguously describe the layout of note
segments and sections and the format of the note header, although the
historic interpretation has been that the note header has three 32-bit
words, independently of the ELF class, particularly on GNU/Linux.
(But HP-UX uses 8-byte words in the note header in the 64-bit case.)

gABI does say that ELF notes should be 8-byte aligned for the 64-bit
ELF class. But there are commonly used existing 4-byte-aligned notes:
the ABI tag and the build ID. The Linux kernel, when generating
coredumps, may put 0 or 1 in the p_align field of the PT_NOTE segment;
this is a clear bug, the notes are actually 4-byte-aligned.

There could be relocations in the note; nothing currently forbids
that. The stack size in GNU_PROPERTY_STACK_SIZE is 64-bit for the 64-bit
ELF class (> 4 GiB may not make sense, but is theoretically possible).
x86-64 does not need alignment, but strict alignment targets need it,
including perhaps for relocations.

Therefore, a conscious decision was made to follow gABI and use 8-byte
alignment for the new note types (GNU property notes).

# Current state

8-byte aligned notes have been in binutils (BFD ld) for two years.
GCC 8 includes them (in <cet.h>). glibc 2.28 ignores property notes
which are not 8-byte-aligned.

Distributions have built binaries with 8-byte-aligned GNU property notes.

Current binutils gold merges all allocated note section into a 4-byte
aligned note segment.

Relocations in notes work, and annobin uses 8-byte relocations in
4-byte-aligned notes in relocatable (ET_REL) objects on multiple
64-bit targets.

Due to the kernel bug which generates PT_NOTE headers with p_align
having values 0 or 1, some consumers ignore the p_align field and
always assume 4.

# Some available options

(1) Only 4-byte-aligned notes are valid. Existing binaries with 8-byte
alignment become invalid.

(2) The current 8+4 alignment mix (8-byte alignment for GNU Property
notes on ELFCLASS64, 4-byte alignment for older note types such as ABI
tag and build ID) as implemented by BFD ld, GCC, and glibc.

(3) Change GCC and linkers to generate 4-byte-aligned notes
(producers), but keep backwards compatibility with existing
8-byte-aligned notes in binaries (in consumers).

(4) A new PT_* segment for property notes.

# Consensus position of those present

(4) Does not have working code today, so it was not considered
further. (It is also considered larger risk, as discussed above.)

A simplification over the status quo is not achievable with (3)
because consumers still need to be updated. This means the choice is
between (1) and (2).

(1) makes existing binaries invalid, and there was general agreement
that this is a bad idea. It also fails to support notes with relocation
on ELFCLASS64 strict-alignment targets.

This leaves us with (2). There was agreement that link editor
behavior for this approach is currently underspecified. Some
ABI-level document should indicate what linkers should do (for
example, produce separate PT_NOTE segments for different alignments,
report link failures for note sections of different alignment and the
same name in relocatable links).

In order to obtain consensus among those present, we agreed that option
(2) (that is, 8+4 alignment mix as implemented in BFD ld, GCC, and glibc
today) should only be adopted with 8-byte-alignment for the GNU property
notes, and not automatically for all future notes. No precedent for
future notes should be established by this decision.

# Further actions

If the community adopts the consensus of those present at the meeting,
binutils gold should change to adapt to BFD ld, but only for
.note.gnu.property.

Florian agreed to try to document the expected link editor behavior
regarding notes and note merging.

Michael Matz

2018-09-21 12:55:16 UTC

Hi,

(didn't take part in the side meeting, so sorry if this was discussed)

Post by Florian Weimer
Distributions have built binaries with 8-byte-aligned GNU property notes.

Which ones? I know that we (SUSE) avoided CET-marked binaries for the
reason we are in the current predicament. I.e. is that _really_ a problem
at all (or a large one)?

Thing is, there are also binaries (or there could be) that are currently
"valid" that the align-8 approach makes invalid, so why say that's a
problem for the align-4 approach, but not for the align-8 one?

Ciao,
Michael.

Florian Weimer

2018-09-21 13:04:04 UTC

Post by Michael Matz
(didn't take part in the side meeting, so sorry if this was discussed)

I don't think this came up.

Post by Michael Matz

Post by Florian Weimer
Distributions have built binaries with 8-byte-aligned GNU property notes.

Which ones?

Fedora 29 and downstream.

Post by Michael Matz
Thing is, there are also binaries (or there could be) that are currently
"valid" that the align-8 approach makes invalid, so why say that's a
problem for the align-4 approach, but not for the align-8 one?

The 4-byte-aligned GNU property notes are ignored by glibc even if they
are otherwise syntactically valid. So even if there is a hypothetical
distribution that is mostly built with binutils gold and thus has
4-byte-aligned GNU property notes, it would still not work as intended,
which makes me believe that such a distribution does not exist. 8-)

Thanks,
Florian

Cary Coutant

2018-09-26 17:39:30 UTC

Post by Florian Weimer
At the GNU Tools Cauldron in Manchester, we had an ad-hoc side meeting
to work out a consensus among many of those involved so far.
I'm including the minutes below. It is my hope that this can serve as a
guide for obtaining community consensus.
Cary, after reviewing the minutes, would you be willing to join the
emerging consensus, and re-review H.J.'s patch to align gold's behavior
with BFD ld?

Yes, I will yield to the consensus, and I'll change gold's treatment
of the new note section.

But please allow me to express my disappointment and a few minor
quibbles for the record....

Post by Florian Weimer
Putting the markup in a new, separate program header segment (PT_*)
has more risk across the ecosystem than PT_NOTE. We know that PT_NOTE
works because different new notes have been added to the PT_NOTE
segment over time.

It's discouraging that the community is so reluctant to make use of
the established and well-proven extension mechanism that was designed
into ELF. An extension mechanism that people are afraid to use is not
an extension mechanism. I think it's a bit hyperbolic to imply that we
*don't* know that a new segment type for this purpose will work, since
that very mechanism has already been put to use on at least 3
platforms.

Post by Florian Weimer
gABI does not completely and unambiguously describe the layout of note
segments and sections and the format of the note header, although the
historic interpretation has been that the note header has three 32-bit
words, independently of the ELF class, particularly on GNU/Linux.
(But HP-UX uses 8-byte words in the note header in the 64-bit case.)

I think it's more fair to say that the gABI does in fact completely
and unambiguously describe the layout of 64-bit note sections, but
that Sun and Gnu accidentally misread or ignored the spec in their
implementations, establishing a precedent that led to confusion and
compatibility issues. The choice to use 8-byte alignment for this note
-- ignoring public objections to the design -- added even more
confusion.

Post by Florian Weimer
# Some available options
(1) Only 4-byte-aligned notes are valid. Existing binaries with 8-byte
alignment become invalid.
(2) The current 8+4 alignment mix (8-byte alignment for GNU Property
notes on ELFCLASS64, 4-byte alignment for older note types such as ABI
tag and build ID) as implemented by BFD ld, GCC, and glibc.
(3) Change GCC and linkers to generate 4-byte-aligned notes
(producers), but keep backwards compatibility with existing
8-byte-aligned notes in binaries (in consumers).
(4) A new PT_* segment for property notes.
# Consensus position of those present
(4) Does not have working code today, so it was not considered
further. (It is also considered larger risk, as discussed above.)
A simplification over the status quo is not achievable with (3)
because consumers still need to be updated. This means the choice is
between (1) and (2).

If the solution must meet the criterion that consumers don't need to
be updated, then (1) is not a choice, as you noted below. If that was
indeed an immutable criterion, this meeting wasn't really necessary,
was it?

Post by Florian Weimer
(1) makes existing binaries invalid, and there was general agreement
that this is a bad idea. It also fails to support notes with relocation
on ELFCLASS64 strict-alignment targets.
This leaves us with (2). There was agreement that link editor
behavior for this approach is currently underspecified. Some
ABI-level document should indicate what linkers should do (for
example, produce separate PT_NOTE segments for different alignments,
report link failures for note sections of different alignment and the
same name in relocatable links).
In order to obtain consensus among those present, we agreed that option
(2) (that is, 8+4 alignment mix as implemented in BFD ld, GCC, and glibc
today) should only be adopted with 8-byte-alignment for the GNU property
notes, and not automatically for all future notes. No precedent for
future notes should be established by this decision.

I'm happy you've decided to restrict this solution to just the one
note. I'd prefer, however, that we do establish a precedent that this
should be the only exception, and that *all* future notes must follow
the a priori convention of using 4-byte aligned notes (on those
platforms that did not implement according to the gABI spec).

Post by Florian Weimer
Florian agreed to try to document the expected link editor behavior
regarding notes and note merging.

Some requirements I'd like to see:

(1) An NT_GNU_PROPERTY_TYPE_0 note section must be named
".note.gnu.property" with type SHT_NOTE.

(2) A ".note.gnu.property" section must contain one and only one note,
whose type must be NT_GNU_PROPERTY_TYPE_0.

(3) A ".note.gnu.property" section must have sh_align of 8, and its
length must be a multiple of 8.

(4) A linker is expected to combine ".note.gnu.property" sections in a
manner described by the ABI documentation, and place the combined
result, as a single note, in a unique SHT_NOTE section named
".note.gnu.property", and in a unique PT_NOTE segment.

-cary

H.J. Lu

2018-09-26 18:36:09 UTC