forked from boostorg/endian
522 lines
20 KiB
Plaintext
522 lines
20 KiB
Plaintext
////
|
|
Copyright 2011-2016 Beman Dawes
|
|
|
|
Distributed under the Boost Software License, Version 1.0.
|
|
(http://www.boost.org/LICENSE_1_0.txt)
|
|
////
|
|
|
|
[#buffers]
|
|
# Endian Buffer Types
|
|
|
|
## Introduction
|
|
|
|
The internal byte order of arithmetic types is traditionally called
|
|
*endianness*. See the http://en.wikipedia.org/wiki/Endian[Wikipedia] for a full
|
|
exploration of *endianness*, including definitions of *big endian* and *little
|
|
endian*.
|
|
|
|
Header `boost/endian/buffers.hpp` provides `endian_buffer`, a portable endian
|
|
integer binary buffer class template with control over byte order, value type,
|
|
size, and alignment independent of the platform's native endianness. Typedefs
|
|
provide easy-to-use names for common configurations.
|
|
|
|
Use cases primarily involve data portability, either via files or network
|
|
connections, but these byte-holders may also be used to reduce memory use, file
|
|
size, or network activity since they provide binary numeric sizes not otherwise
|
|
available.
|
|
|
|
Class `endian_buffer` is aimed at users who wish explicit control over when
|
|
endianness conversions occur. It also serves as the base class for the
|
|
<<arithmetic,endian_arithmetic>> class template, which is aimed at users who
|
|
wish fully automatic endianness conversion and direct support for all normal
|
|
arithmetic operations.
|
|
|
|
## Example
|
|
|
|
The `example/endian_example.cpp` program writes a binary file containing
|
|
four-byte, big-endian and little-endian integers:
|
|
|
|
```
|
|
#include <iostream>
|
|
#include <cstdio>
|
|
#include <boost/endian/buffers.hpp> // see Synopsis below
|
|
#include <boost/static_assert.hpp>
|
|
|
|
using namespace boost::endian;
|
|
|
|
namespace
|
|
{
|
|
// This is an extract from a very widely used GIS file format.
|
|
// Why the designer decided to mix big and little endians in
|
|
// the same file is not known. But this is a real-world format
|
|
// and users wishing to write low level code manipulating these
|
|
// files have to deal with the mixed endianness.
|
|
|
|
struct header
|
|
{
|
|
big_int32_`buf_`t file_code;
|
|
big_int32_`buf_`t file_length;
|
|
little_int32_`buf_`t version;
|
|
little_int32_`buf_`t shape_type;
|
|
};
|
|
|
|
const char* filename = "test.dat";
|
|
}
|
|
|
|
int main(int, char* [])
|
|
{
|
|
header h;
|
|
|
|
BOOST_STATIC_ASSERT(sizeof(h) == 16U); // reality check
|
|
|
|
h.file_code = 0x01020304;
|
|
h.file_length = sizeof(header);
|
|
h.version = 1;
|
|
h.shape_type = 0x01020304;
|
|
|
|
// Low-level I/O such as POSIX read/write or <cstdio>
|
|
// fread/fwrite is sometimes used for binary file operations
|
|
// when ultimate efficiency is important. Such I/O is often
|
|
// performed in some {cpp} wrapper class, but to drive home the
|
|
// point that endian integers are often used in fairly
|
|
// low-level code that does bulk I/O operations, <cstdio>
|
|
// fopen/fwrite is used for I/O in this example.
|
|
|
|
std::FILE* fi = std::fopen(filename, "wb"); // MUST BE BINARY
|
|
|
|
if (!fi)
|
|
{
|
|
std::cout << "could not open " << filename << '\n';
|
|
return 1;
|
|
}
|
|
|
|
if (std::fwrite(&h, sizeof(header), 1, fi)!= 1)
|
|
{
|
|
std::cout << "write failure for " << filename << '\n';
|
|
return 1;
|
|
}
|
|
|
|
std::fclose(fi);
|
|
|
|
std::cout << "created file " << filename << '\n';
|
|
|
|
return 0;
|
|
}
|
|
```
|
|
|
|
After compiling and executing `example/endian_example.cpp`, a hex dump of
|
|
`test.dat` shows:
|
|
|
|
```
|
|
01020304 00000010 01000000 04030201
|
|
```
|
|
|
|
Notice that the first two 32-bit integers are big endian while the second two
|
|
are little endian, even though the machine this was compiled and run on was
|
|
little endian.
|
|
|
|
## Limitations
|
|
|
|
Requires `<climits>`, `CHAR_BIT == 8`. If `CHAR_BIT` is some other value,
|
|
compilation will result in an `#error`. This restriction is in place because the
|
|
design, implementation, testing, and documentation has only considered issues
|
|
related to 8-bit bytes, and there have been no real-world use cases presented
|
|
for other sizes.
|
|
|
|
In {cpp}03, `endian_buffer` does not meet the requirements for POD types because
|
|
it has constructors, private data members, and a base class. This means that
|
|
common use cases are relying on unspecified behavior in that the {cpp} Standard
|
|
does not guarantee memory layout for non-POD types. This has not been a problem
|
|
in practice since all known {cpp} compilers lay out memory as if `endian` were
|
|
a POD type. In {cpp}11, it is possible to specify the default constructor as
|
|
trivial, and private data members and base classes no longer disqualify a type
|
|
from being a POD type. Thus under {cpp}11, `endian_buffer` will no longer be
|
|
relying on unspecified behavior.
|
|
|
|
## Feature set
|
|
|
|
* Big endian| little endian | native endian byte ordering.
|
|
* Signed | unsigned
|
|
* Unaligned | aligned
|
|
* 1-8 byte (unaligned) | 1, 2, 4, 8 byte (aligned)
|
|
* Choice of value type
|
|
|
|
## Enums and typedefs
|
|
|
|
Two scoped enums are provided:
|
|
|
|
```
|
|
enum class order {big, little, native};
|
|
|
|
enum class align {no, yes};
|
|
```
|
|
|
|
One class template is provided:
|
|
|
|
```
|
|
template <order Order, typename T, std::size_t Nbits,
|
|
align Align = align::no>
|
|
class endian_buffer;
|
|
```
|
|
|
|
Typedefs, such as `big_int32_buf_t`, provide convenient naming conventions for
|
|
common use cases:
|
|
|
|
[%header,cols=5*]
|
|
|===
|
|
|Name |Alignment |Endianness |Sign |Sizes in bits (n)
|
|
|big_intn_buf_t |no |big |signed |8,16,24,32,40,48,56,64
|
|
|big_uintn_buf_t |no |big |unsigned |8,16,24,32,40,48,56,64
|
|
|little_intn_buf_t |no |little |signed |8,16,24,32,40,48,56,64
|
|
|little_uintn_buf_t |no |little |unsigned |8,16,24,32,40,48,56,64
|
|
|native_intn_buf_t |no |native |signed |8,16,24,32,40,48,56,64
|
|
|native_uintn_buf_t |no |native |unsigned |8,16,24,32,40,48,56,64
|
|
|big_intn_buf_at |yes |big |signed |8,16,32,64
|
|
|big_uintn_buf_at |yes |big |unsigned |8,16,32,64
|
|
|little_intn_buf_at |yes |little |signed |8,16,32,64
|
|
|little_uintn_buf_at |yes |little |unsigned |8,16,32,64
|
|
|===
|
|
|
|
The unaligned types do not cause compilers to insert padding bytes in classes
|
|
and structs. This is an important characteristic that can be exploited to
|
|
minimize wasted space in memory, files, and network transmissions.
|
|
|
|
CAUTION: Code that uses aligned types is possibly non-portable because alignment
|
|
requirements vary between hardware architectures and because alignment may be
|
|
affected by compiler switches or pragmas. For example, alignment of an 64-bit
|
|
integer may be to a 32-bit boundary on a 32-bit machine and to a 64-bit boundary
|
|
on a 64-bit machine. Furthermore, aligned types are only available on
|
|
architectures with 8, 16, 32, and 64-bit integer types.
|
|
|
|
TIP: Prefer unaligned buffer types.
|
|
|
|
TIP: Protect yourself against alignment ills. For example:
|
|
[none]
|
|
{blank}::
|
|
+
|
|
```
|
|
static_assert(sizeof(containing_struct) == 12, "sizeof(containing_struct) is wrong");
|
|
```
|
|
|
|
Note: One-byte big and little buffer types have identical layout on all
|
|
platforms, so they never actually reverse endianness. They are provided to
|
|
enable generic code, and to improve code readability and searchability.
|
|
|
|
## Class template `endian_buffer`
|
|
|
|
An `endian_buffer` is a byte-holder for arithmetic types with
|
|
user-specified <<buffers_endianness,endianness>>, value type, size, and
|
|
<<buffers_alignment,alignment>>.
|
|
|
|
### Synopsis
|
|
|
|
```
|
|
#include <boost/endian/conversion.hpp
|
|
|
|
namespace boost
|
|
{
|
|
namespace endian
|
|
{
|
|
// C++11 features emulated if not available
|
|
|
|
enum class align {no, yes};
|
|
|
|
template <order Order, class T, std::size_t Nbits,
|
|
align Align = align::no>
|
|
class endian_buffer
|
|
{
|
|
public:
|
|
typedef T value_type;
|
|
|
|
endian_buffer() noexcept = default;
|
|
explicit endian_buffer(T v) noexcept;
|
|
|
|
endian_buffer& operator=(T v) noexcept;
|
|
value_type value() const noexcept;
|
|
const char* data() const noexcept;
|
|
protected:
|
|
implementaton-defined endian_value; // for exposition only
|
|
};
|
|
|
|
// stream inserter
|
|
template <class charT, class traits, order Order, class T,
|
|
std::size_t n_bits, align Align>
|
|
std::basic_ostream<charT, traits>&
|
|
operator<<(std::basic_ostream<charT, traits>& os,
|
|
const endian_buffer<Order, T, n_bits, Align>& x);
|
|
|
|
// stream extractor
|
|
template <class charT, class traits, order Order, class T,
|
|
std::size_t n_bits, align A>
|
|
std::basic_istream<charT, traits>&
|
|
operator>>(std::basic_istream<charT, traits>& is,
|
|
endian_buffer<Order, T, n_bits, Align>& x);
|
|
|
|
// typedefs
|
|
|
|
// unaligned big endian signed integer buffers
|
|
typedef endian_buffer<order::big, int_least8_t, 8> big_int8_buf_t;
|
|
typedef endian_buffer<order::big, int_least16_t, 16> big_int16_buf_t;
|
|
typedef endian_buffer<order::big, int_least32_t, 24> big_int24_buf_t;
|
|
typedef endian_buffer<order::big, int_least32_t, 32> big_int32_buf_t;
|
|
typedef endian_buffer<order::big, int_least64_t, 40> big_int40_buf_t;
|
|
typedef endian_buffer<order::big, int_least64_t, 48> big_int48_buf_t;
|
|
typedef endian_buffer<order::big, int_least64_t, 56> big_int56_buf_t;
|
|
typedef endian_buffer<order::big, int_least64_t, 64> big_int64_buf_t;
|
|
|
|
// unaligned big endian unsigned integer buffers
|
|
typedef endian_buffer<order::big, uint_least8_t, 8> big_uint8_buf_t;
|
|
typedef endian_buffer<order::big, uint_least16_t, 16> big_uint16_buf_t;
|
|
typedef endian_buffer<order::big, uint_least32_t, 24> big_uint24_buf_t;
|
|
typedef endian_buffer<order::big, uint_least32_t, 32> big_uint32_buf_t;
|
|
typedef endian_buffer<order::big, uint_least64_t, 40> big_uint40_buf_t;
|
|
typedef endian_buffer<order::big, uint_least64_t, 48> big_uint48_buf_t;
|
|
typedef endian_buffer<order::big, uint_least64_t, 56> big_uint56_buf_t;
|
|
typedef endian_buffer<order::big, uint_least64_t, 64> big_uint64_buf_t;
|
|
|
|
// unaligned little endian signed integer buffers
|
|
typedef endian_buffer<order::little, int_least8_t, 8> little_int8_buf_t;
|
|
typedef endian_buffer<order::little, int_least16_t, 16> little_int16_buf_t;
|
|
typedef endian_buffer<order::little, int_least32_t, 24> little_int24_buf_t;
|
|
typedef endian_buffer<order::little, int_least32_t, 32> little_int32_buf_t;
|
|
typedef endian_buffer<order::little, int_least64_t, 40> little_int40_buf_t;
|
|
typedef endian_buffer<order::little, int_least64_t, 48> little_int48_buf_t;
|
|
typedef endian_buffer<order::little, int_least64_t, 56> little_int56_buf_t;
|
|
typedef endian_buffer<order::little, int_least64_t, 64> little_int64_buf_t;
|
|
|
|
// unaligned little endian unsigned integer buffers
|
|
typedef endian_buffer<order::little, uint_least8_t, 8> little_uint8_buf_t;
|
|
typedef endian_buffer<order::little, uint_least16_t, 16> little_uint16_buf_t;
|
|
typedef endian_buffer<order::little, uint_least32_t, 24> little_uint24_buf_t;
|
|
typedef endian_buffer<order::little, uint_least32_t, 32> little_uint32_buf_t;
|
|
typedef endian_buffer<order::little, uint_least64_t, 40> little_uint40_buf_t;
|
|
typedef endian_buffer<order::little, uint_least64_t, 48> little_uint48_buf_t;
|
|
typedef endian_buffer<order::little, uint_least64_t, 56> little_uint56_buf_t;
|
|
typedef endian_buffer<order::little, uint_least64_t, 64> little_uint64_buf_t;
|
|
|
|
// unaligned native endian signed integer types
|
|
typedef implementation-defined_int8_buf_t native_int8_buf_t;
|
|
typedef implementation-defined_int16_buf_t native_int16_buf_t;
|
|
typedef implementation-defined_int24_buf_t native_int24_buf_t;
|
|
typedef implementation-defined_int32_buf_t native_int32_buf_t;
|
|
typedef implementation-defined_int40_buf_t native_int40_buf_t;
|
|
typedef implementation-defined_int48_buf_t native_int48_buf_t;
|
|
typedef implementation-defined_int56_buf_t native_int56_buf_t;
|
|
typedef implementation-defined_int64_buf_t native_int64_buf_t;
|
|
|
|
// unaligned native endian unsigned integer types
|
|
typedef implementation-defined_uint8_buf_t native_uint8_buf_t;
|
|
typedef implementation-defined_uint16_buf_t native_uint16_buf_t;
|
|
typedef implementation-defined_uint24_buf_t native_uint24_buf_t;
|
|
typedef implementation-defined_uint32_buf_t native_uint32_buf_t;
|
|
typedef implementation-defined_uint40_buf_t native_uint40_buf_t;
|
|
typedef implementation-defined_uint48_buf_t native_uint48_buf_t;
|
|
typedef implementation-defined_uint56_buf_t native_uint56_buf_t;
|
|
typedef implementation-defined_uint64_buf_t native_uint64_buf_t;
|
|
|
|
// aligned big endian signed integer buffers
|
|
typedef endian_buffer<order::big, int8_t, 8, align::yes> big_int8_buf_at;
|
|
typedef endian_buffer<order::big, int16_t, 16, align::yes> big_int16_buf_at;
|
|
typedef endian_buffer<order::big, int32_t, 32, align::yes> big_int32_buf_at;
|
|
typedef endian_buffer<order::big, int64_t, 64, align::yes> big_int64_buf_at;
|
|
|
|
// aligned big endian unsigned integer buffers
|
|
typedef endian_buffer<order::big, uint8_t, 8, align::yes> big_uint8_buf_at;
|
|
typedef endian_buffer<order::big, uint16_t, 16, align::yes> big_uint16_buf_at;
|
|
typedef endian_buffer<order::big, uint32_t, 32, align::yes> big_uint32_buf_at;
|
|
typedef endian_buffer<order::big, uint64_t, 64, align::yes> big_uint64_buf_at;
|
|
|
|
// aligned little endian signed integer buffers
|
|
typedef endian_buffer<order::little, int8_t, 8, align::yes> little_int8_buf_at;
|
|
typedef endian_buffer<order::little, int16_t, 16, align::yes> little_int16_buf_at;
|
|
typedef endian_buffer<order::little, int32_t, 32, align::yes> little_int32_buf_at;
|
|
typedef endian_buffer<order::little, int64_t, 64, align::yes> little_int64_buf_at;
|
|
|
|
// aligned little endian unsigned integer buffers
|
|
typedef endian_buffer<order::little, uint8_t, 8, align::yes> little_uint8_buf_at;
|
|
typedef endian_buffer<order::little, uint16_t, 16, align::yes> little_uint16_buf_at;
|
|
typedef endian_buffer<order::little, uint32_t, 32, align::yes> little_uint32_buf_at;
|
|
typedef endian_buffer<order::little, uint64_t, 64, align::yes> little_uint64_buf_at;
|
|
|
|
// aligned native endian typedefs are not provided because
|
|
// <cstdint> types are superior for this use case
|
|
|
|
} // namespace endian
|
|
} // namespace boost
|
|
```
|
|
|
|
The `implementation-defined` text in typedefs above is either `big` or `little`
|
|
according to the native endianness of the platform.
|
|
|
|
The expository data member `endian_value` stores the current value of an
|
|
`endian_value` object as a sequence of bytes ordered as specified by the
|
|
`Order` template parameter. The `implementation-defined` type of
|
|
`endian_value` is a type such as `char[Nbits/CHAR_BIT]` or `T` that meets the
|
|
requirements imposed by the `Nbits` and `Align` template parameters. The
|
|
`CHAR_BIT` macro is defined in `<climits>`. The only value of
|
|
`CHAR_BIT` that is required to be supported is 8.
|
|
|
|
Template parameter `T` is required to be a standard integer type ({cpp}std,
|
|
3.9.1) and `sizeof(T)*CHAR_BIT` is required to be greater or equal to `Nbits`.
|
|
|
|
### Members
|
|
|
|
```
|
|
endian_buffer() noexcept = default;
|
|
```
|
|
[none]
|
|
* {blank}
|
|
+
|
|
Effects:: Constructs an uninitialized object of type `endian_buffer<Order, T,
|
|
Nbits, Align>`.
|
|
|
|
```
|
|
explicit endian_buffer(T v) noexcept;
|
|
```
|
|
[none]
|
|
* {blank}
|
|
+
|
|
Effects:: Constructs an object of type `endian_buffer<Order, T, Nbits, Align>`.
|
|
Postcondition:: `value() == v & mask`, where `mask` is a constant of type
|
|
`value_type` with `Nbits` low-order bits set to one.
|
|
Remarks:: If `Align` is `align::yes` then endianness conversion, if required,
|
|
is performed by `boost::endian::endian_reverse`.
|
|
|
|
```
|
|
endian_buffer& operator=(T v) noexcept;
|
|
```
|
|
[none]
|
|
* {blank}
|
|
+
|
|
Postcondition:: `value() == v & mask`, where `mask` is a constant of type
|
|
`value_type` with `Nbits` low-order bits set to one.
|
|
Returns:: `*this`.
|
|
Remarks:: If `Align` is `align::yes` then endianness conversion, if required, is
|
|
performed by `boost::endian::endian_reverse`.
|
|
|
|
```
|
|
value_type value() const noexcept;
|
|
```
|
|
[none]
|
|
* {blank}
|
|
+
|
|
Returns:: `endian_value`, converted to `value_type`, if required, and having the
|
|
endianness of the native platform.
|
|
Remarks:: If `Align` is `align::yes` then endianness conversion, if required, is
|
|
performed by `boost::endian::endian_reverse`.
|
|
|
|
```
|
|
const char* data</a>() const noexcept;
|
|
```
|
|
[none]
|
|
* {blank}
|
|
+
|
|
Returns:: A pointer to the first byte of `endian_value`.
|
|
[none]
|
|
|
|
### Non-member functions
|
|
|
|
```
|
|
template <class charT, class traits, order Order, class T,
|
|
std::size_t n_bits, align Align>
|
|
std::basic_ostream<charT, traits>& operator<<(std::basic_ostream<charT, traits>& os,
|
|
const endian_buffer<Order, T, n_bits, Align>& x);
|
|
```
|
|
[none]
|
|
* {blank}
|
|
+
|
|
Returns:: `os << x.value()`.
|
|
|
|
```
|
|
template <class charT, class traits, order Order, class T,
|
|
std::size_t n_bits, align A>
|
|
std::basic_istream<charT, traits>& operator>>(std::basic_istream<charT, traits>& is,
|
|
endian_buffer<Order, T, n_bits, Align>& x);
|
|
```
|
|
[none]
|
|
* {blank}
|
|
+
|
|
Effects:: As if:
|
|
+
|
|
```
|
|
T i;
|
|
if (is >> i)
|
|
x = i;
|
|
```
|
|
Returns:: `is`.
|
|
|
|
## FAQ
|
|
|
|
See the <<overview_faq,Endian home page>> FAQ for a library-wide FAQ.
|
|
|
|
Why not just use Boost.Serialization?::
|
|
Serialization involves a conversion for every object involved in I/O. Endian
|
|
integers require no conversion or copying. They are already in the desired
|
|
format for binary I/O. Thus they can be read or written in bulk.
|
|
|
|
Are endian types PODs?::
|
|
Yes for {cpp}11. No for {cpp}03, although several
|
|
<<buffers_compilation,macros>> are available to force PODness in all cases.
|
|
|
|
What are the implications of endian integer types not being PODs with {cpp}03 compilers?::
|
|
They can't be used in unions. Also, compilers aren't required to align or lay
|
|
out storage in portable ways, although this potential problem hasn't prevented
|
|
use of Boost.Endian with real compilers.
|
|
|
|
What good is native endianness?::
|
|
It provides alignment and size guarantees not available from the built-in
|
|
types. It eases generic programming.
|
|
|
|
Why bother with the aligned endian types?::
|
|
Aligned integer operations may be faster (as much as 10 to 20 times faster) if
|
|
the endianness and alignment of the type matches the endianness and alignment
|
|
requirements of the machine. The code, however, is likely to be somewhat less
|
|
portable than with the unaligned types.
|
|
|
|
## Design considerations for Boost.Endian buffers
|
|
|
|
* Must be suitable for I/O - in other words, must be memcpyable.
|
|
* Must provide exactly the size and internal byte ordering specified.
|
|
* Must work correctly when the internal integer representation has more bits
|
|
that the sum of the bits in the external byte representation. Sign extension
|
|
must work correctly when the internal integer representation type has more
|
|
bits than the sum of the bits in the external bytes. For example, using
|
|
a 64-bit integer internally to represent 40-bit (5 byte) numbers must work for
|
|
both positive and negative values.
|
|
* Must work correctly (including using the same defined external
|
|
representation) regardless of whether a compiler treats char as signed or
|
|
unsigned.
|
|
* Unaligned types must not cause compilers to insert padding bytes.
|
|
* The implementation should supply optimizations with great care. Experience
|
|
has shown that optimizations of endian integers often become pessimizations
|
|
when changing machines or compilers. Pessimizations can also happen when
|
|
changing compiler switches, compiler versions, or CPU models of the same
|
|
architecture.
|
|
|
|
## {cpp}11
|
|
|
|
The availability of the {cpp}11
|
|
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2346.htm[Defaulted
|
|
Functions] feature is detected automatically, and will be used if present to
|
|
ensure that objects of `class endian_buffer` are trivial, and thus
|
|
PODs.
|
|
|
|
## Compilation
|
|
|
|
Boost.Endian is implemented entirely within headers, with no need to link to
|
|
any Boost object libraries.
|
|
|
|
Several macros allow user control over features:
|
|
|
|
* BOOST_ENDIAN_NO_CTORS causes `class endian_buffer` to have no
|
|
constructors. The intended use is for compiling user code that must be
|
|
portable between compilers regardless of {cpp}11
|
|
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2346.htm[Defaulted
|
|
Functions] support. Use of constructors will always fail,
|
|
* BOOST_ENDIAN_FORCE_PODNESS causes BOOST_ENDIAN_NO_CTORS to be defined if the
|
|
compiler does not support {cpp}11
|
|
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2346.htm[Defaulted
|
|
Functions]. This is ensures that objects of `class endian_buffer` are PODs, and
|
|
so can be used in {cpp}03 unions. In {cpp}11, `class endian_buffer` objects are
|
|
PODs, even though they have constructors, so can always be used in unions.
|