diff --git a/doc/index.html b/doc/index.html index 2db2416..01464d3 100644 --- a/doc/index.html +++ b/doc/index.html @@ -1,4 +1,4 @@ - + @@ -65,8 +65,9 @@

Abstract

-

Boost.Endian provides facilities to manipulate the endianness of integers, -floating point, and user defined data.

+

Boost.Endian provides facilities to manipulate the +endianness of integers, +floating point numbers, and user-defined types.

@@ -114,11 +115,11 @@ at different ends.

See the Wikipedia's Endianness article for an extensive discussion of endianness.

-

Programmers can usually ignore endianness, except perhaps for reading a core +

Programmers can usually ignore endianness, except when reading a core dump on little-endian systems. But programmers will have to deal with endianness when exchanging binary integers and binary floating point values between computer systems with differing endianness, whether by physical file transfer or over a network. -And programmers may want to use the library if minimizing either internal or -external data sizes is advantagous.

+And programmers may also want to use the library when minimizing either internal or +external data sizes is advantageous.

Introduction to the Boost.Endian library

The Boost.Endian library provides three different approaches to dealing with @@ -132,116 +133,137 @@ cases where it is preferred to the other approaches.

Endian conversion functions - The -application uses the built-in integer and floating point types, and calls the +application uses the built-in integer and floating point types to hold values, and calls the provided conversion functions to convert byte ordering as needed. Both mutating and non-mutating conversions are supplied, and each comes in unconditional and conditional variants.

Endian buffer types - The application uses the provided endian buffer types -which mimic the -built-in integer types. For example, big_buf32_t or little_floatbuf64_t. -Buffer types with lengths of 1 through 8 bytes are supported, rather than just -2, 4, and 8 bytes. The types may be aligned (name suffix _t) or unaligned -(name suffix _ut).

+to hold values, and explicitly converts to and from the built-in integer and +floating point types to perform arithmetic. Buffer lengths of 1 through 8 bytes +are supported, rather than just 2, 4, and 8 bytes. The types may be aligned or +unaligned.

Endian arithmetic types - The application uses the provided endian -arithmetic types -which mimic the -built-in arithmetic types. For example, big_int32_t or little_float64_t. -Arithmetic integer types with lengths of 1 through 8 bytes are supported, rather than just -2, 4, and 8 byte integers. The types may be aligned (name suffix _t) or unaligned -(name suffix _ut).

+arithmetic types, which supply the same operations as the built-in C++ +arithmetic types. All conversions are implicit. Arithmetic integer types with +lengths of 1 through 8 bytes are supported, rather than just 2, 4, and 8 byte +integers. The types may be aligned.

Boost Endian is a header-only library.

-

Choosing between conversion functions, buffers types, -and arithmetic types

+

Choosing between endian conversion functions, endian buffer types, +and endian arithmetic types

-

Which endianness approach is best depends on +

Which approach to endianness is best depends on the interaction between +approach characteristics and application needs.

- - - - - - - - - - - - -
Needs that favor one approach over the other
Endian types may be better for - these needsEndian conversion functions may be - better for - these needs
-
    -

  • A need to save overall time when I/O time is more important than - numeric variable CPU time.

    • -

  • A need to simplify program logic and eliminate logic - errors. Since the endian types mimic the built-in types, there is no need to reason about the current endianness of variables - and that can simplify program logic and eliminate logic errors.

    • -

  • A need to use unusual integer sizes (i.e. 3, 5, - 6, or 7 bytes) to reduce internal and external space usage and - save I/O time.

    • -

  • A need to portably use unaligned variables or eliminate padding bytes in - structures, either to save I/O time by reducing internal and external space usage, - or to conform to a data layout over which you have no control. For - example, the structure below relies on a compiler extension, so is - non-portable, and operations on S::b will fail on hardware platforms that - require 32-bit alignment. -

    +

    The characteristics that differentiate the approaches are the endianness +invariants, conversion explicitness, arithmetic operations, sizes available, and +alignment requirements.

    + +

    Endianness invariants

    + +
    + +

    Endian conversion functions use objects of the ordinary C++ arithmetic +types like int or unsigned short to hold values. That +breaks the implicit invariant that the C++ language rules apply. The usual +language rules only apply if the endianness of the object is currently set by +the conversion functions to the native endianness for the platform. That can +make it very hard to reason about complex logic flow, and result in difficult to +find bugs.

    + +

    Endian buffer and arithmetic types hold values internally as arrays of +characters with an invariant that the endianness of the array never changes. +That makes these types easy to use and programs easy to maintain.

    + +
    + +

    Conversion explicitness

    + +
    + +

    Endian conversion functions and buffer types never perform +implicit conversions. This gives users explicit control of when conversion +occurs, and may help avoid unnecessary conversions.

    + +

    Endian arithmetic types perform conversion implicitly. That makes +these types very easy to use, but can result in unnecessary conversions. Failure +to hoist conversions out of inner loops can bring a performance penalty.

    + +
    + +

    Arithmetic operations

    + +
    + +

    Endian conversion functions do not supply arithmetic +operations, but this is not a concern since this approach uses ordinary C++ +arithmetic types to hold values.

    + +

    Endian buffer types do not supply arithmetic operations. Although this +approach avoids unnecessary conversions, it can result in the introduction of +additional variables and confuse maintenance programmers.

    + +

    Endian arithmetic types do supply arithmetic operations. They +are very easy to use if lots of arithmetic is involved.

    + +
    + +

    Sizes available

    + +
    + +

    Endianness conversion functions only support 1, 2, 4, and 8 byte +integers. That's sufficient for many applications.

    + +

    Endian buffer and arithmetic types support 1, 2, 3, 4, 5, 6, 7, and 8 +byte integers. For an application where memory use or I/O speed is the limiting +factor, using sizes tailored to application needs can be very useful.

    + +
    + +

    Alignments available

    + +
    + +

    Endianness conversion functions only support aligned integer and +floating-point types. That's sufficient for most applications.

    + +

    Endian buffer and arithmetic types support both aligned and unaligned +integer and floating-point types. Unaligned types are rarely needed, but when +needed they are often very useful and workarounds are painful. For example,

    + +
    +

    Non-portable code like this:

    struct S {
    -   uint16_t a;
    -   uint32_t b;
    +   uint16_t a;  // big endian
    +   uint32_t b;  // big endian
    } __attribute__ ((packed));

    -

    These problems are eliminated by defining S like this:

    +

    Can be replaced with portable code like this:

    struct S {
      big_uint16_ut a;
      big_uint32_ut b;
    };

    -
    • -
  • -

    Programmer preference.

    • -
- 		-
    -
  • A need to save overall time when numeric variable CPU use time is more - important that I/O time.
    • -
  • -

    A need to pass structures to third-party libraries expecting a - specific structure format.

    • -
  • -

    A need to leverage knowledge of developers who have been using C byte - swapping - functions for years.

    • -
  • -

    Programmer preference.

    • -
-
+ -

Warning: Endian conversion functions are dangerous unless the current -endianness of a variable is always known. For example, if an exception is thrown -and there is no way in a catch block to know a variable's current endianness, -then any use of that variable including I/O is inherently unsafe. This danger is -not limited to exceptions; all uses of a variable whose endianness is unknown -are unsafe, period.

+

Built-in support for Intrinsics

-

Recent compilers, including GCC, Clang, and Microsoft, supply built-in support for byte swapping -intrinsics. Such support is automatically detected and -used since it may in smaller and faster generated code, particularly for release +

Supply compilers, including GCC, Clang, and Visual C++, supply built-in support for byte swapping intrinsics. +The library uses these intrinsics when available since they may result in smaller and faster generated code, particularly for release builds.

Defining BOOST_ENDIAN_NO_INTRINSICS will suppress use -of the intrinsics. Please try defining it if you get compiler errors, such as -header byteswap.h not being found.

+of the intrinsics. Useful when intrinsic headers such as +byteswap.h are not being found on your platform.

The macro BOOST_ENDIAN_INTRINSIC_MSG is defined as either "no byte swap intrinsics" or a string describing the particular set of intrinsics being used.

@@ -260,8 +282,8 @@ particular set of intrinsics being used.

result to a file - Endian type approach - Endian conversion approach + Endian arithmetic type approach + Endian conversion function approach @@ -280,9 +302,9 @@ int32_t x; ... read into x from a file ... -big_endian(x); +big_to_native_inplace(x); x += 100; -big_endian(x); +native_to_big_inplace(x); ... write x to a file ... @@ -294,7 +316,8 @@ big_endian(x); release builds, regardless of the native endianness of the machine. That's because optimizing compilers will likely generate exactly the same code for each. That conclusion was confirmed by -studying the generated assembly code for GCC and Visual C++.

+studying the generated assembly code for GCC and Visual C++. Furthermore, time +spent doing I/O will determine the speed of this application.

Now consider a slightly different problem: 

@@ -308,8 +331,8 @@ studying the generated assembly code for GCC and Visual C++.

result to a file - Endian type approach - Endian conversion approach + Endian arithmetic type approach + Endian conversion function approach @@ -328,12 +351,12 @@ for (int32_t i = 0; i < 1000000; ++i) ... read into x from a file ... -big_endian(x); +big_to_native_inplace(x); for (int32_t i = 0; i < 1000000; ++i) x += i; -big_endian(x); +native_to_big_inplace(x); ... write x to a file ... @@ -341,10 +364,10 @@ big_endian(x); -

With the Endian type approach, an implicit conversion from and then back to +

With the Endian arithmetic approach, on little endian platforms an implicit conversion from and then back to big endian is done inside the loop. With the Endian conversion function -approach, the conversions are explicit, so only need to be done once, before and -after the loop.

+approach, the user has ensured the conversions are done outside the loop, so the +code may run more quickly on little endian platforms.

Timings for Example 2 (conversion functions hoisted out of loop)

@@ -361,201 +384,204 @@ setup. 32-bit intrinsics.)

- - - - - + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
GNU C++ version 4.7.0
Iterations: 1000000000, Intrinsics: __builtin_bswap16, etc.
Test CaseEndian
type		Endian
conversion
function
GNU C++ version 4.7.0
Iterations: 1000000000, Intrinsics: __builtin_bswap16, etc.
Test CaseEndian
arithmetic		Endian
conversion
function
16-bit aligned big endian1.37 s0.81 s
16-bit aligned little endian0.83 s0.81 s
16-bit unaligned big endian1.09 s0.83 s
16-bit unaligned little endian1.09 s0.81 s
32-bit aligned big endian0.98 s0.27 s
32-bit aligned little endian0.28 s0.27 s
32-bit unaligned big endian3.82 s0.27 s
32-bit unaligned little endian3.82 s0.27 s
64-bit aligned big endian1.65 s0.41 s
64-bit aligned little endian0.41 s0.41 s
64-bit unaligned big endian17.53 s0.41 s
64-bit unaligned little endian17.52 s0.41 s
16-bit aligned big endian1.37 s0.81 s
16-bit aligned little endian0.83 s0.81 s
16-bit unaligned big endian1.09 s0.83 s
16-bit unaligned little endian1.09 s0.81 s
32-bit aligned big endian0.98 s0.27 s
32-bit aligned little endian0.28 s0.27 s
32-bit unaligned big endian3.82 s0.27 s
32-bit unaligned little endian3.82 s0.27 s
64-bit aligned big endian1.65 s0.41 s
64-bit aligned little endian0.41 s0.41 s
64-bit unaligned big endian17.53 s0.41 s
64-bit unaligned little endian17.52 s0.41 s
Iterations: 1000000000, Intrinsics: no byte swap intrinsics
Test CaseEndian
type		Endian
conversion
function
Iterations: 1000000000, Intrinsics: no byte swap intrinsics
Test CaseEndian
arithmetic		Endian
conversion
function
16-bit aligned big endian1.95 s0.81 s
16-bit aligned little endian0.83 s0.81 s
16-bit unaligned big endian1.19 s0.81 s
16-bit unaligned little endian1.20 s0.81 s
32-bit aligned big endian0.97 s0.28 s
32-bit aligned little endian0.27 s0.28 s
32-bit unaligned big endian4.10 s0.27 s
32-bit unaligned little endian4.10 s0.27 s
64-bit aligned big endian1.64 s0.42 s
64-bit aligned little endian0.41 s0.41 s
64-bit unaligned big endian17.52 s0.42 s
64-bit unaligned little endian17.52 s0.41 s
16-bit aligned big endian1.95 s0.81 s
16-bit aligned little endian0.83 s0.81 s
16-bit unaligned big endian1.19 s0.81 s
16-bit unaligned little endian1.20 s0.81 s
32-bit aligned big endian0.97 s0.28 s
32-bit aligned little endian0.27 s0.28 s
32-bit unaligned big endian4.10 s0.27 s
32-bit unaligned little endian4.10 s0.27 s
64-bit aligned big endian1.64 s0.42 s
64-bit aligned little endian0.41 s0.41 s
64-bit unaligned big endian17.52 s0.42 s
64-bit unaligned little endian17.52 s0.41 s
-

+

Comment: Note that the 32-bit aligned big endian +timings are the same with or without intrinsics turned on. Presumably the +optimizer is recognizing the byte swapping and applying the intrinsics itself.

- - - - - - +
Microsoft Visual C++ version 11.0
Iterations: 1000000000, Intrinsics: cstdlib _byteswap_ushort, etc.
Test CaseEndian
type		Endian
conversion
function
+ + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Microsoft Visual C++ version 11.0
Iterations: 1000000000, Intrinsics: cstdlib _byteswap_ushort, etc.
Test CaseEndian
type		Endian
conversion
function
16-bit aligned big endian0.83 s0.51 s
16-bit aligned little endian0.51 s0.50 s
16-bit unaligned big endian1.37 s0.51 s
16-bit unaligned little endian1.37 s0.50 s
32-bit aligned big endian0.81 s0.50 s
32-bit aligned little endian0.51 s0.51 s
32-bit unaligned big endian2.98 s0.53 s
32-bit unaligned little endian3.00 s0.51 s
64-bit aligned big endian1.33 s0.33 s
64-bit aligned little endian0.34 s0.27 s
64-bit unaligned big endian7.05 s0.33 s
64-bit unaligned little endian7.11 s0.31 s
16-bit aligned big endian0.83 s0.51 s
16-bit aligned little endian0.51 s0.50 s
16-bit unaligned big endian1.37 s0.51 s
16-bit unaligned little endian1.37 s0.50 s
32-bit aligned big endian0.81 s0.50 s
32-bit aligned little endian0.51 s0.51 s
32-bit unaligned big endian2.98 s0.53 s
32-bit unaligned little endian3.00 s0.51 s
64-bit aligned big endian1.33 s0.33 s
64-bit aligned little endian0.34 s0.27 s
64-bit unaligned big endian7.05 s0.33 s
64-bit unaligned little endian7.11 s0.31 s
Iterations: 1000000000, Intrinsics: no byte swap intrinsics
Test CaseEndian
type		Endian
conversion
function
Iterations: 1000000000, Intrinsics: no byte swap intrinsics
Test CaseEndian
type		Endian
conversion
function
16-bit aligned big endian0.83 s0.51 s
16-bit aligned little endian0.51 s0.51 s
16-bit unaligned big endian1.36 s0.51 s
16-bit unaligned little endian1.37 s0.51 s
32-bit aligned big endian3.42 s0.50 s
32-bit aligned little endian0.51 s0.51 s
32-bit unaligned big endian2.93 s0.50 s
32-bit unaligned little endian2.95 s0.50 s
64-bit aligned big endian5.99 s0.33 s
64-bit aligned little endian0.33 s0.33 s
64-bit unaligned big endian7.02 s0.27 s
64-bit unaligned little endian7.02 s0.27 s
16-bit aligned big endian0.83 s0.51 s
16-bit aligned little endian0.51 s0.51 s
16-bit unaligned big endian1.36 s0.51 s
16-bit unaligned little endian1.37 s0.51 s
32-bit aligned big endian3.42 s0.50 s
32-bit aligned little endian0.51 s0.51 s
32-bit unaligned big endian2.93 s0.50 s
32-bit unaligned little endian2.95 s0.50 s
64-bit aligned big endian5.99 s0.33 s
64-bit aligned little endian0.33 s0.33 s
64-bit unaligned big endian7.02 s0.27 s
64-bit unaligned little endian7.02 s0.27 s

Conclusions

-

When program logic dictates many more conversions for the Endian integer +

When program logic dictates many more conversions for the Endian arithmetic approach than the Endian conversion function approach (example 2):

There may be a considerable performance difference. If machine endianness differs from the -desired endianness, the Endian type approach must do the byte reversal many +desired endianness, the Endian arithmetic approach must do the byte reversal many times while the Endian conversion approach only does the reversal once. But if the endianness is the same, there is no conversion with either approach and no conversion code is generated for typical release builds.

Whether or not compiler byte swap intrinsics are explicitly available has little -impact as tested. Byte swap intrinsics are not available on some older -compilers and on some machine architectures, such as pre-486 X86 CPUs.

+impact on GCC but a lot of impact on Visual C++, for the tested compiler +versions. Yet another example of why actual timing tests are needed to +determine if some coding technique has significant impact on performance.

Unaligned types are much slower that aligned types, regardless of endianness considerations. Instead of single instruction register loads and @@ -595,9 +621,9 @@ memory space is a minor secondary use case.

Why bother with binary I/O? Why not just use C++ Standard Library stream inserters and extractors?

+

Data interchange formats often specify binary arithmetic data.

Binary arithmetic data is smaller and therefore I/O is faster and file sizes -are smaller. Transfer between systems is less expensive. Standard interchange -formats often specify binary arithmetic data.

+are smaller. Transfer between systems is less expensive.

Furthermore, binary arithmetic data is of fixed size, and so fixed-size disk records are possible without padding, easing sorting and allowing direct access. Disadvantages, such as the inability to use text utilities on the resulting @@ -614,7 +640,7 @@ CPU's. The Wikipedia article gives more pros and cons.

-

Why is only big, little, and native endianness supported?

+

Why are only big, little, and native endianness supported?

These are the only endian schemes that have any practical value today. PDP-11 and the other middle endian approaches are interesting historical curiosities @@ -625,10 +651,10 @@ but have no relevance to today's C++ developers.

-

The only supported types are four byte float and eight byte +

The only supported types are four-byte float and eight-byte double. Even after endianness has been accounted for, floating point values will not be portable between systems that use different floating -point formats. Systems where the integer endianness differs from floating point +point formats. Systems where integer endianness differs from floating point endianness are not supported.

@@ -648,23 +674,28 @@ and 16, 32, and 64-bit aligned integers.

Release history

Changes since formal review