Endian Library

Abstract

Boost.Endian provides facilities to manipulate the endianness of integers, floating point numbers, and user-defined types.

• The primary use case is binary I/O for portable data exchange with other systems, via either external media or network transmission.
   
• An additional use case is minimizing storage size via sizes and/or alignments not supported by the built-in types.
   
• Three approaches to dealing with endianness are provided. Each approach has a long history of successful use, and each approach has use cases where it is preferred over the other approaches.

Introduction to endianness

Consider the following code:

int16_t i = 0x0102;
FILE * file = fopen("test.bin", "wb");   // binary file!
fwrite(&i, sizeof(int16_t), 1, file);
fclose(file);

On OS X, Linux, or Windows systems with an Intel CPU, a hex dump of the "test.bin" output file produces:

0201

On OS X systems with a PowerPC CPU, or Solaris systems with a SPARC CPU, a hex dump of the "test.bin" output file produces:

0102

What's happening here is that Intel CPUs order the bytes of an integer with the least-significant byte first, while SPARC CPUs place the most-significant byte first. Some CPUs, such as the PowerPC, allow the operating system to choose which ordering applies.

Most-significant-byte-first ordering is traditionally called "big endian" ordering and the least-significant-byte-first is traditionally called "little-endian" ordering. The names are derived from Jonathan Swift's satirical novel Gulliver’s Travels, where rival kingdoms opened their soft-boiled eggs at different ends.

See the Wikipedia's Endianness article for an extensive discussion of endianness.

Programmers can usually ignore endianness, except when reading a core dump on little-endian systems. But programmers will have to deal with endianness when exchanging binary integers and binary floating point values between computer systems with differing endianness, whether by physical file transfer or over a network. And programmers may also want to use the library when minimizing either internal or external data sizes is advantageous.

Introduction to the Boost.Endian library

The Boost.Endian library provides three different approaches to dealing with endianness. All three approaches support integers, floating point types except  long double, and user defined types (UDTs).

Each approach has a long history of successful use, and each approach has use cases where it is preferred to the other approaches.

Endian conversion functions - The application uses the built-in integer and floating point types to hold values, and calls the provided conversion functions to convert byte ordering as needed. Both mutating and non-mutating conversions are supplied, and each comes in unconditional and conditional variants.

Endian buffer types - The application uses the provided endian buffer types to hold values, and explicitly converts to and from the built-in integer and floating point types to perform arithmetic. Buffer lengths of 1 through 8 bytes are supported, rather than just 2, 4, and 8 bytes. The types may be aligned or unaligned.

Endian arithmetic types - The application uses the provided endian arithmetic types, which supply the same operations as the built-in C++ arithmetic types. All conversions are implicit. Arithmetic integer types with lengths of 1 through 8 bytes are supported, rather than just 2, 4, and 8 byte integers. The types may be aligned.

Boost Endian is a header-only library.

Choosing between endian conversion functions, endian buffer types, and endian arithmetic types

Which approach to endianness is best depends on the interaction between approach characteristics and application needs.

The characteristics that differentiate the approaches are the endianness invariants, conversion explicitness, arithmetic operations, sizes available, and alignment requirements.

Endianness invariants

Endian conversion functions use objects of the ordinary C++ arithmetic types like int or unsigned short to hold values. That breaks the implicit invariant that the C++ language rules apply. The usual language rules only apply if the endianness of the object is currently set by the conversion functions to the native endianness for the platform. That can make it very hard to reason about complex logic flow, and result in difficult to find bugs.

Endian buffer and arithmetic types hold values internally as arrays of characters with an invariant that the endianness of the array never changes. That makes these types easy to use and programs easy to maintain.

Conversion explicitness

Endian conversion functions and buffer types never perform implicit conversions. This gives users explicit control of when conversion occurs, and may help avoid unnecessary conversions.

Endian arithmetic types perform conversion implicitly. That makes these types very easy to use, but can result in unnecessary conversions. Failure to hoist conversions out of inner loops can bring a performance penalty.

Arithmetic operations

Endian conversion functions do not supply arithmetic operations, but this is not a concern since this approach uses ordinary C++ arithmetic types to hold values.

Endian buffer types do not supply arithmetic operations. Although this approach avoids unnecessary conversions, it can result in the introduction of additional variables and confuse maintenance programmers.

Endian arithmetic types do supply arithmetic operations. They are very easy to use if lots of arithmetic is involved.

Sizes available

Endianness conversion functions only support 1, 2, 4, and 8 byte integers. That's sufficient for many applications.

Endian buffer and arithmetic types support 1, 2, 3, 4, 5, 6, 7, and 8 byte integers. For an application where memory use or I/O speed is the limiting factor, using sizes tailored to application needs can be very useful.

Alignments available

Endianness conversion functions only support aligned integer and floating-point types. That's sufficient for most applications.

Endian buffer and arithmetic types support both aligned and unaligned integer and floating-point types. Unaligned types are rarely needed, but when needed they are often very useful and workarounds are painful. For example,

Non-portable code like this:

struct S {
  uint16_t a;  // big endian
  uint32_t b;  // big endian
} __attribute__ ((packed));

Can be replaced with portable code like this:

struct S {
  big_uint16_ut a;
  big_uint32_ut b;
};

Built-in support for Intrinsics

Supply compilers, including GCC, Clang, and Visual C++, supply built-in support for byte swapping intrinsics. The library uses these intrinsics when available since they may result in smaller and faster generated code, particularly for release builds.

Defining BOOST_ENDIAN_NO_INTRINSICS will suppress use of the intrinsics. Useful when intrinsic headers such as byteswap.h are not being found on your platform.

The macro BOOST_ENDIAN_INTRINSIC_MSG is defined as either "no byte swap intrinsics" or a string describing the particular set of intrinsics being used.

Performance

Consider this problem:

big_int32_t x;

... read into x from a file ...

x += 100;

... write x to a file ...

  
int32_t x;

... read into x from a file ...

big_to_native_inplace(x);
x += 100;
native_to_big_inplace(x);

... write x to a file ...

There will be no performance difference between the two approaches in release builds, regardless of the native endianness of the machine. That's because optimizing compilers will likely generate exactly the same code for each. That conclusion was confirmed by studying the generated assembly code for GCC and Visual C++. Furthermore, time spent doing I/O will determine the speed of this application.

Now consider a slightly different problem: 

big_int32_t x;

... read into x from a file ...

for (int32_t i = 0; i < 1000000; ++i)
  x += i;

... write x to a file ...

int32_t x;

... read into x from a file ...

big_to_native_inplace(x);

for (int32_t i = 0; i < 1000000; ++i)
  x += i;

native_to_big_inplace(x);

... write x to a file ...

With the Endian arithmetic approach, on little endian platforms an implicit conversion from and then back to big endian is done inside the loop. With the Endian conversion function approach, the user has ensured the conversions are done outside the loop, so the code may run more quickly on little endian platforms.

Timings for Example 2 (conversion functions hoisted out of loop)

These tests were run against release builds on a circa 2012 4-core little endian X64 Intel Core i5-3570K CPU @ 3.40GHz under Windows 7.

Caveat emptor: The Windows CPU timer has very high granularity. Repeated runs of the same tests often yield considerably different results.

See loop_time_test.cpp and Jamfile.v2 for the actual code and build setup. (For GCC 4.7, there are no 16-bit intrinsics, so they are emulated by using 32-bit intrinsics.)

Comment: Note that the 32-bit aligned big endian timings are the same with or without intrinsics turned on. Presumably the optimizer is recognizing the byte swapping and applying the intrinsics itself.

Conclusions

When program logic dictates many more conversions for the Endian arithmetic approach than the Endian conversion function approach (example 2):

There may be a considerable performance difference. If machine endianness differs from the desired endianness, the Endian arithmetic approach must do the byte reversal many times while the Endian conversion approach only does the reversal once. But if the endianness is the same, there is no conversion with either approach and no conversion code is generated for typical release builds.

Whether or not compiler byte swap intrinsics are explicitly available has little impact on GCC but a lot of impact on Visual C++, for the tested compiler versions. Yet another example of why actual timing tests are needed to determine if some coding technique has significant impact on performance.

Unaligned types are much slower that aligned types, regardless of endianness considerations. Instead of single instruction register loads and stores, multiple instructions are required.

Overall FAQ

Is the implementation header only?

Yes.

Does the implementation use compiler intrinsic built-in byte swapping?

Yes, if available. See Intrinsic built-in support.

Why bother with endianness?

Binary data portability is the primary use case.

Does endianness have any uses outside of portable binary file or network I/O formats?

Using the unaligned integer types to save internal or external memory space is a minor secondary use case.

Why bother with binary I/O? Why not just use C++ Standard Library stream inserters and extractors?

Data interchange formats often specify binary arithmetic data.

Binary arithmetic data is smaller and therefore I/O is faster and file sizes are smaller. Transfer between systems is less expensive.

Furthermore, binary arithmetic data is of fixed size, and so fixed-size disk records are possible without padding, easing sorting and allowing direct access. Disadvantages, such as the inability to use text utilities on the resulting files, limit usefulness to applications where the binary I/O advantages are paramount.

Which is better, big-endian or little-endian?

Big-endian tends to be preferred in a networking environment and is a bit more of an industry standard, but little-endian may be preferred for applications that run primarily on x86, x64, and other little-endian CPU's. The Wikipedia article gives more pros and cons.

Why are only big, little, and native endianness supported?

These are the only endian schemes that have any practical value today. PDP-11 and the other middle endian approaches are interesting historical curiosities but have no relevance to today's C++ developers.

What are the limitations of floating point support?

The only supported types are four-byte float and eight-byte double. Even after endianness has been accounted for, floating point values will not be portable between systems that use different floating point formats. Systems where integer endianness differs from floating point endianness are not supported.

What are the limitations of integer support?

Tests have only been performed on machines that use two's complement arithmetic. The Endian conversion functions support 16, 32, and 64-bit aligned integers only. The Endian types support 8, 16, 24, 32, 40, 48, 56, and 64-bit unaligned integers and 16, 32, and 64-bit aligned integers.

Release history

Changes since formal review

• The endian types have been decomposed into endian buffer types and endian arithmetic types, as requested. The arithmetic types derive from the buffer types.

• Headers have been renamed to endian/arithmetic.hpp and endian/conversion.hpp. endian/buffers.hpp has been added. Infrastructure file names were changed accordingly.

• The endian buffer and arithmetic types and endian conversion functions now support 32-bit (float) and 64-bit (double) floating point, as requested.

• The endian types now have stream inserter and extractor templates, as requested.
• Both the endian types and endian conversion functions now support UDTs, as requested.
• The endian type aliases have been renamed, using a naming pattern that is consistent for both integer and floating point, .
• The conversion functions have been much revised, refactored, and otherwise improved based on review comments.
  • Functions have been renamed to clarify their functionality.
  • Both return-by-value and modify-in-place interfaces are provided, as requested.
  • In addition to the named-endianness functions, functions that perform compile-time (via template) and run-time (via function argument) dispatch are now provided, as requested.
• Compiler (Clang, GCC, VisualC++, etc.) intrinsics and built-in functions are used in the implementation where appropriate, as requested.
• For the endian types, the implementation uses the endian conversion functions, and thus the intrinsics, as requested.
• order::native is now a synonym for order::big or order::little according to the endianness of the platform, as requested. This reduces the number of template specializations required.
• endian_reverse() overloads for int8_t and uint8_t have been added for improved generality. (Pierre Talbot)
• Overloads of endian_reverse_inplace() have been replaced with a single endian_reverse_inplace() template. (Pierre Talbot)
• C++11 features such as noexcept are now used, while still supporting C++03 compilers.
• Headers have been reorganized to make them easier to read, with a synopsis at the front and implementation following, as requested.
• Documentation has been revised to address most, but not all, concerns raised during formal review.
• Acknowledgements have been updated.

Acknowledgements

Comments and suggestions were received from Adder, Benaka Moorthi, Christopher Kohlhoff, Cliff Green, Daniel James, Gennaro Proto, Giovanni Piero Deretta, Gordon Woodhull, dizzy, Hartmut Kaiser, Jeff Flinn, John Filo, John Maddock, Kim Barrett, Marsh Ray, Martin Bonner, Mathias Gaunard, Matias Capeletto, Neil Mayhew, Paul Bristow, Pierre Talbot, Phil Endecott, Pyry Jahkola, Rene Rivera, Robert Stewart, Roland Schwarz, Scott McMurray, Sebastian Redl, Tim Blechmann, Tim Moore, tymofey, Tomas Puverle, Vincente Botet, Yuval Ronen and Vitaly Budovski,.

Last revised: 15 December, 2014

© Copyright Beman Dawes, 2011, 2013

Distributed under the Boost Software License, Version 1.0. See www.boost.org/ LICENSE_1_0.txt