Endian Library

Endian Library

Boost Home Endian Home Conversion Functions Arithmetic Types Buffer Types

Contents

Abstract
Introduction to endianness
Introduction to the Boost.Endian library
Choosing approaches
Built-in support for Intrinsics
Performance
Timings
Conclusions
FAQ
Release history
Acknowledgements

Headers

Abstract

Boost.Endian provides facilities to manipulate the endianness of integers, floating point, and user defined data.

The primary use case is binary I/O for portable data exchange with other systems, via either external media or network transmission.
An additional use case is minimizing storage size via sizes and/or alignments not supported by the built-in types.
Three distinct approaches to dealing with endianness are provided. Each approach has a long history of successful use, and each approach has use cases where it is preferred over the other approaches.

Introduction to endianness

Consider the following code:

int16_t i = 0x0102;
FILE * file = fopen("test.bin", "wb");   // binary file!
fwrite(&i, sizeof(int16_t), 1, file);
fclose(file);

On OS X, Linux, or Windows systems with an Intel CPU, a hex dump of the "test.bin" output file produces:

0201

On OS X systems with a PowerPC CPU, or Solaris systems with a SPARC CPU, a hex dump of the "test.bin" output file produces:

0102

What's happening here is that Intel CPUs order the bytes of an integer with the least-significant byte first, while SPARC CPUs place the most-significant byte first. Some CPUs, such as the PowerPC, allow the operating system to choose which ordering applies.

Most-significant-byte-first ordering is traditionally called "big endian" ordering and the least-significant-byte-first is traditionally called "little-endian" ordering. The names are derived from Jonathan Swift's satirical novel Gulliver’s Travels, where rival kingdoms opened their soft-boiled eggs at different ends.

See the Wikipedia's Endianness article for an extensive discussion of endianness.

Programmers can usually ignore endianness, except perhaps for reading a core dump on little-endian systems. But programmers will have to deal with endianness when exchanging binary integers and binary floating point values between computer systems with differing endianness, whether by physical file transfer or over a network. And programmers may want to use the library if minimizing either internal or external data sizes is advantagous.

Introduction to the Boost.Endian library

The Boost.Endian library provides three different approaches to dealing with endianness. All three approaches support integers, floating point types except long double, and user defined types (UDTs).

Each approach has a long history of successful use, and each approach has use cases where it is preferred to the other approaches.

Endian conversion functions - The application uses the built-in integer and floating point types, and calls the provided conversion functions to convert byte ordering as needed. Both mutating and non-mutating conversions are supplied, and each comes in unconditional and conditional variants.

Endian buffer types - The application uses the provided endian buffer types which mimic the built-in integer types. For example, big_buf32_t or little_floatbuf64_t. Buffer types with lengths of 1 through 8 bytes are supported, rather than just 2, 4, and 8 bytes. The types may be aligned (name suffix _t) or unaligned (name suffix _ut).

Endian arithmetic types - The application uses the provided endian arithmetic types which mimic the built-in arithmetic types. For example, big_int32_t or little_float64_t. Arithmetic integer types with lengths of 1 through 8 bytes are supported, rather than just 2, 4, and 8 byte integers. The types may be aligned (name suffix _t) or unaligned (name suffix _ut).

Boost Endian is a header-only library.

Choosing between conversion functions, buffers types, and arithmetic types

Which endianness approach is best depends on application needs.

Needs that favor one approach over the other

Endian types may be better for these needs Endian conversion functions may be better for these needs

Needs that favor one approach over the other
Endian types may be better for these needs	Endian conversion functions may be better for these needs
A need to save overall time when I/O time is more important than numeric variable CPU time. A need to simplify program logic and eliminate logic errors. Since the endian types mimic the built-in types, there is no need to reason about the current endianness of variables and that can simplify program logic and eliminate logic errors. A need to use unusual integer sizes (i.e. 3, 5, 6, or 7 bytes) to reduce internal and external space usage and save I/O time. A need to portably use unaligned variables or eliminate padding bytes in structures, either to save I/O time by reducing internal and external space usage, or to conform to a data layout over which you have no control. For example, the structure below relies on a compiler extension, so is non-portable, and operations on S::b will fail on hardware platforms that require 32-bit alignment. `struct S { uint16_t a; uint32_t b; } __attribute__ ((packed));` These problems are eliminated by defining S like this: `struct S { big_uint16_t a; big_uint32_t b; };` Programmer preference.	A need to save overall time when numeric variable CPU use time is more important that I/O time. A need to pass structures to third-party libraries expecting a specific structure format. A need to leverage knowledge of developers who have been using C byte swapping functions for years. Programmer preference.

A need to save overall time when I/O time is more important than numeric variable CPU time.
A need to simplify program logic and eliminate logic errors. Since the endian types mimic the built-in types, there is no need to reason about the current endianness of variables and that can simplify program logic and eliminate logic errors.
A need to use unusual integer sizes (i.e. 3, 5, 6, or 7 bytes) to reduce internal and external space usage and save I/O time.
A need to portably use unaligned variables or eliminate padding bytes in structures, either to save I/O time by reducing internal and external space usage, or to conform to a data layout over which you have no control. For example, the structure below relies on a compiler extension, so is non-portable, and operations on S::b will fail on hardware platforms that require 32-bit alignment.

struct S { uint16_t a; uint32_t b; } __attribute__ ((packed));

These problems are eliminated by defining S like this:

struct S { big_uint16_t a; big_uint32_t b; };
Programmer preference.

A need to save overall time when numeric variable CPU use time is more important that I/O time.
A need to pass structures to third-party libraries expecting a specific structure format.
A need to leverage knowledge of developers who have been using C byte swapping functions for years.
Programmer preference.

Warning: Endian conversion functions are dangerous unless the current endianness of a variable is always known. For example, if an exception is thrown and there is no way in a catch block to know a variable's current endianness, then any use of that variable including I/O is inherently unsafe. This danger is not limited to exceptions; all uses of a variable whose endianness is unknown are unsafe, period.

Built-in support for Intrinsics

Recent compilers, including GCC, Clang, and Microsoft, supply built-in support for byte swapping intrinsics. Such support is automatically detected and used since it may in smaller and faster generated code, particularly for release builds.

Defining BOOST_ENDIAN_NO_INTRINSICS will suppress use of the intrinsics. Please try defining it if you get compiler errors, such as header byteswap.h not being found.

The macro BOOST_ENDIAN_INTRINSIC_MSG is defined as either "no byte swap intrinsics" or a string describing the particular set of intrinsics being used.

Performance

Consider this problem:

Example 1
*Add 100 to a big endian value in a file, then write the result to a file*
Endian type approach	Endian conversion approach
big_int32_t x; ... read into x from a file ... x += 100; ... write x to a file ...	int32_t x; ... read into x from a file ... big_endian(x); x += 100; big_endian(x); ... write x to a file ...

There will be no performance difference between the two approaches in release builds, regardless of the native endianness of the machine. That's because optimizing compilers will likely generate exactly the same code for each. That conclusion was confirmed by studying the generated assembly code for GCC and Visual C++.

Now consider a slightly different problem:

*Example 2*
Add a million values to a big endian value in a file, then write the result to a file
Endian type approach	Endian conversion approach
big_int32_t x; ... read into x from a file ... for (int32_t i = 0; i < 1000000; ++i) x += i; ... write x to a file ...	int32_t x; ... read into x from a file ... big_endian(x); for (int32_t i = 0; i < 1000000; ++i) x += i; big_endian(x); ... write x to a file ...

With the Endian type approach, an implicit conversion from and then back to big endian is done inside the loop. With the Endian conversion function approach, the conversions are explicit, so only need to be done once, before and after the loop.

Timings for Example 2 (conversion functions hoisted out of loop)

These tests were run against release builds on a circa 2012 4-core little endian X64 Intel Core i5-3570K CPU @ 3.40GHz under Windows 7.

Caveat emptor: The Windows CPU timer has very high granularity. Repeated runs of the same tests often yield considerably different results.

See loop_time_test.cpp and Jamfile.v2 for the actual code and build setup. (For GCC 4.7, there are no 16-bit intrinsics, so they are emulated by using 32-bit intrinsics.)

GNU C++ version 4.7.0
Iterations: 1000000000, Intrinsics: __builtin_bswap16, etc.
Test Case	Endian type	Endian conversion function
16-bit aligned big endian	1.37 s	0.81 s
16-bit aligned little endian	0.83 s	0.81 s
16-bit unaligned big endian	1.09 s	0.83 s
16-bit unaligned little endian	1.09 s	0.81 s
32-bit aligned big endian	0.98 s	0.27 s
32-bit aligned little endian	0.28 s	0.27 s
32-bit unaligned big endian	3.82 s	0.27 s
32-bit unaligned little endian	3.82 s	0.27 s
64-bit aligned big endian	1.65 s	0.41 s
64-bit aligned little endian	0.41 s	0.41 s
64-bit unaligned big endian	17.53 s	0.41 s
64-bit unaligned little endian	17.52 s	0.41 s
Iterations: 1000000000, Intrinsics: no byte swap intrinsics
Test Case	Endian type	Endian conversion function
16-bit aligned big endian	1.95 s	0.81 s
16-bit aligned little endian	0.83 s	0.81 s
16-bit unaligned big endian	1.19 s	0.81 s
16-bit unaligned little endian	1.20 s	0.81 s
32-bit aligned big endian	0.97 s	0.28 s
32-bit aligned little endian	0.27 s	0.28 s
32-bit unaligned big endian	4.10 s	0.27 s
32-bit unaligned little endian	4.10 s	0.27 s
64-bit aligned big endian	1.64 s	0.42 s
64-bit aligned little endian	0.41 s	0.41 s
64-bit unaligned big endian	17.52 s	0.42 s
64-bit unaligned little endian	17.52 s	0.41 s

Microsoft Visual C++ version 11.0
Iterations: 1000000000, Intrinsics: cstdlib _byteswap_ushort, etc.
Test Case	Endian type	Endian conversion function
16-bit aligned big endian	0.83 s	0.51 s
16-bit aligned little endian	0.51 s	0.50 s
16-bit unaligned big endian	1.37 s	0.51 s
16-bit unaligned little endian	1.37 s	0.50 s
32-bit aligned big endian	0.81 s	0.50 s
32-bit aligned little endian	0.51 s	0.51 s
32-bit unaligned big endian	2.98 s	0.53 s
32-bit unaligned little endian	3.00 s	0.51 s
64-bit aligned big endian	1.33 s	0.33 s
64-bit aligned little endian	0.34 s	0.27 s
64-bit unaligned big endian	7.05 s	0.33 s
64-bit unaligned little endian	7.11 s	0.31 s
Iterations: 1000000000, Intrinsics: no byte swap intrinsics
Test Case	Endian type	Endian conversion function
16-bit aligned big endian	0.83 s	0.51 s
16-bit aligned little endian	0.51 s	0.51 s
16-bit unaligned big endian	1.36 s	0.51 s
16-bit unaligned little endian	1.37 s	0.51 s
32-bit aligned big endian	3.42 s	0.50 s
32-bit aligned little endian	0.51 s	0.51 s
32-bit unaligned big endian	2.93 s	0.50 s
32-bit unaligned little endian	2.95 s	0.50 s
64-bit aligned big endian	5.99 s	0.33 s
64-bit aligned little endian	0.33 s	0.33 s
64-bit unaligned big endian	7.02 s	0.27 s
64-bit unaligned little endian	7.02 s	0.27 s

Conclusions

When program logic dictates many more conversions for the Endian integer approach than the Endian conversion function approach (example 2):

There may be a considerable performance difference. If machine endianness differs from the desired endianness, the Endian type approach must do the byte reversal many times while the Endian conversion approach only does the reversal once. But if the endianness is the same, there is no conversion with either approach and no conversion code is generated for typical release builds.

Whether or not compiler byte swap intrinsics are explicitly available has little impact as tested. Byte swap intrinsics are not available on some older compilers and on some machine architectures, such as pre-486 X86 CPUs.

Unaligned types are much slower that aligned types, regardless of endianness considerations. Instead of single instruction register loads and stores, multiple instructions are required.

Overall FAQ

Is the implementation header only?

Yes.

Does the implementation use compiler intrinsic built-in byte swapping?

Yes, if available. See Intrinsic built-in support.

Why bother with endianness?

Binary data portability is the primary use case.

Does endianness have any uses outside of portable binary file or network I/O formats?

Using the unaligned integer types to save internal or external memory space is a minor secondary use case.

Why bother with binary I/O? Why not just use C++ Standard Library stream inserters and extractors?

Binary arithmetic data is smaller and therefore I/O is faster and file sizes are smaller. Transfer between systems is less expensive. Standard interchange formats often specify binary arithmetic data.

Furthermore, binary arithmetic data is of fixed size, and so fixed-size disk records are possible without padding, easing sorting and allowing direct access. Disadvantages, such as the inability to use text utilities on the resulting files, limit usefulness to applications where the binary I/O advantages are paramount.

Which is better, big-endian or little-endian?

Big-endian tends to be preferred in a networking environment and is a bit more of an industry standard, but little-endian may be preferred for applications that run primarily on x86, x64, and other little-endian CPU's. The Wikipedia article gives more pros and cons.

Why is only big, little, and native endianness supported?

These are the only endian schemes that have any practical value today. PDP-11 and the other middle endian approaches are interesting historical curiosities but have no relevance to today's C++ developers.

What are the limitations of floating point support?

The only supported types are four byte float and eight byte double. Even after endianness has been accounted for, floating point values will not be portable between systems that use different floating point formats. Systems where the integer endianness differs from floating point endianness are not supported.

What are the limitations of integer support?

Tests have only been performed on machines that use two's complement arithmetic. The Endian conversion functions support 16, 32, and 64-bit aligned integers only. The Endian types support 8, 16, 24, 32, 40, 48, 56, and 64-bit unaligned integers and 16, 32, and 64-bit aligned integers.

Release history

Changes since formal review

Headers have been renamed to endian/types.hpp and endian/conversion.hpp. Infrastructure file names were changed accordingly.
The endian types and endian conversion functions now support 32-bit (float) and 64-bit (double) floating point, as requested.
The endian types now have stream inserter and extractor templates, as requested.
Both the endian types and endian conversion functions now support UDTs, as requested.
The endian type aliases have been renamed, using a naming pattern that is consistent for both integer and floating point.
The conversion functions have been much revised, refactored, and otherwise improved based on review comments.
- Functions have been renamed to clarify their functionality.
- Both return-by-value and modify-in-place interfaces are provided, as requested.
- Synonyms for the BSD byte swapping function names popularized by OS X and Linux are provided, so that that developers already used to these names can continue to use them if they wish.
- In addition to the named-endianness functions, functions that perform compile-time (via template) and run-time (via function argument) dispatch are now provided, as requested.
Compiler (Clang, GCC, VisualC++, etc.) intrinsics and built-in functions are used in the implementation where appropriate, as requested.
For the endian types, the implementation uses the endian conversion functions, and thus the intrinsics, as requested.
order::native is now a synonym for order::big or order::little according to the endianness of the platform, as requested. This reduces the number of template specializations required.
reverse_value() overloads for int8_t and uint8_t have been added for improved generality. (Pierre Talbot)
Overloads of reverse() have been replaced with a single reverse() template. (Pierre Talbot)
C++11 features such as noexcept are now used, while still supporting C++03 compilers.
Headers have been reorganized to make them easier to read, with a synopsis at the front and implementation following, as requested.
Documentation has been revised to address most, but not all, concerns raised during formal review.
Acknowledgements have been updated.

Acknowledgements

Comments and suggestions were received from Adder, Benaka Moorthi, Christopher Kohlhoff, Cliff Green, Daniel James, Gennaro Proto, Giovanni Piero Deretta, Gordon Woodhull, dizzy, Hartmut Kaiser, Jeff Flinn, John Filo, John Maddock, Kim Barrett, Marsh Ray, Martin Bonner, Mathias Gaunard, Matias Capeletto, Neil Mayhew, Paul Bristow, Pierre Talbot, Phil Endecott, Pyry Jahkola, Rene Rivera, Robert Stewart, Roland Schwarz, Scott McMurray, Sebastian Redl, Tim Blechmann, Tim Moore, tymofey, Tomas Puverle, Vincente Botet, Yuval Ronen and Vitaly Budovski,.

Last revised: 19 November, 2014

Distributed under the Boost Software License, Version 1.0. See www.boost.org/ LICENSE_1_0.txt