Endian Arithmetic Types

Endian Arithmetic Types

Boost Home Endian Home Conversion Functions Arithmetic Types Buffer Types

Contents

Introduction
Example
Limitations
Feature set
Enums and typedefs
Class template endian
    Synopsis
    Members
    Stream inserter
    Stream extractor
FAQ
Design
Experience
Motivating use cases
C++11
Compilation
Acknowledgements

Headers

Introduction

Header boost/endian/arithmetic.hpp provides integer and floating point binary types with control over byte order, value type, size, and alignment. Typedefs provide easy-to-use names for common configurations.

These types provide portable byte-holders for integer data, independent of particular computer architectures. Use cases almost always involve I/O, either via files or network connections. Although data portability is the primary motivation, these integer byte-holders may also be used to reduce memory use, file size, or network activity since they provide binary integer sizes not otherwise available.

Such integer byte-holder types are traditionally called endian types. See the Wikipedia for a full exploration of endianness, including definitions of big endian and little endian.

Boost endian integers provide the same full set of C++ assignment, arithmetic, and relational operators as C++ standard integral types, with the standard semantics.

Unary arithmetic operators are +, -, ~, !, prefix and postfix -- and ++. Binary arithmetic operators are +, +=, -, -=, *, *=, /, /=, %/ %=, &, &=, |, |=, ^, ^=, <<, <<=, >>, >>=. Binary relational operators are ==, !=, <, <=, >, >=.

Automatic implicit conversion to the underlying value type is provided. A conversion constructor from the underlying value type is provided.

Example

The endian_example.cpp program writes a binary file containing four byte big-endian and little-endian integers:

#include <iostream>
#include <cstdio>
#include <boost/endian/arithmetic.hpp>
#include <boost/static_assert.hpp>

using namespace boost::endian;

namespace 
{
  //  This is an extract from a very widely used GIS file format. It seems odd
  //  to mix big and little endians in the same file - but this is a real-world
  //  format and users wishing to write low level code manipulating these files
  //  must deal with the mixed endianness.

  struct header
  {
    big_int32_t     file_code;
    big_int32_t     file_length;
    little_int32_t  version;
    little_int32_t  shape_type;
  };

  const char* filename = "test.dat";
}

int main(int, char* [])
{
  BOOST_STATIC_ASSERT(sizeof(header) == 16U);  // reality check
  
  header h;

  h.file_code   = 0x01020304;
  h.file_length = sizeof(header);
  h.version     = 1;
  h.shape_type  = 0x01020304;

  //  Low-level I/O such as POSIX read/write or <cstdio> fread/fwrite is sometimes
  //  used for binary file operations when ultimate efficiency is important. Such
  //  I/O is often performed in some C++ wrapper class, but to drive home the
  //  point that endian integers are often used in fairly low-level code that does
  //  bulk I/O operations, <cstdio> fopen/fwrite is used for I/O in this example.

  std::FILE* fi = std::fopen(filename, "wb");  // MUST BE BINARY
  
  if (!fi)
  {
    std::cout << "could not open " << filename << '\n';
    return 1;
  }

  if (std::fwrite(&h, sizeof(header), 1, fi)!= 1)
  {
    std::cout << "write failure for " << filename << '\n';
    return 1;
  }

  std::fclose(fi);

  std::cout << "created file " << filename << '\n';

  return 0;
}

After compiling and executing endian_example.cpp, a hex dump of test.dat shows:

01020304 00000010 01000000 04030201

Notice that the first two 32-bit integers are big endian while the second two are little endian, even though the machine this was compiled and run on was little endian.

Limitations

Requires <climits> CHAR_BIT == 8. If CHAR_BIT is some other value, compilation will result in an #error. This restriction is in place because the design, implementation, testing, and documentation has only considered issues related to 8-bit bytes, and there have been no real-world use cases presented for other sizes.

In C++03, endian does not meet the requirements for POD types because it has constructors, private data members, and a base class. This means that common use cases are relying on unspecified behavior in that the C++ Standard does not guarantee memory layout for non-POD types. This has not been a problem in practice since all known C++ compilers do layout memory as if endian were a POD type. In C++11, it is possible to specify the default constructor as trivial, and private data members and base classes will no longer disqualify a type from being a POD. Thus under C++11, endian will no longer be relying on unspecified behavior.

Feature set

Big endian| little endian | native endian byte ordering.
Signed | unsigned
Unaligned | aligned
Integer | floating point
1-8 byte (unaligned) | 2, 4, 8 byte (aligned)
Choice of value type

Enums and typedefs

Two scoped enums are provided:

enum class order {big, little, native};

enum class align {no, yes};

One class template is provided:

template <order Order, typename T, std::size_t n_bits, align A = align::no>
  class endian_arithmetic;

Typedefs, such as big_int32_t, provide convenient naming conventions for common use cases:

Name Endianness Alignment Sign Sizes in bits (n)

big_intn_t big yes signed 16,32,64

big_uintn_t big yes unsigned 16,32,64

big_floatn_t big yes signed 32,64

big_intn_ut big no signed 8,16,24,32,40,48,56,64

big_uintn_ut big no unsigned 8,16,24,32,40,48,56,64

big_floatn_ut big no signed 32,64

little_intn_t little yes signed 16,32,64

little_uintn_t little yes unsigned 16,32,64

little_floatn_t little yes signed 32,64

little_intn_ut little no signed 8,16,24,32,40,48,56,64

little_uintn_ut little no unsigned 8,16,24,32,40,48,56,64

little_floatn_ut little no signed 32,64

native_floatn_t native yes signed 32,64

native_intn_ut native no signed 8,16,24,32,40,48,56,64

native_uintn_ut native no unsigned 8,16,24,32,40,48,56,64

native_floatn_ut native no signed 32,64

The unaligned types do not cause compilers to insert padding bytes in classes and structs. This is an important characteristic that can be exploited to minimize wasted space in memory, files, and network transmissions.

Warning: Code that uses aligned types is possibly non-portable because alignment requirements vary between hardware architectures and because alignment may be affected by compiler switches or pragmas. For example, alignment of an 64-bit integer may be to a 32-bit boundary on a 32-bit machine. Furthermore, aligned types are only available on architectures with 16, 32, and 64-bit integer types.

Note: One-byte types have identical functionality. They are provided to improve code readability and searchability.

Class template `endian_arithmetic`

An endian is an integer byte-holder with user-specified endianness, value type, size, and alignment. The usual operations on integers are supplied.

Synopsis

namespace boost
{
  namespace endian
  {
    //  C++11 features emulated if not available
   
    enum class order
    {
      big,                             // big-endian
      little,                          // little-endian
      native = implementation-defined  // same as order::big or order::little
    };

    enum class align {no, yes};            

    template <order Order, class T, std::size_t n_bits, align A = align::no>
    class endian_arithmetic
      : public endian_buffer<Order, T, n_bits, A>
    {
    public:
      typedef T value_type;

      // if BOOST_ENDIAN_FORCE_PODNESS is defined && C++11 POD's are not
      // available then these two constructors will not be present
      endian_arithmetic() noexcept = default;
      endian_arithmetic(T v) noexcept;

      endian_arithmetic& operator=(T v) noexcept;
      operator value_type() const noexcept;
      value_type value() const noexcept;  // exposition only; see endian_buffer
      const char* data() const noexcept;  // exposition only; see endian_buffer

      // arithmetic operations
      //   note that additional operations are provided by the value_type 
      value_type operator+(const endian& x) noexcept;
      endian& operator+=(endian& x, value_type y) noexcept;
      endian& operator-=(endian& x, value_type y) noexcept;
      endian& operator*=(endian& x, value_type y) noexcept;
      endian& operator/=(endian& x, value_type y) noexcept;
      endian& operator%=(endian& x, value_type y) noexcept;
      endian& operator&=(endian& x, value_type y) noexcept;
      endian& operator|=(endian& x, value_type y) noexcept;
      endian& operator^=(endian& x, value_type y) noexcept;
      endian& operator<<=(endian& x, value_type y) noexcept;
      endian& operator>>=(endian& x, value_type y noexcept;
      value_type operator<<(const endian& x, value_type y) noexcept;
      value_type operator>>(const endian& x, value_type y) noexcept;
      endian& operator++(endian& x) noexcept;
      endian& operator--(endian& x) noexcept;
      endian operator++(endian& x, int) noexcept;
      endian operator--(endian& x, int) noexcept;

      // Stream inserter
      template <class charT, class traits>
      friend std::basic_ostream<charT, traits>&
        operator<<(std::basic_ostream<charT, traits>& os, const T& x);

      // Stream extractor 
      template <class charT, class traits>
      friend std::basic_istream<charT, traits>&
        operator>>(std::basic_istream<charT, traits>& is, T& x);
    };

    // typedefs  

    // aligned big endian floating point types
    typedef endian<order::big, float, 32, align::yes>       big_float32_t;
    typedef endian<order::big, double, 64, align::yes>      big_float64_t;

    // aligned little endian floating point types
    typedef endian<order::little, float, 32, align::yes>    little_float32_t;
    typedef endian<order::little, double, 64, align::yes>   little_float64_t;

    // unaligned big endian floating point types
    typedef endian<order::big, float, 32, align::no>        big_float32un_t;
    typedef endian<order::big, double, 64, align::no>       big_float64un_t;
  
    // unaligned little endian floating point types
    typedef endian<order::little, float, 32, align::no>     little_float32un_t;
    typedef endian<order::little, double, 64, align::no>    little_float64un_t;
  
    // aligned big endian signed integer types
    typedef endian<order::big, int16_t, 16, align::yes>     big_int16_t;
    typedef endian<order::big, int32_t, 32, align::yes>     big_int32_t;
    typedef endian<order::big, int64_t, 64, align::yes>     big_int64_t;

    // aligned big endian unsigned integer types
    typedef endian<order::big, uint16_t, 16, align::yes>    big_uint16_t;
    typedef endian<order::big, uint32_t, 32, align::yes>    big_uint32_t;
    typedef endian<order::big, uint64_t, 64, align::yes>    big_uint64_t;

    // aligned little endian signed integer types
    typedef endian<order::little, int16_t, 16, align::yes>  little_int16_t;
    typedef endian<order::little, int32_t, 32, align::yes>  little_int32_t;
    typedef endian<order::little, int64_t, 64, align::yes>  little_int64_t;

    // aligned little endian unsigned integer types
    typedef endian<order::little, uint16_t, 16, align::yes> little_uint16_t;
    typedef endian<order::little, uint32_t, 32, align::yes> little_uint32_t;
    typedef endian<order::little, uint64_t, 64, align::yes> little_uint64_t;

    // aligned native endian typedefs are not provided because
    // <cstdint> types are superior for that use case
  
    // unaligned big endian signed integer types
    typedef endian<order::big, int_least8_t, 8>        big_int8_ut;
    typedef endian<order::big, int_least16_t, 16>      big_int16_ut;
    typedef endian<order::big, int_least32_t, 24>      big_int24_ut;
    typedef endian<order::big, int_least32_t, 32>      big_int32_ut;
    typedef endian<order::big, int_least64_t, 40>      big_int40_ut;
    typedef endian<order::big, int_least64_t, 48>      big_int48_ut;
    typedef endian<order::big, int_least64_t, 56>      big_int56_ut;
    typedef endian<order::big, int_least64_t, 64>      big_int64_ut;
  
    // unaligned big endian unsigned integer types
    typedef endian<order::big, uint_least8_t, 8>       big_uint8_ut;
    typedef endian<order::big, uint_least16_t, 16>     big_uint16_ut;
    typedef endian<order::big, uint_least32_t, 24>     big_uint24_ut;
    typedef endian<order::big, uint_least32_t, 32>     big_uint32_ut;
    typedef endian<order::big, uint_least64_t, 40>     big_uint40_ut;
    typedef endian<order::big, uint_least64_t, 48>     big_uint48_ut;
    typedef endian<order::big, uint_least64_t, 56>     big_uint56_ut;
    typedef endian<order::big, uint_least64_t, 64>     big_uint64_ut;
  
    // unaligned little endian signed integer types
    typedef endian<order::little, int_least8_t, 8>     little_int8_ut;
    typedef endian<order::little, int_least16_t, 16>   little_int16_ut;
    typedef endian<order::little, int_least32_t, 24>   little_int24_ut;
    typedef endian<order::little, int_least32_t, 32>   little_int32_ut;
    typedef endian<order::little, int_least64_t, 40>   little_int40_ut;
    typedef endian<order::little, int_least64_t, 48>   little_int48_ut;
    typedef endian<order::little, int_least64_t, 56>   little_int56_ut;
    typedef endian<order::little, int_least64_t, 64>   little_int64_ut;
  
    // unaligned little endian unsigned integer types
    typedef endian<order::little, uint_least8_t, 8>    little_uint8_ut;
    typedef endian<order::little, uint_least16_t, 16>  little_uint16_ut;
    typedef endian<order::little, uint_least32_t, 24>  little_uint24_ut;
    typedef endian<order::little, uint_least32_t, 32>  little_uint32_ut;
    typedef endian<order::little, uint_least64_t, 40>  little_uint40_ut;
    typedef endian<order::little, uint_least64_t, 48>  little_uint48_ut;
    typedef endian<order::little, uint_least64_t, 56>  little_uint56_ut;
    typedef endian<order::little, uint_least64_t, 64>  little_uint64_ut;
  
  // unaligned native endian signed integer types
    typedef implementation-defined_int8_t   native_int8_ut;
    typedef implementation-defined_int16_t  native_int16_ut;
    typedef implementation-defined_int24_t  native_int24_ut;
    typedef implementation-defined_int32_t  native_int32_ut;
    typedef implementation-defined_int40_t  native_int40_ut;
    typedef implementation-defined_int48_t  native_int48_ut;
    typedef implementation-defined_int56_t  native_int56_ut;
    typedef implementation-defined_int64_t  native_int64_ut;

  // unaligned native endian unsigned integer types
    typedef implementation-defined_uint8_t   native_uint8_ut;
    typedef implementation-defined_uint16_t  native_uint16_ut;
    typedef implementation-defined_uint24_t  native_uint24_ut;
    typedef implementation-defined_uint32_t  native_uint32_ut;
    typedef implementation-defined_uint40_t  native_uint40_ut;
    typedef implementation-defined_uint48_t  native_uint48_ut;
    typedef implementation-defined_uint56_t  native_uint56_ut;
    typedef implementation-defined_uint64_t  native_uint64_ut;

  } // namespace endian
} // namespace boost

The implementation-defined text above is either big or little according to the endianness of the platform.

Members

endian() = default;  // C++03: endian(){}

Effects: Constructs an object of type endian<E, T, n_bits, A>.

endian(T v);

Effects: Constructs an object of type endian<E, T, n_bits, A>.

Postcondition: x == v, where x is the constructed object.

endian& operator=(T v);

Postcondition: x == v, where x is the constructed object.

Returns: *this.

operator T() const;

Returns: The current value stored in *this, converted to value_type.

const char* data() const;

Returns: A pointer to the first byte of the endian binary value stored in *this.

Other operators

Other operators on endian objects are forwarded to the equivalent operator on value_type.

Stream inserter

template <class charT, class traits>
friend std::basic_ostream<charT, traits>&
  operator<<(std::basic_ostream<charT, traits>& os, const T& x);

Returns: os << +x.

Stream extractor

template <class charT, class traits>
friend std::basic_istream<charT, traits>&
  operator>>(std::basic_istream<charT, traits>& is, T& x);

Effects: As if:
T i;
if (is >> i)
  x = i;
Returns: is.

FAQ

See the Endian home page FAQ for a library-wide FAQ.

Why not just use Boost.Serialization? Serialization involves a conversion for every object involved in I/O. Endian integers require no conversion or copying. They are already in the desired format for binary I/O. Thus they can be read or written in bulk.

Are endian types POD's? Yes for C++11. No for C++03, although several macros are available to force PODness in all cases.

What are the implications of endian integer types not being POD's with C++03 compilers? They can't be used in unions. Also, compilers aren't required to align or lay out storage in portable ways, although this potential problem hasn't prevented use of Boost.Endian with real compilers.

What good is native endianness? It provides alignment and size guarantees not available from the built-in types. It eases generic programming.

Why bother with the aligned endian types? Aligned integer operations may be faster (as much as 10 to 20 times faster) if the endianness and alignment of the type matches the endianness and alignment requirements of the machine. The code, however, is likely to be somewhat less portable than with the unaligned types.

Why provide the arithmetic operations? Providing a full set of operations reduces program clutter and makes code both easier to write and to read. Consider incrementing a variable in a record. It is very convenient to write:

    ++record.foo;

Rather than:

    int temp(record.foo);
    ++temp;
    record.foo = temp;

Design considerations for Boost.Endian types

Must be suitable for I/O - in other words, must be memcpyable.
Must provide exactly the size and internal byte ordering specified.
Must work correctly when the internal integer representation has more bits that the sum of the bits in the external byte representation. Sign extension must work correctly when the internal integer representation type has more bits than the sum of the bits in the external bytes. For example, using a 64-bit integer internally to represent 40-bit (5 byte) numbers must work for both positive and negative values.
Must work correctly (including using the same defined external representation) regardless of whether a compiler treats char as signed or unsigned.
Unaligned types must not cause compilers to insert padding bytes.
The implementation should supply optimizations with great care. Experience has shown that optimizations of endian integers often become pessimizations when changing machines or compilers. Pessimizations can also happen when changing compiler switches, compiler versions, or CPU models of the same architecture.

Experience

Classes with similar functionality have been independently developed by several Boost programmers and used very successful in high-value, high-use applications for many years. These independently developed endian libraries often evolved from C libraries that were also widely used. Endian types have proven widely useful across a wide range of computer architectures and applications.

Motivating use cases

Neil Mayhew writes: "I can also provide a meaningful use-case for this library: reading TrueType font files from disk and processing the contents. The data format has fixed endianness (big) and has unaligned values in various places. Using Boost.Endian simplifies and cleans the code wonderfully."

C++11

The availability of the C++11 Defaulted Functions feature is detected automatically, and will be used if present to ensure that objects of class endian are trivial, and thus POD's.

Compilation

Boost.Endian is implemented entirely within headers, with no need to link to any Boost object libraries.

Several macros allow user control over features:

BOOST_ENDIAN_NO_CTORS causes class endian to have no constructors. The intended use is for compiling user code that must be portable between compilers regardless of C++11 Defaulted Functions support. Use of constructors will always fail,
BOOST_ENDIAN_FORCE_PODNESS causes BOOST_ENDIAN_NO_CTORS to be defined if the compiler does not support C++11 Defaulted Functions. This is ensures that , and so can be used in unions. In C++11, class endian objects are POD's even though they have constructors.

Acknowledgements

Original design developed by Darin Adler based on classes developed by Mark Borgerding. Four original class templates combined into a single endian class template by Beman Dawes, who put the library together, provided documentation, added the typedefs, and also added the unrolled_byte_loops sign partial specialization to correctly extend the sign when cover integer size differs from endian representation size. Vicente Botet and other reviewers suggested supporting floating point types.

Last revised: 05 December, 2014

Distributed under the Boost Software License, Version 1.0. See www.boost.org/ LICENSE_1_0.txt

*Name*	*Endianness*	*Alignment*	*Sign*	*Sizes in bits (n)*
`big_int`n`_t`	`big`	`yes`	signed	16,32,64
`big_uint`n`_t`	`big`	`yes`	unsigned	16,32,64
`big_float`n`_t`	`big`	`yes`	signed	32,64
`big_int`n`_ut`	`big`	`no`	signed	8,16,24,32,40,48,56,64
`big_uint`n`_ut`	`big`	`no`	unsigned	8,16,24,32,40,48,56,64
`big_float`n`_ut`	`big`	`no`	signed	32,64
`little_int`n`_t`	`little`	`yes`	signed	16,32,64
`little_uint`n`_t`	`little`	`yes`	unsigned	16,32,64
`little_float`n`_t`	`little`	`yes`	signed	32,64
`little_int`n`_ut`	`little`	`no`	signed	8,16,24,32,40,48,56,64
`little_uint`n`_ut`	`little`	`no`	unsigned	8,16,24,32,40,48,56,64
`little_float`n`_ut`	`little`	`no`	signed	32,64
`native_float`n`_t`	`native`	`yes`	signed	32,64
`native_int`n`_ut`	`native`	`no`	signed	8,16,24,32,40,48,56,64
`native_uint`n`_ut`	`native`	`no`	unsigned	8,16,24,32,40,48,56,64
`native_float`n`_ut`	`native`	`no`	signed	32,64

Enums and typedefs

Class template endian_arithmetic

Other operators

Design considerations for Boost.Endian types

Class template `endian_arithmetic`