Merge pull request #230 from k3DW/feature/226

Optimize `emplace()` for arguments of the form `k, v`

doc/unordered/changes.adoc

@@ -6,6 +6,11 @@
 :github-pr-url: https://github.com/boostorg/unordered/pull
 :cpp: C++
 
+== Release 1.85.0
+
+* Optimized `emplace()` for a `value_type` or `init_type` (if applicable) argument to bypass creating an intermediate object. The argument is already the same type as the would-be intermediate object.
+* Optimized `emplace()` for `k,v` arguments on map containers to delay constructing the object until it is certain that an element should be inserted. This optimization happens when the map's `key_type` is move constructible or when the `k` argument is a `key_type`.
+
 == Release 1.84.0 - Major update
 
 * Added `boost::concurrent_flat_set`.

doc/unordered/concurrent_flat_map.adoc

@@ -908,7 +908,9 @@ Inserts an object, constructed with the arguments `args`, in the table if and on
 Requires:;; `value_type` is constructible from `args`.
 Returns:;; `true` if an insert took place.
 Concurrency:;; Blocking on rehashing of `*this`.
-Notes:;; Invalidates pointers and references to elements if a rehashing is issued.
+Notes:;; Invalidates pointers and references to elements if a rehashing is issued. +
++
+If `args...` is of the form `k,v`, it delays constructing the whole object until it is certain that an element should be inserted, using only the `k` argument to check.
 
 ---
 

doc/unordered/copyright.adoc

@@ -15,4 +15,6 @@ Copyright (C) 2022-2023 Joaquín M López Muñoz
 
 Copyright (C) 2022-2023 Peter Dimov
 
+Copyright (C) 2024 Braden Ganetsky
+
 Distributed under the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)

doc/unordered/unordered_flat_map.adoc

@@ -750,6 +750,8 @@ Returns:;; The `bool` component of the return type is `true` if an insert took p
 If an insert took place, then the iterator points to the newly inserted element. Otherwise, it points to the element with equivalent key.
 Throws:;; If an exception is thrown by an operation other than a call to `hasher` the function has no effect.
 Notes:;; Can invalidate iterators, pointers and references, but only if the insert causes the load to be greater than the maximum load. +
++
+If `args...` is of the form `k,v`, it delays constructing the whole object until it is certain that an element should be inserted, using only the `k` argument to check.
 
 ---
 
@@ -769,6 +771,8 @@ Returns:;; The `bool` component of the return type is `true` if an insert took p
 If an insert took place, then the iterator points to the newly inserted element. Otherwise, it points to the element with equivalent key.
 Throws:;; If an exception is thrown by an operation other than a call to `hasher` the function has no effect.
 Notes:;; Can invalidate iterators, pointers and references, but only if the insert causes the load to be greater than the maximum load. +
++
+If `args...` is of the form `k,v`, it delays constructing the whole object until it is certain that an element should be inserted, using only the `k` argument to check.
 
 ---
 

doc/unordered/unordered_map.adoc

@@ -797,7 +797,9 @@ If an insert took place, then the iterator points to the newly inserted element.
 Throws:;; If an exception is thrown by an operation other than a call to `hasher` the function has no effect.
 Notes:;; Can invalidate iterators, but only if the insert causes the load factor to be greater to or equal to the maximum load factor. +
 +
-Pointers and references to elements are never invalidated.
+Pointers and references to elements are never invalidated. +
++
+If `args...` is of the form `k,v`, it delays constructing the whole object until it is certain that an element should be inserted, using only the `k` argument to check. This optimization happens when the map's `key_type` is move constructible or when the `k` argument is a `key_type`.
 
 ---
 
@@ -818,7 +820,9 @@ Notes:;; The standard is fairly vague on the meaning of the hint. But the only p
 +
 Can invalidate iterators, but only if the insert causes the load factor to be greater to or equal to the maximum load factor. +
 +
-Pointers and references to elements are never invalidated.
+Pointers and references to elements are never invalidated. +
++
+If `args...` is of the form `k,v`, it delays constructing the whole object until it is certain that an element should be inserted, using only the `k` argument to check. This optimization happens when the map's `key_type` is move constructible or when the `k` argument is a `key_type`.
 
 ---
 

doc/unordered/unordered_node_map.adoc

@@ -767,6 +767,8 @@ Returns:;; The `bool` component of the return type is `true` if an insert took p
 If an insert took place, then the iterator points to the newly inserted element. Otherwise, it points to the element with equivalent key.
 Throws:;; If an exception is thrown by an operation other than a call to `hasher` the function has no effect.
 Notes:;; Can invalidate iterators, but only if the insert causes the load to be greater than the maximum load. +
++
+If `args...` is of the form `k,v`, it delays constructing the whole object until it is certain that an element should be inserted, using only the `k` argument to check. This optimization happens when `key_type` is move constructible or when the `k` argument is a `key_type`.
 
 ---
 
@@ -786,6 +788,8 @@ Returns:;; The `bool` component of the return type is `true` if an insert took p
 If an insert took place, then the iterator points to the newly inserted element. Otherwise, it points to the element with equivalent key.
 Throws:;; If an exception is thrown by an operation other than a call to `hasher` the function has no effect.
 Notes:;; Can invalidate iterators, but only if the insert causes the load to be greater than the maximum load. +
++
+If `args...` is of the form `k,v`, it delays constructing the whole object until it is certain that an element should be inserted, using only the `k` argument to check. This optimization happens when `key_type` is move constructible or when the `k` argument is a `key_type`.
 
 ---
 

include/boost/unordered/detail/allocator_constructed.hpp

@@ -0,0 +1,59 @@
+/* Copyright 2024 Braden Ganetsky.
+ * Distributed under the Boost Software License, Version 1.0.
+ * (See accompanying file LICENSE_1_0.txt or copy at
+ * http://www.boost.org/LICENSE_1_0.txt)
+ *
+ * See https://www.boost.org/libs/unordered for library home page.
+ */
+
+#ifndef BOOST_UNORDERED_DETAIL_ALLOCATOR_CONSTRUCTED_HPP
+#define BOOST_UNORDERED_DETAIL_ALLOCATOR_CONSTRUCTED_HPP
+
+#include <boost/core/allocator_traits.hpp>
+#include <boost/unordered/detail/opt_storage.hpp>
+
+namespace boost {
+  namespace unordered {
+    namespace detail {
+
+      struct allocator_policy
+      {
+        template <class Allocator, class T, class... Args>
+        static void construct(Allocator& a, T* p, Args&&... args)
+        {
+          boost::allocator_construct(a, p, std::forward<Args>(args)...);
+        }
+
+        template <class Allocator, class T>
+        static void destroy(Allocator& a, T* p)
+        {
+          boost::allocator_destroy(a, p);
+        }
+      };
+
+      /* constructs a stack-based object with the given policy and allocator */
+      template <class Allocator, class T, class Policy = allocator_policy>
+      class allocator_constructed
+      {
+        opt_storage<T> storage;
+        Allocator alloc;
+
+      public:
+        template <class... Args>
+        allocator_constructed(Allocator const& alloc_, Args&&... args)
+            : alloc(alloc_)
+        {
+          Policy::construct(
+            alloc, storage.address(), std::forward<Args>(args)...);
+        }
+
+        ~allocator_constructed() { Policy::destroy(alloc, storage.address()); }
+
+        T& value() { return *storage.address(); }
+      };
+
+    } /* namespace detail */
+  }   /* namespace unordered */
+} /* namespace boost */
+
+#endif

include/boost/unordered/detail/foa/concurrent_table.hpp

@@ -1,6 +1,7 @@
 /* Fast open-addressing concurrent hash table.
  *
  * Copyright 2023 Joaquin M Lopez Munoz.
+ * Copyright 2024 Braden Ganetsky.
  * Distributed under the Boost Software License, Version 1.0.
  * (See accompanying file LICENSE_1_0.txt or copy at
  * http://www.boost.org/LICENSE_1_0.txt)
@@ -691,6 +692,18 @@ public:
     return emplace_impl(std::forward<Value>(x));
   }
 
+  /* Optimizations for maps for (k,v) to avoid eagerly constructing value */
+  template <typename K, typename V>
+  BOOST_FORCEINLINE auto emplace(K&& k, V&& v) ->
+    typename std::enable_if<is_emplace_kv_able<concurrent_table, K>::value,
+      bool>::type
+  {
+    alloc_cted_or_fwded_key_type<type_policy, Allocator, K&&> x(
+      this->al(), std::forward<K>(k));
+    return emplace_impl(
+      try_emplace_args_t{}, x.move_or_fwd(), std::forward<V>(v));
+  }
+
   BOOST_FORCEINLINE bool
   insert(const init_type& x){return emplace_impl(x);}
 
@@ -1320,7 +1333,7 @@ private:
   {
     auto lck=shared_access();
 
-    auto x=alloc_make_insert_type<type_policy>(
+    alloc_cted_insert_type<type_policy,Allocator,Args...> x(
       this->al(),std::forward<Args>(args)...);
     int res=unprotected_norehash_emplace_or_visit(
       access_mode,std::forward<F>(f),type_policy::move(x.value()));

include/boost/unordered/detail/foa/core.hpp

@@ -2,6 +2,7 @@
  *
  * Copyright 2022-2023 Joaquin M Lopez Munoz.
  * Copyright 2023 Christian Mazakas.
+ * Copyright 2024 Braden Ganetsky.
  * Distributed under the Boost Software License, Version 1.0.
  * (See accompanying file LICENSE_1_0.txt or copy at
  * http://www.boost.org/LICENSE_1_0.txt)
@@ -22,6 +23,7 @@
 #include <boost/core/pointer_traits.hpp>
 #include <boost/cstdint.hpp>
 #include <boost/predef.h>
+#include <boost/unordered/detail/allocator_constructed.hpp>
 #include <boost/unordered/detail/narrow_cast.hpp>
 #include <boost/unordered/detail/mulx.hpp>
 #include <boost/unordered/detail/static_assert.hpp>
@@ -1220,22 +1222,15 @@ class alloc_cted_insert_type
     emplace_type,typename TypePolicy::element_type
   >::type;
 
-  alignas(insert_type) unsigned char storage[sizeof(insert_type)];
+  using alloc_cted = allocator_constructed<Allocator, insert_type, TypePolicy>;
-  Allocator                          al;
+  alloc_cted val;
 
 public:
-  alloc_cted_insert_type(const Allocator& al_,Args&&... args):al{al_}
+  alloc_cted_insert_type(const Allocator& al_,Args&&... args):val{al_,std::forward<Args>(args)...}
   {
-    TypePolicy::construct(al,data(),std::forward<Args>(args)...);
   }
 
-  ~alloc_cted_insert_type()
+  insert_type& value(){return val.value();}
-  {
-    TypePolicy::destroy(al,data());
-  }
-
-  insert_type* data(){return reinterpret_cast<insert_type*>(&storage);}
-  insert_type& value(){return *data();}
 };
 
 template<typename TypePolicy,typename Allocator,typename... Args>
@@ -1245,6 +1240,51 @@ alloc_make_insert_type(const Allocator& al,Args&&... args)
   return {al,std::forward<Args>(args)...};
 }
 
+template <typename TypePolicy, typename Allocator, typename KFwdRef,
+  typename = void>
+class alloc_cted_or_fwded_key_type
+{
+  using key_type = typename TypePolicy::key_type;
+  allocator_constructed<Allocator, key_type, TypePolicy> val;
+
+public:
+  alloc_cted_or_fwded_key_type(const Allocator& al_, KFwdRef k)
+      : val(al_, std::forward<KFwdRef>(k))
+  {
+  }
+
+  key_type&& move_or_fwd() { return std::move(val.value()); }
+};
+
+template <typename TypePolicy, typename Allocator, typename KFwdRef>
+class alloc_cted_or_fwded_key_type<TypePolicy, Allocator, KFwdRef,
+  typename std::enable_if<
+    is_similar<KFwdRef, typename TypePolicy::key_type>::value>::type>
+{
+  // This specialization acts as a forwarding-reference wrapper
+  BOOST_UNORDERED_STATIC_ASSERT(std::is_reference<KFwdRef>::value);
+  KFwdRef ref;
+
+public:
+  alloc_cted_or_fwded_key_type(const Allocator&, KFwdRef k)
+      : ref(std::forward<KFwdRef>(k))
+  {
+  }
+
+  KFwdRef move_or_fwd() { return std::forward<KFwdRef>(ref); }
+};
+
+template <typename Container>
+using is_map =
+  std::integral_constant<bool, !std::is_same<typename Container::key_type,
+                                 typename Container::value_type>::value>;
+
+template <typename Container, typename K>
+using is_emplace_kv_able = std::integral_constant<bool,
+  is_map<Container>::value &&
+    (is_similar<K, typename Container::key_type>::value ||
+      is_complete_and_move_constructible<typename Container::key_type>::value)>;
+
 /* table_core. The TypePolicy template parameter is used to generate
  * instantiations suitable for either maps or sets, and introduces non-standard
  * init_type and element_type:
@@ -1262,7 +1302,7 @@ alloc_make_insert_type(const Allocator& al,Args&&... args)
  *
  *   - TypePolicy::construct and TypePolicy::destroy are used for the
  *     construction and destruction of the internal types: value_type,
- *     init_type and element_type.
+ *     init_type, element_type, and key_type.
  * 
  *   - TypePolicy::move is used to provide move semantics for the internal
  *     types used by the container during rehashing and emplace. These types

include/boost/unordered/detail/foa/flat_map_types.hpp

@@ -1,4 +1,5 @@
 // Copyright (C) 2023 Christian Mazakas
+// Copyright (C) 2024 Braden Ganetsky
 // Distributed under the Boost Software License, Version 1.0. (See accompanying
 // file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
 
@@ -56,6 +57,12 @@ namespace boost {
             boost::allocator_construct(al, p, std::forward<Args>(args)...);
           }
 
+          template <class A, class... Args>
+          static void construct(A& al, key_type* p, Args&&... args)
+          {
+            boost::allocator_construct(al, p, std::forward<Args>(args)...);
+          }
+
           template <class A> static void destroy(A& al, init_type* p) noexcept
           {
             boost::allocator_destroy(al, p);
@@ -65,6 +72,11 @@ namespace boost {
           {
             boost::allocator_destroy(al, p);
           }
+
+          template <class A> static void destroy(A& al, key_type* p) noexcept
+          {
+            boost::allocator_destroy(al, p);
+          }
         };
       } // namespace foa
     }   // namespace detail

include/boost/unordered/detail/foa/node_map_types.hpp

@@ -1,4 +1,5 @@
 // Copyright (C) 2023 Christian Mazakas
+// Copyright (C) 2024 Braden Ganetsky
 // Distributed under the Boost Software License, Version 1.0. (See accompanying
 // file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
 
@@ -82,6 +83,12 @@ namespace boost {
             boost::allocator_construct(al, p, std::forward<Args>(args)...);
           }
 
+          template <class A, class... Args>
+          static void construct(A& al, key_type* p, Args&&... args)
+          {
+            boost::allocator_construct(al, p, std::forward<Args>(args)...);
+          }
+
           template <class A, class... Args>
           static void construct(A& al, element_type* p, Args&&... args)
           {
@@ -109,6 +116,11 @@ namespace boost {
             boost::allocator_destroy(al, p);
           }
 
+          template <class A> static void destroy(A& al, key_type* p) noexcept
+          {
+            boost::allocator_destroy(al, p);
+          }
+
           template <class A>
           static void destroy(A& al, element_type* p) noexcept
           {

include/boost/unordered/detail/foa/table.hpp

@@ -2,6 +2,7 @@
  *
  * Copyright 2022-2023 Joaquin M Lopez Munoz.
  * Copyright 2023 Christian Mazakas.
+ * Copyright 2024 Braden Ganetsky.
  * Distributed under the Boost Software License, Version 1.0.
  * (See accompanying file LICENSE_1_0.txt or copy at
  * http://www.boost.org/LICENSE_1_0.txt)
@@ -401,7 +402,7 @@ public:
   template<typename... Args>
   BOOST_FORCEINLINE std::pair<iterator,bool> emplace(Args&&... args)
   {
-    auto x=alloc_make_insert_type<type_policy>(
+    alloc_cted_insert_type<type_policy,Allocator,Args...> x(
       this->al(),std::forward<Args>(args)...);
     return emplace_impl(type_policy::move(x.value()));
   }
@@ -416,6 +417,19 @@ public:
     return emplace_impl(std::forward<T>(x));
   }
 
+  /* Optimizations for maps for (k,v) to avoid eagerly constructing value */
+  template <typename K, typename V>
+  BOOST_FORCEINLINE
+    typename std::enable_if<is_emplace_kv_able<table, K>::value,
+      std::pair<iterator, bool> >::type
+    emplace(K&& k, V&& v)
+  {
+    alloc_cted_or_fwded_key_type<type_policy, Allocator, K&&> x(
+      this->al(), std::forward<K>(k));
+    return emplace_impl(
+      try_emplace_args_t{}, x.move_or_fwd(), std::forward<V>(v));
+  }
+
   template<typename Key,typename... Args>
   BOOST_FORCEINLINE std::pair<iterator,bool> try_emplace(
     Key&& x,Args&&... args)

include/boost/unordered/detail/implementation.hpp

@@ -2,6 +2,7 @@
 // Copyright (C) 2005-2016 Daniel James
 // Copyright (C) 2022-2023 Joaquin M Lopez Munoz.
 // Copyright (C) 2022-2023 Christian Mazakas
+// Copyright (C) 2024 Braden Ganetsky
 //
 // Distributed under the Boost Software License, Version 1.0. (See accompanying
 // file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
@@ -14,6 +15,7 @@
 #pragma once
 #endif
 
+#include <boost/unordered/detail/allocator_constructed.hpp>
 #include <boost/unordered/detail/fca.hpp>
 #include <boost/unordered/detail/opt_storage.hpp>
 #include <boost/unordered/detail/serialize_tracked_address.hpp>
@@ -104,6 +106,10 @@ namespace boost {
         template <class T> no_key(T const&) {}
       };
 
+      struct converting_key
+      {
+      };
+
       namespace func {
         template <class T> inline void ignore_unused_variable_warning(T const&)
         {
@@ -1913,6 +1919,16 @@ namespace boost {
           }
         }
 
+        template <typename K, typename V>
+        emplace_return emplace_unique(converting_key, K&& k, V&& v)
+        {
+          using alloc_cted = allocator_constructed<node_allocator_type,
+            typename Types::key_type>;
+          alloc_cted key(this->node_alloc(), std::forward<K>(k));
+          return emplace_unique(
+            key.value(), std::move(key.value()), std::forward<V>(v));
+        }
+
         template <typename Key> emplace_return try_emplace_unique(Key&& k)
         {
           std::size_t key_hash = this->hash(k);
@@ -2835,9 +2851,13 @@ namespace boost {
         }
 
         template <class Arg1, class Arg2>
-        static no_key extract(Arg1 const&, Arg2 const&)
+        static typename std::conditional<
+          (is_similar<Arg1, key_type>::value ||
+            is_complete_and_move_constructible<key_type>::value),
+          converting_key, no_key>::type
+        extract(Arg1 const&, Arg2 const&)
         {
-          return no_key();
+          return {};
         }
 
         template <class Arg1, class Arg2, class Arg3, class... Args>

include/boost/unordered/detail/map.hpp

@@ -1,6 +1,7 @@
 
 // Copyright (C) 2005-2016 Daniel James
 // Copyright (C) 2022 Christian Mazakas
+// Copyright (C) 2024 Braden Ganetsky
 // Distributed under the Boost Software License, Version 1.0. (See accompanying
 // file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
 
@@ -18,6 +19,7 @@ namespace boost {
         typedef std::pair<K const, M> value_type;
         typedef H hasher;
         typedef P key_equal;
+        typedef K key_type;
         typedef K const const_key_type;
 
         typedef

include/boost/unordered/detail/type_traits.hpp

@@ -1,4 +1,5 @@
 // Copyright (C) 2022-2023 Christian Mazakas
+// Copyright (C) 2024 Braden Ganetsky
 //
 // Distributed under the Boost Software License, Version 1.0. (See accompanying
 // file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
@@ -48,6 +49,20 @@ namespace boost {
 
       template <typename... Ts> using void_t = typename make_void<Ts...>::type;
 
+      template <class T, class = void> struct is_complete : std::false_type
+      {
+      };
+
+      template <class T>
+      struct is_complete<T, void_t<int[sizeof(T)]> > : std::true_type
+      {
+      };
+
+      template <class T>
+      using is_complete_and_move_constructible =
+        typename std::conditional<is_complete<T>::value,
+          std::is_move_constructible<T>, std::false_type>::type;
+
 #if BOOST_WORKAROUND(BOOST_LIBSTDCXX_VERSION, < 50000)
       /* std::is_trivially_default_constructible not provided */
       template <class T>

test/cfoa/emplace_tests.cpp

@@ -1,9 +1,11 @@
 // Copyright (C) 2023 Christian Mazakas
 // Copyright (C) 2023 Joaquin M Lopez Munoz
+// Copyright (C) 2024 Braden Ganetsky
 // Distributed under the Boost Software License, Version 1.0. (See accompanying
 // file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
 
 #include "helpers.hpp"
+#include "../helpers/count.hpp"
 
 #include <boost/unordered/concurrent_flat_map.hpp>
 #include <boost/unordered/concurrent_flat_set.hpp>
@@ -51,7 +53,7 @@ namespace {
 
   struct lvalue_emplacer_type
   {
-    template <class T, class X> void operator()(std::vector<T>& values, X& x)
+    template <class T, class X> void call_impl(std::vector<T>& values, X& x)
     {
       static constexpr auto value_type_cardinality = 
         value_cardinality<typename X::value_type>::value;
@@ -66,28 +68,33 @@ namespace {
         }
       });
       BOOST_TEST_EQ(num_inserts, x.size());
-      BOOST_TEST_EQ(
-        raii::default_constructor, value_type_cardinality * values.size());
 
-      BOOST_TEST_EQ(raii::copy_constructor, 0u);
+      std::uint64_t const default_constructors = value_type_cardinality == 2
-      BOOST_TEST_GE(raii::move_constructor, value_type_cardinality * x.size());
+                                                   ? values.size() + num_inserts
+                                                   : values.size();
+      BOOST_TEST_EQ(raii::default_constructor, default_constructors);
 
       BOOST_TEST_EQ(raii::copy_constructor, 0u);
       BOOST_TEST_EQ(raii::copy_assignment, 0u);
       BOOST_TEST_EQ(raii::move_assignment, 0u);
     }
+    template <class T, class X> void operator()(std::vector<T>& values, X& x)
+    {
+      static constexpr auto value_type_cardinality =
+        value_cardinality<typename X::value_type>::value;
+
+      call_impl(values, x);
+      BOOST_TEST_GE(raii::move_constructor, value_type_cardinality * x.size());
+    }
   } lvalue_emplacer;
 
   struct norehash_lvalue_emplacer_type : public lvalue_emplacer_type
   {
     template <class T, class X> void operator()(std::vector<T>& values, X& x)
     {
-      static constexpr auto value_type_cardinality = 
-        value_cardinality<typename X::value_type>::value;
-
       x.reserve(values.size());
-      lvalue_emplacer_type::operator()(values, x);
+      lvalue_emplacer_type::call_impl(values, x);
-      BOOST_TEST_EQ(raii::move_constructor, value_type_cardinality * x.size());
+      BOOST_TEST_EQ(raii::move_constructor, x.size());
     }
   } norehash_lvalue_emplacer;
 
@@ -293,4 +300,174 @@ UNORDERED_TEST(
 
 // clang-format on
 
+namespace {
+  using converting_key_type = basic_raii<struct converting_key_tag_>;
+  using converting_value_type = basic_raii<struct converting_value_tag_>;
+
+  class counted_key_type : public basic_raii<struct counted_key_tag_>
+  {
+  public:
+    using basic_raii::basic_raii;
+    counted_key_type() = default;
+    counted_key_type(const converting_key_type& k) : counted_key_type(k.x_) {}
+  };
+  class counted_value_type : public basic_raii<struct counted_value_tag_>
+  {
+  public:
+    using basic_raii::basic_raii;
+    counted_value_type() = default;
+    counted_value_type(const converting_value_type& v)
+        : counted_value_type(v.x_)
+    {
+    }
+  };
+
+  void reset_counts()
+  {
+    counted_key_type::reset_counts();
+    counted_value_type::reset_counts();
+    converting_key_type::reset_counts();
+    converting_value_type::reset_counts();
+  }
+
+  using test::smf_count;
+
+  template <class T> smf_count count_for()
+  {
+    return test::smf_count{
+      (int)T::default_constructor.load(std::memory_order_relaxed),
+      (int)T::copy_constructor.load(std::memory_order_relaxed),
+      (int)T::move_constructor.load(std::memory_order_relaxed),
+      (int)T::copy_assignment.load(std::memory_order_relaxed),
+      (int)T::move_assignment.load(std::memory_order_relaxed),
+      (int)T::destructor.load(std::memory_order_relaxed)};
+  }
+
+  enum emplace_kind
+  {
+    copy,
+    move
+  };
+
+  enum emplace_status
+  {
+    fail,
+    success
+  };
+
+  struct counted_key_checker_type
+  {
+    using key_type = counted_key_type;
+    void operator()(emplace_kind kind, emplace_status status)
+    {
+      int copies = (kind == copy && status == success) ? 1 : 0;
+      int moves = (kind == move && status == success) ? 1 : 0;
+      BOOST_TEST_EQ(
+        count_for<counted_key_type>(), (smf_count{0, copies, moves, 0, 0, 0}));
+    }
+  } counted_key_checker;
+
+  struct converting_key_checker_type
+  {
+    using key_type = converting_key_type;
+    void operator()(emplace_kind, emplace_status status)
+    {
+      int moves = (status == success) ? 1 : 0;
+      BOOST_TEST_EQ(
+        count_for<counted_key_type>(), (smf_count{1, 0, moves, 0, 0, 1}));
+    }
+  } converting_key_checker;
+
+  struct counted_value_checker_type
+  {
+    using mapped_type = counted_value_type;
+    void operator()(emplace_kind kind, emplace_status status)
+    {
+      int copies = (kind == copy && status == success) ? 1 : 0;
+      int moves = (kind == move && status == success) ? 1 : 0;
+      BOOST_TEST_EQ(count_for<counted_value_type>(),
+        (smf_count{0, copies, moves, 0, 0, 0}));
+    }
+  } counted_value_checker;
+
+  struct converting_value_checker_type
+  {
+    using mapped_type = converting_value_type;
+    void operator()(emplace_kind, emplace_status status)
+    {
+      int ctors = (status == success) ? 1 : 0;
+      BOOST_TEST_EQ(
+        count_for<counted_value_type>(), (smf_count{ctors, 0, 0, 0, 0, 0}));
+    }
+  } converting_value_checker;
+
+  template <class X, class KC, class VC>
+  void emplace_map_key_value(
+    X*, emplace_kind kind, KC key_checker, VC value_checker)
+  {
+    using container = X;
+    using key_type = typename KC::key_type;
+    using mapped_type = typename VC::mapped_type;
+
+    container x;
+    key_type key{};
+    key_type key2 = key;
+    mapped_type value{};
+    mapped_type value2 = value;
+
+    {
+      reset_counts();
+      auto ret = (kind == copy) ? x.emplace(key, value)
+                                : x.emplace(std::move(key), std::move(value));
+      BOOST_TEST_EQ(ret, true);
+      key_checker(kind, success);
+      value_checker(kind, success);
+      BOOST_TEST_EQ(
+        count_for<converting_key_type>(), (smf_count{0, 0, 0, 0, 0, 0}));
+      BOOST_TEST_EQ(
+        count_for<converting_value_type>(), (smf_count{0, 0, 0, 0, 0, 0}));
+    }
+
+    {
+      reset_counts();
+      bool ret = x.emplace(key2, value2);
+      BOOST_TEST_EQ(ret, false);
+      key_checker(kind, fail);
+      value_checker(kind, fail);
+      BOOST_TEST_EQ(
+        count_for<converting_key_type>(), (smf_count{0, 0, 0, 0, 0, 0}));
+      BOOST_TEST_EQ(
+        count_for<converting_value_type>(), (smf_count{0, 0, 0, 0, 0, 0}));
+    }
+
+    {
+      reset_counts();
+      bool ret = x.emplace(std::move(key2), std::move(value2));
+      BOOST_TEST_EQ(ret, false);
+      key_checker(kind, fail);
+      value_checker(kind, fail);
+      BOOST_TEST_EQ(
+        count_for<converting_key_type>(), (smf_count{0, 0, 0, 0, 0, 0}));
+      BOOST_TEST_EQ(
+        count_for<converting_value_type>(), (smf_count{0, 0, 0, 0, 0, 0}));
+    }
+  }
+
+  boost::unordered::concurrent_flat_map<counted_key_type, counted_value_type>*
+    test_counted_flat_map = {};
+
+} // namespace
+
+// clang-format off
+
+UNORDERED_TEST(
+  emplace_map_key_value,
+  ((test_counted_flat_map))
+  ((copy)(move))
+  ((counted_key_checker)(converting_key_checker))
+  ((counted_value_checker)(converting_value_checker))
+)
+
+// clang-format on
+
 RUN_TESTS()

test/cfoa/helpers.hpp

@@ -1,5 +1,6 @@
 // Copyright (C) 2023 Christian Mazakas
 // Copyright (C) 2023 Joaquin M Lopez Munoz
+// Copyright (C) 2024 Braden Ganetsky
 // Distributed under the Boost Software License, Version 1.0. (See accompanying
 // file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
 
@@ -207,7 +208,8 @@ template <class T> struct stateful_allocator2
   bool operator!=(stateful_allocator2 const& rhs) const { return x_ != rhs.x_; }
 };
 
-struct raii
+template <class Tag>
+struct basic_raii
 {
   static std::atomic<std::uint32_t> default_constructor;
   static std::atomic<std::uint32_t> copy_constructor;
@@ -219,17 +221,17 @@ struct raii
 
   int x_ = -1;
 
-  raii() { ++default_constructor; }
+  basic_raii() { ++default_constructor; }
-  raii(int const x) : x_{x} { ++default_constructor; }
+  basic_raii(int const x) : x_{x} { ++default_constructor; }
-  raii(raii const& rhs) : x_{rhs.x_} { ++copy_constructor; }
+  basic_raii(basic_raii const& rhs) : x_{rhs.x_} { ++copy_constructor; }
-  raii(raii&& rhs) noexcept : x_{rhs.x_}
+  basic_raii(basic_raii&& rhs) noexcept : x_{rhs.x_}
   {
     rhs.x_ = -1;
     ++move_constructor;
   }
-  ~raii() { ++destructor; }
+  ~basic_raii() { ++destructor; }
 
-  raii& operator=(raii const& rhs)
+  basic_raii& operator=(basic_raii const& rhs)
   {
     ++copy_assignment;
     if (this != &rhs) {
@@ -238,7 +240,7 @@ struct raii
     return *this;
   }
 
-  raii& operator=(raii&& rhs) noexcept
+  basic_raii& operator=(basic_raii&& rhs) noexcept
   {
     ++move_assignment;
     if (this != &rhs) {
@@ -248,37 +250,37 @@ struct raii
     return *this;
   }
 
-  friend bool operator==(raii const& lhs, raii const& rhs)
+  friend bool operator==(basic_raii const& lhs, basic_raii const& rhs)
   {
     return lhs.x_ == rhs.x_;
   }
 
-  friend bool operator!=(raii const& lhs, raii const& rhs)
+  friend bool operator!=(basic_raii const& lhs, basic_raii const& rhs)
   {
     return !(lhs == rhs);
   }
 
-  friend bool operator==(raii const& lhs, int const x) { return lhs.x_ == x; }
+  friend bool operator==(basic_raii const& lhs, int const x) { return lhs.x_ == x; }
-  friend bool operator!=(raii const& lhs, int const x)
+  friend bool operator!=(basic_raii const& lhs, int const x)
   {
     return !(lhs.x_ == x);
   }
 
-  friend bool operator==(int const x, raii const& rhs) { return rhs.x_ == x; }
+  friend bool operator==(int const x, basic_raii const& rhs) { return rhs.x_ == x; }
 
-  friend bool operator!=(int const x, raii const& rhs)
+  friend bool operator!=(int const x, basic_raii const& rhs)
   {
     return !(rhs.x_ == x);
   }
 
-  friend std::ostream& operator<<(std::ostream& os, raii const& rhs)
+  friend std::ostream& operator<<(std::ostream& os, basic_raii const& rhs)
   {
     os << "{ x_: " << rhs.x_ << " }";
     return os;
   }
 
   friend std::ostream& operator<<(
-    std::ostream& os, std::pair<raii const, raii> const& rhs)
+    std::ostream& os, std::pair<basic_raii const, basic_raii> const& rhs)
   {
     os << "pair<" << rhs.first << ", " << rhs.second << ">";
     return os;
@@ -294,16 +296,30 @@ struct raii
     move_assignment = 0;
   }
 
-  friend void swap(raii& lhs, raii& rhs) { std::swap(lhs.x_, rhs.x_); }
+  friend void swap(basic_raii& lhs, basic_raii& rhs) { std::swap(lhs.x_, rhs.x_); }
 };
 
-std::atomic<std::uint32_t> raii::default_constructor{0};
+template <class Tag> std::atomic<std::uint32_t> basic_raii<Tag>::default_constructor(0);
-std::atomic<std::uint32_t> raii::copy_constructor{0};
+template <class Tag> std::atomic<std::uint32_t> basic_raii<Tag>::copy_constructor(0);
-std::atomic<std::uint32_t> raii::move_constructor{0};
+template <class Tag> std::atomic<std::uint32_t> basic_raii<Tag>::move_constructor(0);
-std::atomic<std::uint32_t> raii::destructor{0};
+template <class Tag> std::atomic<std::uint32_t> basic_raii<Tag>::destructor(0);
-std::atomic<std::uint32_t> raii::copy_assignment{0};
+template <class Tag> std::atomic<std::uint32_t> basic_raii<Tag>::copy_assignment(0);
-std::atomic<std::uint32_t> raii::move_assignment{0};
+template <class Tag> std::atomic<std::uint32_t> basic_raii<Tag>::move_assignment(0);
 
+struct raii_tag_
+{
+};
+class raii : public basic_raii<raii_tag_>
+{
+  using basic_raii::basic_raii;
+};
+
+template <class Tag>
+std::size_t hash_value(basic_raii<Tag> const& r) noexcept
+{
+  boost::hash<int> hasher;
+  return hasher(r.x_);
+}
 std::size_t hash_value(raii const& r) noexcept
 {
   boost::hash<int> hasher;
@@ -311,6 +327,13 @@ std::size_t hash_value(raii const& r) noexcept
 }
 
 namespace std {
+  template <class Tag> struct hash<basic_raii<Tag>>
+  {
+    std::size_t operator()(basic_raii<Tag> const& r) const noexcept
+    {
+      return hash_value(r);
+    }
+  };
   template <> struct hash<raii>
   {
     std::size_t operator()(raii const& r) const noexcept

test/helpers/count.hpp

@@ -1,5 +1,6 @@
 
 // Copyright 2008-2009 Daniel James.
+// Copyright 2024 Braden Ganetsky.
 // Distributed under the Boost Software License, Version 1.0. (See accompanying
 // file LICENSE_1_0.txt or move at http://www.boost.org/LICENSE_1_0.txt)
 
@@ -147,6 +148,8 @@ namespace test {
     static smf_count count;
     static void reset_count() { count.reset(); }
 
+    smf_counted_object(int index) : smf_counted_object() { index_ = index; }
+
     smf_counted_object() : index_(++running_index)
     {
       count.default_construct();
@@ -184,9 +187,10 @@ namespace test {
       return boost::hash<int>()(x.index_);
     }
 
+    int index_;
+
   private:
     static int running_index;
-    int index_;
   };
   template <class Tag> smf_count smf_counted_object<Tag>::count = {};
   template <class Tag> int smf_counted_object<Tag>::running_index = 0;

test/helpers/unordered.hpp

@@ -21,4 +21,10 @@
 #include "postfix.hpp"
 // clang-format on
 
+#if defined(BOOST_LIBSTDCXX_VERSION)
+#if BOOST_LIBSTDCXX_VERSION < 60000
+#define BOOST_UNORDERED_NO_INIT_TYPE_TESTS
+#endif
+#endif
+
 #endif

test/unordered/emplace_smf_tests.cpp

@@ -1,5 +1,5 @@
 //
-// Copyright 2023 Braden Ganetsky.
+// Copyright 2023-2024 Braden Ganetsky.
 // Distributed under the Boost Software License, Version 1.0. (See accompanying
 // file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
 
@@ -12,12 +12,47 @@ namespace emplace_smf_tests {
   using test::smf_count;
   using test::smf_counted_object;
 
-  using counted_key = smf_counted_object<struct key_tag_>;
+  using converting_key = smf_counted_object<struct cvt_key_tag_>;
-  using counted_value = smf_counted_object<struct value_tag_>;
+  using converting_value = smf_counted_object<struct cvt_value_tag_>;
+
+  class counted_key : public smf_counted_object<struct key_tag_>
+  {
+  public:
+    using smf_counted_object::smf_counted_object;
+    counted_key() = default;
+    counted_key(const converting_key& k) : counted_key(k.index_) {}
+  };
+  class counted_value : public smf_counted_object<struct value_tag_>
+  {
+  public:
+    using smf_counted_object::smf_counted_object;
+    counted_value() = default;
+    counted_value(const converting_value& v) : counted_value(v.index_) {}
+  };
+
+  class immovable_key : public smf_counted_object<struct imm_key_tag_>
+  {
+  public:
+    using smf_counted_object::smf_counted_object;
+    immovable_key(immovable_key&&) = delete;
+    immovable_key& operator=(immovable_key&&) = delete;
+  };
+  class immovable_value : public smf_counted_object<struct imm_value_tag_>
+  {
+  public:
+    using smf_counted_object::smf_counted_object;
+    immovable_value(immovable_value&&) = delete;
+    immovable_value& operator=(immovable_value&&) = delete;
+  };
+
   void reset_counts()
   {
     counted_key::reset_count();
     counted_value::reset_count();
+    converting_key::reset_count();
+    converting_value::reset_count();
+    immovable_key::reset_count();
+    immovable_value::reset_count();
   }
 
 #ifdef BOOST_UNORDERED_FOA_TESTS
@@ -170,6 +205,173 @@ namespace emplace_smf_tests {
   }
 
   UNORDERED_TEST(emplace_smf_value_type_set, EMPLACE_SMF_TESTS_SET_ARGS)
+
+  enum emplace_kind
+  {
+    copy,
+    move
+  };
+
+  enum emplace_status
+  {
+    fail,
+    success
+  };
+
+  struct counted_key_checker_type
+  {
+    using key_type = counted_key;
+    void operator()(emplace_kind kind, emplace_status status)
+    {
+      int copies = (kind == copy && status == success) ? 1 : 0;
+      int moves = (kind == move && status == success) ? 1 : 0;
+      BOOST_TEST_EQ(counted_key::count, (smf_count{0, copies, moves, 0, 0, 0}));
+    }
+  } counted_key_checker;
+
+  struct converting_key_checker_type
+  {
+    using key_type = converting_key;
+    void operator()(emplace_kind, emplace_status status)
+    {
+      int moves = (status == success) ? 1 : 0;
+      BOOST_TEST_EQ(counted_key::count, (smf_count{1, 0, moves, 0, 0, 1}));
+    }
+  } converting_key_checker;
+
+  struct counted_value_checker_type
+  {
+    using mapped_type = counted_value;
+    void operator()(emplace_kind kind, emplace_status status)
+    {
+      int copies = (kind == copy && status == success) ? 1 : 0;
+      int moves = (kind == move && status == success) ? 1 : 0;
+      BOOST_TEST_EQ(
+        counted_value::count, (smf_count{0, copies, moves, 0, 0, 0}));
+    }
+  } counted_value_checker;
+
+  struct converting_value_checker_type
+  {
+    using mapped_type = converting_value;
+    void operator()(emplace_kind, emplace_status status)
+    {
+      int ctors = (status == success) ? 1 : 0;
+      BOOST_TEST_EQ(counted_value::count, (smf_count{ctors, 0, 0, 0, 0, 0}));
+    }
+  } converting_value_checker;
+
+  template <class X, class KC, class VC>
+  static void emplace_smf_key_value_map(
+    X*, emplace_kind kind, KC key_checker, VC value_checker)
+  {
+    using container = X;
+    using key_type = typename KC::key_type;
+    using mapped_type = typename VC::mapped_type;
+
+    container x;
+    key_type key{};
+    key_type key2 = key;
+    mapped_type value{};
+    mapped_type value2 = value;
+
+    {
+      reset_counts();
+      auto ret = (kind == copy) ? x.emplace(key, value)
+                                : x.emplace(std::move(key), std::move(value));
+      BOOST_TEST_EQ(ret.second, true);
+      key_checker(kind, success);
+      value_checker(kind, success);
+      BOOST_TEST_EQ(converting_key::count, (smf_count{0, 0, 0, 0, 0, 0}));
+      BOOST_TEST_EQ(converting_value::count, (smf_count{0, 0, 0, 0, 0, 0}));
+    }
+
+    {
+      reset_counts();
+      auto ret = x.emplace(key2, value2);
+      BOOST_TEST_EQ(ret.second, false);
+      key_checker(kind, fail);
+      value_checker(kind, fail);
+      BOOST_TEST_EQ(converting_key::count, (smf_count{0, 0, 0, 0, 0, 0}));
+      BOOST_TEST_EQ(converting_value::count, (smf_count{0, 0, 0, 0, 0, 0}));
+    }
+
+    {
+      reset_counts();
+      auto ret = x.emplace(std::move(key2), std::move(value2));
+      BOOST_TEST_EQ(ret.second, false);
+      key_checker(kind, fail);
+      value_checker(kind, fail);
+      BOOST_TEST_EQ(converting_key::count, (smf_count{0, 0, 0, 0, 0, 0}));
+      BOOST_TEST_EQ(converting_value::count, (smf_count{0, 0, 0, 0, 0, 0}));
+    }
+  }
+
+  // clang-format off
+
+  UNORDERED_TEST(
+    emplace_smf_key_value_map,
+    EMPLACE_SMF_TESTS_MAP_ARGS
+    ((copy)(move))
+    ((counted_key_checker)(converting_key_checker))
+    ((counted_value_checker)(converting_value_checker))
+  )
+
+  // clang-format on
+
+  template <class X> static void emplace_smf_key_value_map_immovable_key(X*)
+  {
+#ifndef BOOST_UNORDERED_NO_INIT_TYPE_TESTS
+    using container = X;
+    BOOST_STATIC_ASSERT(
+      std::is_same<immovable_key, typename X::key_type>::value);
+    using mapped_type = typename X::mapped_type;
+
+    container x;
+
+    {
+      reset_counts();
+      auto ret = x.emplace(0, 0);
+      BOOST_TEST_EQ(ret.second, true);
+      BOOST_TEST_EQ(immovable_key::count, (smf_count{1, 0, 0, 0, 0, 0}));
+      BOOST_TEST_EQ(mapped_type::count, (smf_count{1, 0, 0, 0, 0, 0}));
+    }
+
+    {
+      reset_counts();
+      auto ret = x.emplace(0, 1);
+      BOOST_TEST_EQ(ret.second, false);
+      BOOST_TEST_EQ(immovable_key::count, (smf_count{1, 0, 0, 0, 0, 1}));
+      BOOST_TEST_EQ(mapped_type::count, (smf_count{1, 0, 0, 0, 0, 1}));
+    }
+#endif
+  }
+
+#ifdef BOOST_UNORDERED_FOA_TESTS
+  static boost::unordered_node_map<immovable_key, counted_value>*
+    test_smf_node_map_immovable_key_counted_value;
+  static boost::unordered_node_map<immovable_key, immovable_value>*
+    test_smf_node_map_immovable_key_immovable_value;
+#define EMPLACE_SMF_TESTS_MAP_IMMOVABLE_ARGS                                   \
+  ((test_smf_node_map_immovable_key_counted_value)(test_smf_node_map_immovable_key_immovable_value))
+#else
+  static boost::unordered_map<immovable_key, counted_value>*
+    test_smf_map_immovable_key_counted_value;
+  static boost::unordered_map<immovable_key, immovable_value>*
+    test_smf_map_immovable_key_immovable_value;
+#define EMPLACE_SMF_TESTS_MAP_IMMOVABLE_ARGS                                   \
+  ((test_smf_map_immovable_key_counted_value)(test_smf_map_immovable_key_immovable_value))
+#endif
+
+  // clang-format off
+
+  UNORDERED_TEST(
+    emplace_smf_key_value_map_immovable_key,
+    EMPLACE_SMF_TESTS_MAP_IMMOVABLE_ARGS
+  )
+
+  // clang-format on
+
 } // namespace emplace_smf_tests
 
 RUN_TESTS()

test/unordered/init_type_insert_tests.cpp

@@ -12,12 +12,6 @@
 
 #include <boost/config.hpp>
 
-#if defined(BOOST_LIBSTDCXX_VERSION)
-#if BOOST_LIBSTDCXX_VERSION < 60000
-#define BOOST_UNORDERED_NO_INIT_TYPE_TESTS
-#endif
-#endif
-
 struct move_only
 {
   int x_ = -1;