Bluetooth: Mesh: Use separate workq for storing mesh settings

Currently mesh settings are stored in the system workqueue context.
Most of other stack functionality, such that advertisements (incl
relay), loopback, transport sar, beacons transmission, etc. is also
processed in the system workqueue context. When a massive amount of
data needs to be stored and in particularly when page erase needs to
be triggered by GC of NVS subsystem to allocate flash pages, the
execution of the stack (and other functionality that uses the system
workqueue) will be blocked until storing is finished. For example,
right after the provisioning of a erased device, a node may not be
responsive for up to 400ms before it can continue sending messages.
The waiting time may increase if there is a GATT connection in the
mean time.

When write or erase operation is triggered, the flash driver waits for
Bluetooth controller to allocate a time needed to perform the operation.
During the whole operation, the context from which the operation was
triggered is put to sleep. This allows other threads to run until
Bluetooth controller finds the time for the flash driver. In other words,
every settings_save_one or settings_delete should be considered as
rescheduling points.

Considering this, Bluetooth mesh can use another thread to store its
settings, thus releasing the system workqueue for other tasks including
the operation of the stack itself.

The consistency of the data to be stored is guaranteed by the current
implementation where the data is copied to another struct before calling
settings_save_one. The pending flag of a particular module is dropped in
settings.c before starting to store the corresponding data. Thus, if
during the sleep the node receives a message that triggers a change in a
module which data is currently being stored, the pending flag will be
restored and the new change will be stored eventually.

Having this option enabled including with the partial erase, will make
the node more responsive in the described situations.

Signed-off-by: Pavel Vasilyev <pavel.vasilyev@nordicsemi.no>

doc/connectivity/bluetooth/api/mesh/core.rst

@@ -67,6 +67,46 @@ vulnerability and flash wear out.
    the RPL between reboots, will make the device vulnerable to replay attacks
    and not perform the replay protection required by the spec.
 
+Persistent storage
+******************
+
+The mesh stack uses the :ref:`Settings Subsystem <settings_api>` for storing the
+device configuration persistently. When the stack configuration changes and
+the change needs to be stored persistently, the stack schedules a work item.
+The delay between scheduling the work item and submitting it to the workqueue
+is defined by the :kconfig:option:`CONFIG_BT_MESH_STORE_TIMEOUT` option. Once
+storing of data is scheduled, it can not be rescheduled until the work item is
+processed. Exceptions are made in certain cases as described below.
+
+When IV index, Sequence Number or CDB configuration have to be stored, the work
+item is submitted to the workqueue without the delay. If the work item was
+previously scheduled, it will be rescheduled without the delay.
+
+The Replay Protection List uses the same work item to store RPL entries. If
+storing of RPL entries is requested and no other configuration is pending to be
+stored, the delay is set to :kconfig:option:`CONFIG_BT_MESH_RPL_STORE_TIMEOUT`.
+If other stack configuration has to be stored, the delay defined by
+the :kconfig:option:`CONFIG_BT_MESH_STORE_TIMEOUT` option is less than
+:kconfig:option:`CONFIG_BT_MESH_RPL_STORE_TIMEOUT`, and the work item was
+scheduled by the Replay Protection List, the work item will be rescheduled.
+
+When the work item is running, the stack will store all pending configuration,
+including the RPL entries.
+
+Work item execution context
+===========================
+
+The :kconfig:option:`CONFIG_BT_MESH_SETTINGS_WORKQ` option configures the
+context from which the work item is executed. This option is enabled by
+default, and results in stack using a dedicated cooperative thread to
+process the work item. This allows the stack to process other incoming and
+outgoing messages, as well as other work items submitted to the system
+workqueue, while the stack configuration is being stored.
+
+When this option is disabled, the work item is submitted to the system workqueue.
+This means that the system workqueue is blocked for the time it takes to store
+the stack's configuration. It is not recommended to disable this option as this
+will make the device non-responsive for a noticeable amount of time.
 
 API reference
 **************

subsys/bluetooth/mesh/Kconfig

@@ -1584,6 +1584,34 @@ config BT_MESH_SEQ_STORE_RATE
 	  off with a value that's guaranteed to be larger than the last
 	  one used before power off.
 
+config BT_MESH_SETTINGS_WORKQ
+	bool "Store the Bluetooth mesh settings in a separate work queue"
+	default y
+	help
+	  This option enables a separate cooperative thread which is used to
+	  store Bluetooth mesh configuration. When this option is disabled,
+	  the stack's configuration is stored in the system workqueue. This
+	  means that the system workqueue will be blocked for the time needed
+	  to store the pending data. This time may significantly increase if
+	  the flash driver does the erase operation. Enabling this option
+	  allows Bluetooth mesh not to block the system workqueue, and thus
+	  process the incoming and outgoing messages while the flash driver
+	  waits for the controller to allocate the time needed to write the
+	  data and/or erase the required flash pages.
+
+if BT_MESH_SETTINGS_WORKQ
+
+config BT_MESH_SETTINGS_WORKQ_PRIO
+	int
+	default 1
+
+config BT_MESH_SETTINGS_WORKQ_STACK_SIZE
+	int "Stack size of the settings workq"
+	default 880
+	help
+	  Size of the settings workqueue stack.
+
+endif # BT_MESH_SETTINGS_WORKQ
 endif # BT_SETTINGS
 
 if BT_MESH_LOG_LEVEL_DBG

subsys/bluetooth/mesh/settings.c

@@ -40,6 +40,21 @@ LOG_MODULE_REGISTER(bt_mesh_settings);
 #define RPL_STORE_TIMEOUT (-1)
 #endif
 
+#ifdef CONFIG_BT_MESH_SETTINGS_WORKQ_PRIO
+#define SETTINGS_WORKQ_PRIO CONFIG_BT_MESH_SETTINGS_WORKQ_PRIO
+#else
+#define SETTINGS_WORKQ_PRIO 1
+#endif
+
+#ifdef CONFIG_BT_MESH_SETTINGS_WORKQ_STACK_SIZE
+#define SETTINGS_WORKQ_STACK_SIZE CONFIG_BT_MESH_SETTINGS_WORKQ_STACK_SIZE
+#else
+#define SETTINGS_WORKQ_STACK_SIZE 0
+#endif
+
+static struct k_work_q settings_work_q;
+static K_THREAD_STACK_DEFINE(settings_work_stack, SETTINGS_WORKQ_STACK_SIZE);
+
 static struct k_work_delayable pending_store;
 static ATOMIC_DEFINE(pending_flags, BT_MESH_SETTINGS_FLAG_COUNT);
 
@@ -144,9 +159,19 @@ void bt_mesh_settings_store_schedule(enum bt_mesh_settings_flag flag)
 	 * deadline.
 	 */
 	if (timeout_ms < remaining_ms) {
-		k_work_reschedule(&pending_store, K_MSEC(timeout_ms));
+		if (IS_ENABLED(CONFIG_BT_MESH_SETTINGS_WORKQ)) {
+			k_work_reschedule_for_queue(&settings_work_q, &pending_store,
+						    K_MSEC(timeout_ms));
+		} else {
+			k_work_reschedule(&pending_store, K_MSEC(timeout_ms));
+		}
 	} else {
-		k_work_schedule(&pending_store, K_MSEC(timeout_ms));
+		if (IS_ENABLED(CONFIG_BT_MESH_SETTINGS_WORKQ)) {
+			k_work_schedule_for_queue(&settings_work_q, &pending_store,
+						  K_MSEC(timeout_ms));
+		} else {
+			k_work_schedule(&pending_store, K_MSEC(timeout_ms));
+		}
 	}
 }
 
@@ -240,6 +265,13 @@ static void store_pending(struct k_work *work)
 
 void bt_mesh_settings_init(void)
 {
+	if (IS_ENABLED(CONFIG_BT_MESH_SETTINGS_WORKQ)) {
+		k_work_queue_start(&settings_work_q, settings_work_stack,
+				   K_THREAD_STACK_SIZEOF(settings_work_stack),
+				   K_PRIO_COOP(SETTINGS_WORKQ_PRIO), NULL);
+		k_thread_name_set(&settings_work_q.thread, "BT Mesh settings workq");
+	}
+
 	k_work_init_delayable(&pending_store, store_pending);
 }