Bluetooth: Mesh: Use separate workq for storing mesh settings

Currently mesh settings are stored in the system workqueue context.
Most of other stack functionality, such that advertisements (incl
relay), loopback, transport sar, beacons transmission, etc. is also
processed in the system workqueue context. When a massive amount of
data needs to be stored and in particularly when page erase needs to
be triggered by GC of NVS subsystem to allocate flash pages, the
execution of the stack (and other functionality that uses the system
workqueue) will be blocked until storing is finished. For example,
right after the provisioning of a erased device, a node may not be
responsive for up to 400ms before it can continue sending messages.
The waiting time may increase if there is a GATT connection in the
mean time.

When write or erase operation is triggered, the flash driver waits for
Bluetooth controller to allocate a time needed to perform the operation.
During the whole operation, the context from which the operation was
triggered is put to sleep. This allows other threads to run until
Bluetooth controller finds the time for the flash driver. In other words,
every settings_save_one or settings_delete should be considered as
rescheduling points.

Considering this, Bluetooth mesh can use another thread to store its
settings, thus releasing the system workqueue for other tasks including
the operation of the stack itself.

The consistency of the data to be stored is guaranteed by the current
implementation where the data is copied to another struct before calling
settings_save_one. The pending flag of a particular module is dropped in
settings.c before starting to store the corresponding data. Thus, if
during the sleep the node receives a message that triggers a change in a
module which data is currently being stored, the pending flag will be
restored and the new change will be stored eventually.

Having this option enabled including with the partial erase, will make
the node more responsive in the described situations.

Signed-off-by: Pavel Vasilyev <pavel.vasilyev@nordicsemi.no>

subsys/bluetooth/mesh/Kconfig

@@ -1584,6 +1584,34 @@ config BT_MESH_SEQ_STORE_RATE
 	  off with a value that's guaranteed to be larger than the last
 	  one used before power off.
 
+config BT_MESH_SETTINGS_WORKQ
+	bool "Store the Bluetooth mesh settings in a separate work queue"
+	default y
+	help
+	  This option enables a separate cooperative thread which is used to
+	  store Bluetooth mesh configuration. When this option is disabled,
+	  the stack's configuration is stored in the system workqueue. This
+	  means that the system workqueue will be blocked for the time needed
+	  to store the pending data. This time may significantly increase if
+	  the flash driver does the erase operation. Enabling this option
+	  allows Bluetooth mesh not to block the system workqueue, and thus
+	  process the incoming and outgoing messages while the flash driver
+	  waits for the controller to allocate the time needed to write the
+	  data and/or erase the required flash pages.
+
+if BT_MESH_SETTINGS_WORKQ
+
+config BT_MESH_SETTINGS_WORKQ_PRIO
+	int
+	default 1
+
+config BT_MESH_SETTINGS_WORKQ_STACK_SIZE
+	int "Stack size of the settings workq"
+	default 880
+	help
+	  Size of the settings workqueue stack.
+
+endif # BT_MESH_SETTINGS_WORKQ
 endif # BT_SETTINGS
 
 if BT_MESH_LOG_LEVEL_DBG

subsys/bluetooth/mesh/settings.c

@@ -40,6 +40,21 @@ LOG_MODULE_REGISTER(bt_mesh_settings);
 #define RPL_STORE_TIMEOUT (-1)
 #endif
 
+#ifdef CONFIG_BT_MESH_SETTINGS_WORKQ_PRIO
+#define SETTINGS_WORKQ_PRIO CONFIG_BT_MESH_SETTINGS_WORKQ_PRIO
+#else
+#define SETTINGS_WORKQ_PRIO 1
+#endif
+
+#ifdef CONFIG_BT_MESH_SETTINGS_WORKQ_STACK_SIZE
+#define SETTINGS_WORKQ_STACK_SIZE CONFIG_BT_MESH_SETTINGS_WORKQ_STACK_SIZE
+#else
+#define SETTINGS_WORKQ_STACK_SIZE 0
+#endif
+
+static struct k_work_q settings_work_q;
+static K_THREAD_STACK_DEFINE(settings_work_stack, SETTINGS_WORKQ_STACK_SIZE);
+
 static struct k_work_delayable pending_store;
 static ATOMIC_DEFINE(pending_flags, BT_MESH_SETTINGS_FLAG_COUNT);
 
@@ -144,9 +159,19 @@ void bt_mesh_settings_store_schedule(enum bt_mesh_settings_flag flag)
 	 * deadline.
 	 */
 	if (timeout_ms < remaining_ms) {
-		k_work_reschedule(&pending_store, K_MSEC(timeout_ms));
+		if (IS_ENABLED(CONFIG_BT_MESH_SETTINGS_WORKQ)) {
+			k_work_reschedule_for_queue(&settings_work_q, &pending_store,
+						    K_MSEC(timeout_ms));
+		} else {
+			k_work_reschedule(&pending_store, K_MSEC(timeout_ms));
+		}
 	} else {
-		k_work_schedule(&pending_store, K_MSEC(timeout_ms));
+		if (IS_ENABLED(CONFIG_BT_MESH_SETTINGS_WORKQ)) {
+			k_work_schedule_for_queue(&settings_work_q, &pending_store,
+						  K_MSEC(timeout_ms));
+		} else {
+			k_work_schedule(&pending_store, K_MSEC(timeout_ms));
+		}
 	}
 }
 
@@ -240,6 +265,13 @@ static void store_pending(struct k_work *work)
 
 void bt_mesh_settings_init(void)
 {
+	if (IS_ENABLED(CONFIG_BT_MESH_SETTINGS_WORKQ)) {
+		k_work_queue_start(&settings_work_q, settings_work_stack,
+				   K_THREAD_STACK_SIZEOF(settings_work_stack),
+				   K_PRIO_COOP(SETTINGS_WORKQ_PRIO), NULL);
+		k_thread_name_set(&settings_work_q.thread, "BT Mesh settings workq");
+	}
+
 	k_work_init_delayable(&pending_store, store_pending);
 }