Commit graph

522 commits

Author SHA1 Message Date
Daniel Leung
b2784c9145 kernel: mm: rename Z_KERNEL_VIRT_* to K_MEM_KERNEL_VIRT_*
Renames:
  Z_KERNEL_VIRT_START to K_MEM_KERNEL_VIRT_START
  Z_KERNEL_VIRT_SIZE to K_MEM_KERNEL_VIRT_SIZE
  Z_KERNEL_VIRT_END to K_MEM_KERNEL_VIRT_END

This is part of a series to move memory management related
stuff from Z_ namespace into its own namespace.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-06-12 21:13:26 -04:00
Daniel Leung
03eded1ed6 kernel: mm: rename Z_VIRT_RAM_* to K_MEM_VIRT_*
Renames:
  Z_VIRT_RAM_START to K_MEM_VIRT_RAM_START
  Z_VIRT_RAM_SIZE to K_MEM_VIRT_RAM_SIZE
  Z_VIRT_RAM_END to K_MEM_VIRT_RAM_END

This is part of a series to move memory management related
stuff from Z_ namespace into its own namespace.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-06-12 21:13:26 -04:00
Daniel Leung
3fd66de508 kernel: mm: rename Z_PHYS_RAM_* to K_MEM_PHYS_*
Renames:
  Z_PHYS_RAM_START to K_MEM_PHYS_RAM_START
  Z_PHYS_RAM_SIZE to K_MEM_PHYS_RAM_SIZE
  Z_PHYS_RAM_END to K_MEM_PHYS_RAM_END

This is part of a series to move memory management related
stuff from Z_ namespace into its own namespace.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-06-12 21:13:26 -04:00
Daniel Leung
b6f951b99e kernel: mm: rename Z_VM_OFFSET to K_MEM_VM_OFFSET
This is part of a series to move memory management related
stuff from the Z_ namespace into its own namespace.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-06-12 21:13:26 -04:00
Daniel Leung
def364ab08 kernel: mm: rename Z_BOOT_* to K_MEM_BOOT_*
Rename Z_BOOT_VIRT_TO_PHYS() and Z_BOOT_PHYS_TO_VIRT() to
K_MEM_BOOT_VIRT_TO_PHYS() and K_MEM_BOOT_PHYS_TO_VIRT()
respectively. This is part of a series to move memory management
functions away from the Z_ namespace and into its own namespace.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-06-12 21:13:26 -04:00
frei tycho
4c2938a295 kernel: added missing parenthesis
- added missing parenthesis around macro argument expansion

Signed-off-by: frei tycho <tfrei@baumer.com>
2024-06-07 12:59:46 +02:00
Peter Mitsis
d8a4c8a90c kernel: Add CONFIG_IPI_OPTIMIZE
The CONFIG_IPI_OPTIMIZE configuration option allows for the flagging
and subsequent signaling of IPIs to be optimized.

It does this by making each bit in the kernel's pending_ipi field
a flag that indicates whether the corresponding CPU might need an IPI
to trigger the scheduling of a new thread on that CPU.

When a new thread is made ready, we compare that thread against each
of the threads currently executing on the other CPUs. If there is a
chance that that thread should preempt the thread on the other CPU
then we flag that an IPI is needed for that CPU. That is, a clear bit
indicates that the CPU absolutely will not need to reschedule, while a
set bit indicates that the target CPU must make that determination for
itself.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2024-06-04 22:35:54 -04:00
Yong Cong Sin
6a3cb93d88 arch: remove the use of z_arch_esf_t completely from internal
Created `GEN_OFFSET_STRUCT` & `GEN_NAMED_OFFSET_STRUCT` that
works for `struct`, and remove the use of `z_arch_esf_t`
completely.

Signed-off-by: Yong Cong Sin <ycsin@meta.com>
2024-06-04 14:02:51 -05:00
Yong Cong Sin
e54b27b967 arch: define struct arch_esf and deprecate z_arch_esf_t
Make `struct arch_esf` compulsory for all architectures by
declaring it in the `arch_interface.h` header.

After this commit, the named struct `z_arch_esf_t` is only used
internally to generate offsets, and is slated to be removed
from the `arch_interface.h` header in the future.

Signed-off-by: Yong Cong Sin <ycsin@meta.com>
2024-06-04 14:02:51 -05:00
Yong Cong Sin
bbe5e1e6eb build: namespace the generated headers with zephyr/
Namespaced the generated headers with `zephyr` to prevent
potential conflict with other headers.

Introduce a temporary Kconfig `LEGACY_GENERATED_INCLUDE_PATH`
that is enabled by default. This allows the developers to
continue the use of the old include paths for the time being
until it is deprecated and eventually removed. The Kconfig will
generate a build-time warning message, similar to the
`CONFIG_TIMER_RANDOM_GENERATOR`.

Updated the includes path of in-tree sources accordingly.

Most of the changes here are scripted, check the PR for more
info.

Signed-off-by: Yong Cong Sin <ycsin@meta.com>
2024-05-28 22:03:55 +02:00
Flavio Ceolin
4d85f3d91c pm: Deprecate z_pm_save_idle_exit
Deprecate z_pm_save_idle_exit and promote pm_system_resume.

Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
2024-05-27 02:10:03 -07:00
Nicolas Pitre
b34e94f362 kernel: demand_paging: fix arch_page_location_get() documentation
Symbols from enum arch_page_location are defined as
ARCH_PAGE_LOCATION_* and not ARCH_PAGE_FAULT_*.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2024-05-24 07:47:49 -04:00
Nicolas Pitre
e9a47d932c kernel: mmu: shrink and align struct z_page_frame
The struct z_page_frame is marked __packed to avoid extra padding as
such padding may represent significant memory waste when lots of page
frames are used. However this is a bad strategy.

The code contained this somewhat dubious comment and code in
free_page_frame_list_put():

	/* The structure is packed, which ensures that this is true */
	void *node = pf;
	sys_slist_append(&free_page_frame_list, node);

This is bad for many reasons:

- type checking is completely bypassed;

- if the sys_snode_t node member is no longer located at the front of
  struct z_page_frame then the code will still compile and possibly run
  but be broken with memory corruption as a likely outcome;

- the sys_slist_append() code is completely unaware of the packed
  attribute which breaks architectures with alignment restrictions.

Let's improve code efficiency as well as memory usage by removing the
packed attribute and manually packing the flags in the unused virtual
address bits. This way the page frame array remains naturally aligned,
data access becomes optimal and the actual array size gets even smaller.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2024-05-13 16:04:40 -04:00
Nicolas Pitre
57305971d1 kernel: mmu: abstract access to page frame flags and address
Introduce z_page_frame_set() and z_page_frame_clear() to manipulate
flags. Obtain the virtual address using the existing
z_page_frame_to_virt(). This will make changes to the page frame
structure easier.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2024-05-13 16:04:40 -04:00
Hess Nathan
6d417d52c2 coding guidelines: comply with MISRA Rule 12.1.
added parentheses verifying lack of ambiguities

Signed-off-by: Hess Nathan <nhess@baumer.com>
2024-05-12 13:37:27 -04:00
Hess Nathan
e05c4a8786 coding guidelines: comply with MISRA Rule 11.8
- modified parameter types to receive a const pointer when a
  non-const pointer is not needed

- avoided redundant casts

Signed-off-by: Hess Nathan <nhess@baumer.com>
2024-05-10 14:45:14 -05:00
Andy Ross
fd340ebf31 sched: Optimize dummy thread usage on SMP
Nicolas Pitre points out that since these thread structs are just
dummies for the context swtiching, they can be presumed to be "write
only" and thus there's no point in having one per CPU, everyone can
share the same one.

The only gotcha is that we never really documented (nor really have a
place to document) that rule, so it's not theoretically impossible for
an architecture to read back what it might have written underneath
arch_switch().  Leave this in a separate commit for bisection
purposes, but the risk seems very low.

Signed-off-by: Andy Ross <andyross@google.com>
2024-05-02 13:55:03 -04:00
Andy Ross
f0fd54cb31 kernel/sched: Fix free-memory write when ISRs abort _current
After a k_thread_abort(), the resulting thread struct is documented as
unused/free memory that may be re-used (for example, to respawn a new
thread).

But in the special case of aborting the current thread from within an
ISR, that wasn't quite happening.  The scheduler cleanup would
complete, but the architecture layer would still try to context switch
away from the aborted thread on exit, and that can include writes to
the now-reused thread struct!  The specifics will depend on
architecture (some do a full context save on entry, most don't), but
in the case of USE_SWITCH=y it will at the very least write the
switch_handle field.

Fix this simply, with a per-cpu "switch dummy" thread struct for use
as a target for context switches like this.  There is some non-trivial
memory cost to that; thread structs on many architectures are large.

Pleasingly, this also addresses a known deadlock on SMP: because the
"spin in ISR" step now happens as the very last stage of
k_thread_abort() handling, the existing scheduler lock works to
serialize calls such that it's impossible for a cycle of threads to
independently decide to spin on each other: at least one will see
itself as "already aborting" and break the cycle.

Fixes #64646

Signed-off-by: Andy Ross <andyross@google.com>
2024-05-02 13:55:03 -04:00
Hess Nathan
527e712448 coding guidelines: comply with MISRA Rule 20.9
- avoid to use undefined macros in #if expressions

Signed-off-by: Hess Nathan <nhess@baumer.com>
2024-05-01 19:48:19 +01:00
Hess Nathan
32af724fbb coding guidelines: comply with MISRA C:2012 Rule 11.2
avoid convert pointers to incomplete type using the pointer to first item

Signed-off-by: Hess Nathan <nhess@baumer.com>
2024-05-01 10:53:20 -04:00
Alberto Escolar Piedras
ea26bcf8d3 Revert "kernel/sched: Fix free-memory write when ISRs abort _current"
This reverts commit 61c70626a5.

This PR introduced 2 regressions in main CI:
71977 & 71978
Let's revert it by now to get main's CI passing again.

Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
2024-04-26 10:10:24 +00:00
Alberto Escolar Piedras
c9ec937d71 Revert "sched: Optimize dummy thread usage on SMP"
This reverts commit 20611f13ca.

This PR introduced 2 regressions in main CI:
71977 & 71978
Let's revert it by now to get main's CI passing again.

Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
2024-04-26 10:10:24 +00:00
Andy Ross
20611f13ca sched: Optimize dummy thread usage on SMP
Nicolas Pitre points out that since these thread structs are just
dummies for the context swtiching, they can be presumed to be "write
only" and thus there's no point in having one per CPU, everyone can
share the same one.

The only gotcha is that we never really documented (nor really have a
place to document) that rule, so it's not theoretically impossible for
an architecture to read back what it might have written underneath
arch_switch().  Leave this in a separate commit for bisection
purposes, but the risk seems very low.

Signed-off-by: Andy Ross <andyross@google.com>
2024-04-25 15:12:02 +02:00
Andy Ross
61c70626a5 kernel/sched: Fix free-memory write when ISRs abort _current
After a k_thread_abort(), the resulting thread struct is documented as
unused/free memory that may be re-used (for example, to respawn a new
thread).

But in the special case of aborting the current thread from within an
ISR, that wasn't quite happening.  The scheduler cleanup would
complete, but the architecture layer would still try to context switch
away from the aborted thread on exit, and that can include writes to
the now-reused thread struct!  The specifics will depend on
architecture (some do a full context save on entry, most don't), but
in the case of USE_SWITCH=y it will at the very least write the
switch_handle field.

Fix this simply, with a per-cpu "switch dummy" thread struct for use
as a target for context switches like this.  There is some non-trivial
memory cost to that; thread structs on many architectures are large.

Pleasingly, this also addresses a known deadlock on SMP: because the
"spin in ISR" step now happens as the very last stage of
k_thread_abort() handling, the existing scheduler lock works to
serialize calls such that it's impossible for a cycle of threads to
independently decide to spin on each other: at least one will see
itself as "already aborting" and break the cycle.

Fixes #64646

Signed-off-by: Andy Ross <andyross@google.com>
2024-04-25 15:12:02 +02:00
Anas Nashif
4593f0d71c kernel: priority queues: declare as static inlines
After the move to C files we got some drop in the performance when
running latency_measure. This patch declares the priority queue
functions as static inlines with minor optimizations.

The result for one metric (on qemu):

3.6 and before the anything was changed:

  Get data from LIFO (w/ ctx switch): 13087 ns

after original change (46484da502):

  Get data from LIFO (w/ ctx switch): 13663 ns

with this change:

  Get data from LIFO (w/ ctx switch): 12543 ns

So overall, a net gain of ~ 500ns that can be seen across the board on many
of the metrics.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-04-22 16:40:11 -04:00
Daniel Leung
378131c266 kernel: add options to cleanup after aborting current thread
This adds the mechanism to do cleanup after k_thread_abort()
is called with the current thread. This is mainly used for
cleaning up things when the thread cannot be running, e.g.,
cleanup the thread stack.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-04-10 07:44:27 -04:00
Anas Nashif
20b2c98add kernel: move nothread support to own file
Do not build threading support when CONFIG_MULTITHREADING=n is set and
move needed calls to a new file with the changes needed instead of the
ifdef party in sched.c

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-04-06 14:22:08 +03:00
Anas Nashif
8a88cd4805 kernel: thread: move thread states to header
Move state string defines into thread header.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-04-01 18:47:36 -04:00
Anas Nashif
5c170c7046 kernel: thread: rename is_preempt
Trivila rename to thread_is_preempt.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-04-01 18:47:36 -04:00
Anas Nashif
17c874f4fc kernel: thread: rename is_metairq
Trivial rename of is_metairq to thread_is_metairq.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-04-01 18:47:36 -04:00
Anas Nashif
37df485463 kernel: split timeslicing/ipi code out of sched.c
Move both timeslicing and IPI code to own files.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-04-01 18:47:36 -04:00
Anas Nashif
31bc210bbc kernel: sched: remove unused prototype: z_is_thread_time_slicing
A prototype not used anywhere.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-04-01 18:47:36 -04:00
Anas Nashif
ebb503ff7b kernel: move thread related helper function kthread.h
Move some helper functions to inernal kthread.h, to offload crowded
sched.c

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-04-01 18:47:36 -04:00
Simon Hein
bcd1d19322 kernel: add closing comments to config endifs
Add a closing comment to the endif with the configuration
information to which the endif belongs too.
To make the code more clearer if the configs need adaptions.

Signed-off-by: Simon Hein <Shein@baumer.com>
2024-03-25 18:03:31 -04:00
Daniel Leung
3664ed64c3 arch: move arch_interface.h under zephyr/arch
arch_interface.h is for architecture and should not be
under sys/. So move it under include/zephyr/arch/.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-03-25 09:58:35 +00:00
TaiJu Wu
1f5f0cf838 sched: Remove multi-level queue priority limit
Modified bitmask to  bitmask array, it can make multilevel queue remove
32 bit prioriry limit.

We can scan bitmask array to find which queue have ready thread.

Only need the number of queues as priority because the priority
is checked on create_thread.

Signed-off-by: TaiJu Wu <tjwu1217@gmail.com>
2024-03-12 19:37:40 -04:00
Peter Mitsis
9f7695dda0 kernel: Remove unused z_pend_curr_irqlock()
The routine z_pend_curr_irqlock() is no longer used anywhere.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2024-03-07 11:51:06 -05:00
Anas Nashif
0d8da5ff93 kernel: rename scheduler spinlock variable and make it private
rename sched_spinlock to _sched_spinglock to maintain it is privacy and
to avoid any misuse.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-03-06 19:27:28 -05:00
Anas Nashif
868f099d61 kernel: sched: z_set_prio -> z_thread_prio_set
Rename private function to make it clear what priority we are setting
and to be consistent across the code.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-03-06 19:27:28 -05:00
Anas Nashif
6c003bdbcf kernel: remove unused code in headers
List of functions defined in headers and not being used anywhere.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-03-06 19:27:28 -05:00
Anas Nashif
9e83413542 kernel: split thread monitor
Move thread monitor related functions, not enabled in most cases outside
of thread.c and cleanup headers.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-03-06 19:27:28 -05:00
Anas Nashif
5e591c38f1 kernel: do not export z_thread_priority_set
This function is only being used by a test, so instead of reimplementing
a syscall in the test, provide a Kconfig option to provide the
functionality that only works with tests and remove some of the
duplication and extra code.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-03-06 19:27:28 -05:00
Anas Nashif
3ca50f5060 kernel: move z_init_static_threads to where it is being used
Move out of thread and put directly in init.c where it is being used.
Also remove definition from kernel.h, this is an internal function and
should not be in a public header.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-03-06 19:27:28 -05:00
Anas Nashif
8791012ed1 kernel: move essential flag related routines out
The functions to manipulate the essential flag indeed operate on
threads, but they are misplaced in the thread implementation file. Put
them alongside other routines setting other thread flags and cleanup
headers a bit.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-03-06 19:27:28 -05:00
Anas Nashif
46484da502 kernel: move priority queue handling to own file/header
clean up headers under include/ and move handling of priority queue to
own file/header.
No need for the header  include/zephyr/kernel/internal/sched_priq.h
anymore. Move the relevant structures where they are being used in
kernel_structs.h.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-03-02 15:06:45 +01:00
Peter Mitsis
ee9c44fee6 linker: Round TLS sizes up in linker script
Instead of rounding up both __tdata_size and __tbss_size at runtime,
perform the calculation when the image is built.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2024-02-25 22:28:56 -05:00
Nguyen Minh Thien
8188be57d3 kernel: fix spelling errors
Fix spelling errors found in comment of the kernel source code.

Signed-off-by: Nguyen Minh Thien <nguyenmthien@live.com>
2024-02-25 20:53:37 -05:00
Krzysztof Chruściński
25173f71cd pm: device_runtime: Extend with synchronous runtime PM
In many cases suspending or resuming of a device is limited to
just a few register writes. Current solution assumes that those
operations may be blocking, asynchronous and take a lot of time.
Due to this assumption runtime PM API cannot be effectively used
from the interrupt context. Zephyr has few driver APIs which
can be used from an interrupt context and now use of runtime PM
is limited in those cases.

Patch introduces a new type of PM device - synchronous PM. If
device is specified as capable of synchronous PM operations then
device runtime getting and putting is executed in the critical
section. In that case, runtime API can be used from an interrupt
context. Additionally, this approach reduces RAM needed for
PM device (104 -> 20 bytes of RAM on ARM Cortex-M).

Signed-off-by: Krzysztof Chruściński <krzysztof.chruscinski@nordicsemi.no>
2024-02-01 15:03:42 +01:00
Daniel Leung
2cdd44801e kernel: move z_init_cpu to private kernel headers
z_init_cpu() is a private kernel API so move it under
kernel/include.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-01-17 11:57:20 -05:00
Daniel Leung
a7dccc4475 kernel: mmu: mitigate range check overflow issue
It is possible that address + size will overflow the available
address space and the pointer wraps around back to zero. Some
of these have been fixed in previous commits. This fixes
the remaining ones with regard to Z_PHYS_RAM_START/_END,
and Z_VIRT_RAM_START/_END.

Fixes #65542

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2023-12-20 11:37:17 -05:00