This commit updates all deprecated `K_KERNEL_PINNED_STACK_ARRAY_EXTERN`
macro usages to use the `K_KERNEL_PINNED_STACK_ARRAY_DECLARE` macro
instead.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
Files including <zephyr/kernel.h> do not have to include
<zephyr/zephyr.h>, a shim to <zephyr/kernel.h>.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Rename the symbols used to denote the locations of the global
constructor lists and modify the Zephyr start-up code accordingly.
On POSIX systems this ensures that the native libc init code won't
find any constructors to run before Zephyr loads.
Fixes#39347, #36858
Signed-off-by: David Palchak <palchak@google.com>
Use a new environment variable,
ZEPHYR_TOOLCHAIN_SUPPORTS_THREAD_LOCAL_STORAGE, to set the value for
TOOLCHAIN_SUPPORTS_THREAD_LOCAL_STORAGE instead of setting it to 'n' for
all non-Zephyr toolchains. In particular, the Debian arm-none-eabi
toolchain has TLS support and with this option, can be used to build
Zephyr with thread local variables.
Signed-off-by: Keith Packard <keithp@keithp.com>
Documentation specifies that aborting/terminating/exiting essential
threads is a system panic condition, but we didn't actually implement
that and allowed it as for other threads. At least one app wants to
exploit this documented behavior as a "watchdog" kind of condition,
and that seems reasonable. Do what we say we're supposed to do.
This also includes a small fix to a test, which seemed like it was
written to exercise exactly this condition. Except that it failed to
detect whether or not a system fatal error was actually signaled and
was (incorrectly) indicating "success". Check that we actually enter
the handler.
Fixes#45545
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
The function k_thread_runtime_stats_all_get() now populates the
current_cycles field in the thread runtime stats structure.
Resets the number of cycles in the CPU's current usage window once
the idle thread is scheduled.
Fixes the average_cycles calcuation.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
For a library which already provides a multi-thread aware errno, use
that instead of creating our own internal value.
Signed-off-by: Keith Packard <keithp@keithp.com>
This adds the internal function z_work_submit_to_queue(), which
submits the work item to the queue but doesn't force the thread to yield,
compared to the public function k_work_submit_to_queue().
When called from poll.c in the context of k_work_poll events, it ensures
that the thread does not yield in the context of the spinlock of object
that became available.
Fixes#45267
Signed-off-by: Lucas Dietrich <ld.adecy@gmail.com>
In order to bring consistency in-tree, migrate all kernel code to the
new prefix <zephyr/...>. Note that the conversion has been scripted,
refer to zephyrproject-rtos#45388 for more details.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Implements a function that application and driver code can use to check
whether it is valid to yield (or block) in the current context. This
check is required for functions that can feasibly be run from multiple
contexts. The primary intended use case is power management transition
functions, which can be run by application code explicitly or
automatically in the idle thread by system PM.
Signed-off-by: Jordan Yates <jordan.yates@data61.csiro.au>
This adds lazy floating point context switching. On svc/irq entrance,
the VFP is disabled and a pointer to the exception stack frame is saved
away. If the esf pointer is still valid on exception exit, then no
other context used the VFP so the context is still valid and nothing
needs to be restored. If the esf pointer is NULL on exception exit,
then some other context used the VFP and the floating point context is
restored from the esf.
The undefined instruction handler is responsible for saving away the
floating point context if needed. If the handler is in the first
irq/svc context and the current thread uses the VFP, then the float
context needs to be saved. Also, if the handler is in a nested context
and the previous context was using the FVP, save the float context.
Signed-off-by: Bradley Bolen <bbolen@lexmark.com>
Do not allow changing the CPU which a thread is pinned when it is
already being executed. This allows further optimizations in some
platforms with incoherent memory since we can safely assume that the
thread will run in the same CPU and avoid invalidate / flush the
cache during context switches.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
The k_timer utility was written to assume that the kernel timeout
handler would never be delayed by more than a tick, so it can naively
reschedule the next interrupt with a simple delay.
Unfortunately real platforms have glitchy hardware and high tick
rates, and on intel_adsp we're seeing this promise being broken in
some circumstances.
It's probably not a good idea to try to plumb the timer driver
interface up into the IPC layer to do this correction, but thankfully
the existing absolute timeout API provides the tools we need (though
it does require that CONFIG_TIMEOUT_64BIT be enabled).
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
The original design intent with arch_sched_ipi() was that
interprocessor interrupts were fast and easily sent, so to reduce
latency the scheduler should notify other CPUs synchronously when
scheduler state changes.
This tends to result in "storms" of IPIs in some use cases, though.
For example, SOF will enumerate over all cores doing a k_sem_give() to
notify a worker thread pinned to each, each call causing a separate
IPI. Add to that the fact that unlike x86's IO-APIC, the intel_adsp
architecture has targeted/non-broadcast IPIs that need to be repeated
for each core, and suddenly we have an O(N^2) scaling problem in the
number of CPUs.
Instead, batch the "pending" IPIs and send them only at known
scheduling points (end-of-interrupt and swap). This semantically
matches the locations where application code will "expect" to see
other threads run, so arguably is a better choice anyway.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
The work queue has a semi/non-standard reschedule point implemented
using k_yield(), with a check to see if the current thread is
preemptible. Just call z_reschedule_unlocked(), it has this check
internally and is the intended API for this.
Really, this is only a half fix. Ideally the schedule point and the
lock release should be atomic[1] via the more idiomatic
z_reschedule(). But that would take some surgery, so let's go with
the simpler cleanup first.
This also avoids having to duplicate logic that gets added to
reschedule points by an upcoming patch.
[1] So that they represent a condition variable and don't race at the
end. In this case the race is present but benign, since the only thing
we really want to know is that the queue thread gets a chance to run.
The only cost is an occasional duplicated/needless context switch if
two threads are racing on a submit.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Removes an unnecessary schedule lock/unlock pair from k_mutex_unlock().
Rationale: Given that only the current thread (which would also be the
mutex owner) will be able to modify the mutex object AND that a
recursive unlock ought never trigger any reschedule (as it does not
touch the pend queue), then performing a schedule lock is not needed
prior to testing for a recursive unlock.
Furthermore, even if it is not a recursive unlock, then a schedule lock
is superfluous as the existing spinlock provides sufficient protection.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
When threads are in more than one state at a time, k_thread_state_str()
returns a string that lists each of its states delimited by a '+'.
This in turn necessitates a change to the API that includes both a
pointer to the buffer to use for the string and the size of the buffer.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Add an API that clears cpu mask from a thread and sets it to a specific
CPU.
This is the equivelent of:
k_thread_cpu_mask_clear(&thread);
k_thread_cpu_mask_enable(&thread, cpu_idx);
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Instead of resizing all devices handles, we just resize devices that are
power domains. This means that a power domain has to be declared as
compatbile with "power-domain" in device tree node.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Add API to add devices to a power domain in runtime. The number of
devices that can be added is defined in build time.
The script gen_handles.py will check the number defined in
`CONFIG_PM_DEVICE_POWER_DOMAIN_DYNAMIC` to resize the handles vector,
adding empty slots in the supported sector to be used later.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Commit b1182bf83b ("kernel/timeout: Serialize handler callbacks on
SMP") introduced an important fix to timeout handling on
multiprocessor systems, but it did it in a clumsy way by holding a
spinlock across the entire timeout process on all cores (everything
would have to spin until one core finished the list). The lock also
delays any nested interrupts that might otherwise be delivered, which
breaks our nested_irq_offload case on xtensa+SMP (where contra x86,
the "synchronous" interrupt is sensitive to mask state).
Doing this right turns out not to be so hard: take the timeout lock,
check to see if someone is already iterating
(i.e. "announce_remaining" is non-zero), and if so just increment the
ticks to announce and exit. The original cpu will then complete the
full timeout list without blocking any others longer than needed to
check the timeout state.
Fixes#44758
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
On multiprocessor systems, it's routine to enter sys_clock_announce()
in parallel (the driver will generally announce zero ticks on all but
one cpu).
When that happens, each call will independently enter the loop over
the timeout list. The access is correctly synchronized, so the list
handling is correct. But the lock is RELEASED around the invocation
of the callback, which means that the individual callbacks may
interleave between cpus. That means that individual
application-provided callbacks may be executed in parallel, which to
the app is indistinguishable from "out of order".
That's surprising and error-prone. Don't do it. Place a secondary
outer spinlock around the announce loop (but not the timeslicing
handling) to correctly serialize the timeout handling on a single cpu.
(It should be noted that this was discovered not because of a timeout
callback race, but because the resulting simultaneous calls to
sys_clock_set_timeout from separate cores seems to cause extremely
high latency excursions on intel_adsp hardware using the cavs_timer
driver. That hardware issue is still poorly understood, but this fix
is desirable regardless.)
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
The idle thread got an index suffix in #23536 to make it easier to
identify different idle threads on different cores. This looks out of
place on single-core devices when the idle thread is listed next to
other kernel threads, such as main.
Remove the idle thread index on single-core platforms, and replace all
references to this format in tests and documentation.
Signed-off-by: Trond Einar Snekvik <Trond.Einar.Snekvik@nordicsemi.no>
This is an attempt at formally distinguishing and supporting the case
described in 40795 where an architecture doesn't preserve/restore the
complete thread state upon entering/exiting interrupt exception state.
This is mainly about promoting the current behavior from the accepted
workaround to a formal API specification. This workaround is currently
used on ARM64 but RISC-V requires it too.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
A reference to the entropy device can be obtained at compile time, so
avoid using device_get_binding().
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
There is an API to get an specific number of random bytes. There is
no need to re-implement this logic here.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Zephyr's timeslice implementation has always been somewhat primitive.
You get a global timeslice that applies broadly to the whole bottom of
the priority space, with no ability (beyond that one priority
threshold) to tune it to work on certain threads, etc...
This adds an (optionally configurable) API that allows timeslicing to
be controlled on a per-thread basis: any thread at any priority can be
set to timeslice, for a configurable per-thread slice time, and at the
end of its slice a callback can be provided that can take action.
This allows the application to implement things like responsiveness
heuristics, "fair" scheduling algorithms, etc... without requiring
that facility in the core kernel.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
According to Kconfig guidelines, boolean prompts must not start with
"Enable...". The following command has been used to automate the changes
in this patch:
sed -i "s/bool \"[Ee]nables\? \(\w\)/bool \"\U\1/g" **/Kconfig*
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Things had gotten a little tangled in there so let's do some cleanup.
Remove the distressingly-special-purpose z_reinit_idle_thread() hook
(which existed to support secondary core bringup when
SMP_BOOT_DELAY=y), and just fold that into a generic z_init_cpu(),
which we can call in obvious and symmetric ways from main
initialization, z_smp_init(), and z_smp_start_cpu() (the now-official
programmatic hook for starting cpus).
Remove the "#if CONFIG_MP_NUM_CPUS > 1" exclusions. These weren't
saving any code size and were propagating themselves into platform
layers trying to avoid build failures.
There are some "special" APIs added for SOF which need to go away in
favor of the newer/generic z_smp_start_cpu(). Collect them in one
place and put them under a "#ifdef CONFIG_SOF" to prevent them from
being used in Zephyr apps.
Move some function declarations that didn't have homes into
<kernel/thread.h>.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
This adds a LOG_DBG() line for z_phys_unmap which mirrors
what is in z_phys_map(). This also fixes a warning from
Clang about a variable being set but never used (addr_offset).
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Commit 678b76e4b0 ("kernel/init.c: allow for memset/memcpy
alternatives during early boot") and commit da28829b64 ("kernel:
zero the bss section of OCM memory at boot time") were created
independently and missed changes from each other.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
The x86 and xtensa implementations of irq_offload() invoke synchronous
interrupts on the local CPU, and are therefore safe to use from within
an interrupt context. This is a cheap and portable way to exercise
nested interrupts, which are otherwise highly platform-dependent to
test. Add a kconfig to signal the capability.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Zeroing the BSS and copying data to RAM with regular memset/memcpy may
cause problems when those functions are assuming a fully initialized
system for their optimizations to work e.g. some instructions require
an active MMU, but turning the MMU on needs the .bss section to be
cleared first, etc.
Commit c5b898743a ("aarch64: Fix alignment fault on z_bss_zero()")
provides a detailed explanation of such a case.
Replacing z_bss_zero() with an architecture specific one is problematic
as the former may see new sections added to it that would be missed by
the later. The same reasoning goes for z_data_copy().
Let's make maintenance much easier by providing weak versions of
memset/memcpy that can be overridden by architecture-specific safe
versions when needed.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Extracting stack usage calculation from k_thread_stack_space_get to
z_stack_space_get so it can be used also for interrupt stack.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
If a chosen entry exists for a memory area of type OCM, zero the OCM
memory's bss section at boot-time.
Signed-off-by: Immo Birnbaum <Immo.Birnbaum@weidmueller.com>
We can't simply use CLAMP to set the next timeout because
when CONFIG_SYSTEM_CLOCK_SLOPPY_IDLE is set, MAX_WAIT is
a negative number and then CLAMP will be called with
the higher boundary lower the lower boundary.
Fixes#41422
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
So that logging and "satellite" subsystems, such as tracing and object
tracking can count on kernel structs being properly initialised, such
as `_current_cpu`.
Fixes#42061.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Updates sched_cpu_update_usage() such that the CPU runtime stats
only update the its non-idle time when the current thread is not the
idle thread. This is necessary as otherwise the CPUs idle-time will
be double counted in k_thread_runtime_stats.execution_cycles.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Introduce a hidden kconfig CONFIG_KERNEL_VM_SUPPORT which
enables some kconfigs that are required for virtual memory
support. CONFIG_KERNEL_VM_BASE, CONFIG_KERNEL_VM_OFFSET,
and CONFIG_KERNEL_VM_SIZE are moved under this new kconfig
so these can be enabled independent of CONFIG_MMU.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This moves CONFIG_MMU and its children from arch/Kconfig into
kernel/Kconfig. These are used to enable kernel support of MMU
so they should be under kernel/.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
There is no need to use conditional compilation for the function
prototypes in the kernel architecture header file. So remove it.
Added bouns is that these functions can appear in documentation
without explicitly enabled in pre-defines during doc build.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Add maximum timeout used for conversion to Kconfig. Option is used
to determine which conversion algorithm to use: faster but overflowing
earlier or slower without early overflow.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
Commit 3457118 changed the sequence of z_smp_thread_init() and
smp_timer_init() in smp_init_top() subroutine, which initializes
other BPs. In some boards (up_squared, acrn_ehl_crb) it will fail to
work while SMP enabling. If the timer interrupt is enabled before the
first thread was initialized. Now change back to its original order.
Fixes#41835
Signed-off-by: Enjia Mai <enjia.mai@intel.com>
For functions returning nothing, there is no need to document
with @return, as Doxgen complains about "documented empty
return type of ...".
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This has bitrotten a bit. Early implementations had a synchronous
arch_start_cpu(), but then we started allowing that to be an async
operation. But that means that CPU start now becomes surprisingly
reentrant to the arch layer (cpu 0 can get a call to start cpu 2 while
cpu 1's initialization code is still running). That's just error
prone; we never documented the requirements cleanly (the window is
very small, but not so small to a slow simulator!).
Add an extra flag so we don't issue the next start until the last is
out of the arch layer and running in smp_init_top().
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Adds two routines to flush pipe objects:
k_pipe_flush()
- This routine flushes the entire pipe. That includes both
the pipe's buffer and all pended writers. It is equivalent
to reading everything into a giant temporary buffer which
is then discarded.
k_pipe_buffer_flush()
- This routine flushes only the pipe's buffer (if it exists).
It is equivalent to reading a maximum of "buffer size" bytes
into a temporary buffer which is then discarded.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Fixes a race condition in the k_pipe_cleanup() routine by adding
a spinlock. Additionally, internal counters are now reset after
freeing the buffer as the pipe has now become a bufferless pipe.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Resolves void pointer arithmetic build warnings in k_pipe_put() by
casting the pointer to a uint8_t pointer.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Extends the CPU usage runtime stats to track current, total, peak
and average usage (as bounded by the scheduling of the idle thread).
This permits a developer to obtain more system information if desired
to tune the system.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
When the new Kconfig option CONFIG_SCHED_THREAD_USAGE_ANALYSIS
is enabled, additional timing stats are collected during context
switches. This extra information allows a developer to obtain the
the current, longest, average and total lengths of the time that
a thread has been scheduled to execute.
A developer can in turn use this information to tune their app and/or
alter their scheduling policies.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
This commit does two things to the z_sched_thread_usage(). First,
it updates the API so that it accepts a pointer to the runtime
stats instead of simply returning the usage cycles. This gives it
the flexibility to retrieve additional statistics in the future.
Second, the runtime stats are only updated if the specified thread
is the current thread running on the current core.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Moves the CONFIG_SCHED_THREAD_USAGE block of code out of sched.c
into its own file. Not only do they employ their own private
spin lock, but it is expected that additional usage routines will be
added in the future.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
The functionality provided by device_usable_check is already provided by
device_is_ready. The (z_)device_usable_check APIs have been
re-implemented using the (z_)device_is_ready APIs and have been marked
as deprecated.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Instead of using device_usable_check() syscall, implement a new syscall
for device_is_ready that uses z_device_is_ready underneath.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Rename z_device_ready to z_device_is_ready. Function name suggests a
boolean result this way, in line with other functions (e.g.
device_is_ready).
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
The resource pool of the short-lived dummy thread "stub" may be
inherited by other threads created during system initialization. This
commit initializes this resource pool to NULL or the system pool to
ensure that a well-defined resource pool propagates to other threads
that inherit it from the dummy thread.
Fixes#41482.
Signed-off-by: Berend Ozceri <berend@recogni.com>
Move z_priq_mq_add and z_priq_mq_remove into #ifdef CONFIG_SCHED_MULTIQ
block, because they are only used with that config.
Signed-off-by: Jeremy Bettis <jbettis@google.com>
Previous commit 55350a93e9 fixing
address-of-packed-mem warnings uncovered an issue with
the alignment of dynamic kernel objects. On 64-bit platforms,
the alignment is 16 bytes instead of 4/8 bytes (as in pointer,
void *). This changes the function of mapping between kernel
object types and alignments to use the dynamic object struct
as basis for alignment instead of simply using pointers.
This also uncomments the assertion added in the previous commit
55350a93e9 so that we can keep
an eye on the alignment in the future. Note that the assertion
is moved after checking if the incoming kernel object is
dynamically allocated. Static kernel objects are not subjected
to this alignment requirement.
Fixes#41062
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Applies the 'static' keyword to the following inlined routines:
z_priq_dumb_add()
z_priq_mq_add()
z_priq_mq_remove()
As those routines are only used in one place, they no longer have
externally visible declarations.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Removed unused functions, or moved inside #ifdefs.
This allows using -Werror=unused-function on the clang compiler. Tested
by building the ChromeOS EC on all supported platforms with
-Werror=unused-functions.
Signed-off-by: Jeremy Bettis <jbettis@google.com>
The warning below appears once -Waddress-of-packed-mem is enabled:
/home/carles/src/zephyr/zephyr/kernel/userspace.c: In function
'unref_check':
/home/carles/src/zephyr/zephyr/kernel/userspace.c:471:28: warning:
converting a packed 'struct z_object' pointer (alignment 4) to a 'struct
dyn_obj' pointer (alignment 16) may result in an unaligned pointer value
[-Waddress-of-packed-mem
ber]
471 | CONTAINER_OF(ko, struct dyn_obj, kobj);
To avoid the warning, use an intermediate void * variable.
More info in #16587.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
The following warning is triggered by GCC when
-Waddress-of-packed-member is enabled:
/home/carles/src/zephyr/zephyr/kernel/mmu.c: In function
'free_page_frame_list_put':
/home/carles/src/zephyr/zephyr/kernel/mmu.c:383:42: warning: taking
address of packed member of 'struct z_page_frame' may result in an
unaligned pointer value [-Waddress-of-packed-member]
383 | sys_slist_append(&free_page_frame_list, &pf->node);
This is due to the fact that sys_snode_t node is an unpacked structure
inside a packed z_page_frame structure, so that the alignment of the
former cannot be ensured if placed inside the latter.
Given that alignment of z_page_frame is ensured by the code, silence the
compiler by going through an intermediate variable.
More info in #16587.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
Storing the state where this is the first GDB break can be done
in the main GDB stub code. There is no need to store the state
in architecture layer.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Removing the 'U' to avoid the type of num_events changed.
And make sure it is meaningful Z_SYSCALL_VERIFY micro.
Fixed#40614
Signed-off-by: NingX Zhao <ningx.zhao@intel.com>
The virtual region bitmap bitarray struct is only used within
the source file, so it can be declared static.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Adds an API to query and visit supported devices. Follows the example
set by the required devices API.
Implements #37793.
Signed-off-by: Jordan Yates <jordan.yates@data61.csiro.au>
This updates k_mem_domain_add_thread() to return errors so
the application has a chance to recover.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This changes both k_mem_domain_add_partition() and
k_mem_domain_remove_partition() to return errors instead of
asserting when errors are encountered. This gives the application
chance to recover.
The arch_mem_domain_parition_add()/_remove() will be modified
later together with all the other arch_mem_domain_*() changes
since the architecture code for partition addition and removal
functions usually cannot be separately changed.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This changes k_mem_domain_init() to return error values
instead of asserting when errors are encountered.
This gives applications a chance to recover if needed.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Remove LOG_MINIMAL kconfig option which was confusing
since LOG_MODE_MINIMAL existed. LOG_MINIMAL was used to
force minimal mode but because of invalid dependencies
it was leading to issues.
Refactored code to use LOG_MODE_MINIMAL everywhere and
renamed LOG_MINIMAL to LOG_DEFAULT_MINIMAL which has impact
on defualt logging mode (which still can be later changed
in conf file or in menuconfig).
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
Added heap reference parameter to k_free tracing
hook to allow tracing of the pointer which was
passed as a parameter to a k_free call.
As part of this update the defines
(for this hook) in the various tracing formats
was also updated.
Signed-off-by: Torbjörn Leksell <torbjorn.leksell@percepio.com>
With `gen_handles.py` now running on the first pre-built image,
`zephyr_pre0.elf` there is no requirement for the device handle arrays
to remain the same size after processing.
Remove the padding generated in `gen_handles.py`, as well as the
temporary option `CONFIG_DEVICE_HANDLE_PADDING` which was added to work
around this issue.
Signed-off-by: Jordan Yates <jordan.yates@data61.csiro.au>
page_frame_dump() and z_page_frames_dump() are used for
debug print, so there is no need to cover those funcs.
__weak function is also excluded, every test overrides it.
Signed-off-by: Lixin Guo <lixinx.guo@intel.com>
It turns out that we have a sample (though not a test) that really
does want to use "k_thread_runtime_stats_all_get()" to measure system
uptime.
Instead of breaking this needlessly, separate the accounting for idle
and non-idle threads. The legacy API can report their sum, and the
more useful value is available via the kernel struct for future
analysis.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Clean up RUNTIME_STATS to separate the API from the individual data
backends. Use the SCHED_THREAD_USAGE tracking instead of the original
for execution_cycles. Move the kconfig for that into the runtime
stats menu, since it's part of the family now.
Also remove a lot of needless #if's around the declarations. Unused
structs and uncalled functions don't need to be explicitly hidden. An
attempt to access a non-existent field (e.g. "execution_cycles" if
that isn't configured) provides all the build time validation we need.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
The runtime stats feature has always supported this, so use the same
kconfig to indirect the timing source in the same way.
(Personally I'm not a fan of the "timing" API, which really doesn't do
anything that the existing core "cycles" API does not except add a
bunch of code due to the separate implementation of frequency
management and conversion routines. It comes from an era where
"cycles" were fixed to a MHz frequency clock on platforms like x86 yet
we had benchmarks that wanted to use the TSC. Those days are behind
us and "cycles" can be fast everywhere.)
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
On older architectures, we don't have the
architecture-independent/scheduler-internal hooks (which require
USE_SWITCH) but there is a hook shared by the tracing layer we can use.
This is sort of a layering violation (stat tracking is a core feature,
tracing is supposed to be optional), but simple and lightweight. And
eventually it will go away as these architectures migrate.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
This is an alternate backend that does what THREAD_RUNTIME_STATS is
doing currently, but with a few advantages:
* Correctly synchronized: you can't race against a running thread
(potentially on another CPU!) while querying its usage.
* Realtime results: you get the right answer always, up to timer
precision, even if a thread has been running for a while
uninterrupted and hasn't updated its total.
* Portable, no need for per-architecture code at all for the simple
case. (It leverages the USE_SWITCH layer to do this, so won't work
on older architectures)
* Faster/smaller: minimizes use of 64 bit math; lower overhead in
thread struct (keeps the scratch "started" time in the CPU struct
instead). One 64 bit counter per thread and a 32 bit scratch
register in the CPU struct.
* Standalone. It's a core (but optional) scheduler feature, no
dependence on para-kernel configuration like the tracing
infrastructure.
* More precise: allows architectures to optionally call a trivial
zero-argument/no-result cdecl function out of interrupt entry to
avoid accounting for ISR runtime in thread totals. No configuration
needed here, if it's called then you get proper ISR accounting, and
if not you don't.
For right now, pending unification, it's added side-by-side with the
older API and left as a z_*() internal symbol.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Add spinlock unlocking before calling timer expiration
handler. Locking was introduced by dde3d6c.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
Instead of returning PM_STATE_ACTIVE for when the cpu didn't enter a
low power state and a different state when it entered, but has
already left the state and is active again, it changes
pm_system_suspend to return true when the cpu has entered a low power
state and false otherwise.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
There was a brief (but seen in practice on real apps on real
hardware!) race with the switch-based z_swap() implementation. The
thread return value was being initialized to -EAGAIN after the
enclosing lock had been released. But that lock is supposed to be
atomic with the thread suspend.
This opened a window for another racing thread to come by and "wake
up" our pending thread (which is fine on its own), set its return
value (e.g. to 0 for success) and then have that value clobbered by
the thread continuing to suspend itself outside the lock.
Melodramatic aside: I continue to hate this
arch_thread_return_value_set() API; it needs to die. At best it's a
mild optimization on a handful of architectures (e.g. x86 implements
it by writing to the EAX register save slot in the context block).
Asynchronous APIs are almost always worse than synchronous ones, and
in this case it's an async operation that races against literal
context switch code that can't use traditional locking strategies.
Fixes#39575
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Threads may wait on an event object such that any events posted to
that event object may wake a waiting thread if the posting satisfies
the waiting threads' event conditions.
The configuration option CONFIG_EVENTS is used to control the inclusion
of events in a system as their use increases the size of
'struct k_thread'.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
The k_work::flags field is not an atomic_t and would cause
-Wpointer-sign warning on some compilers. This function was the only
one in work.c to use atomic_get() so there is no benefit to atomicity.
Signed-off-by: Chris Reed <chris.reed@arm.com>
In the case where the aligned memory range is on top of the allocated
memory range, freeing the 0 sized top unused memory will trigger
an assert in the virt_region_free() call since vaddr could be equal
to Z_VIRT_REGION_END_ADDR.
Signed-off-by: Neil Armstrong <narmstrong@baylibre.com>
On ARM64 platforms, when mapping multiple memory zones with size
not multiple of a L2 block size (2MiB), all the following mappings
will probably use L3 tables.
And a huge mapping will consume all possible L3 tables.
In order to reduce usage of L3 tables, this introduces a new
arch_virt_region_align() optional architecture specific
call to eventually return a more optimal virtual address
alignment than the default MMU_PAGE_SIZE.
This alignment is used in virt_region_alloc() by:
- requesting more pages in virt_region_bitmap to make sure we request
up to the possible aligned virtual address
- freeing the supplementary pages used for alignment
Suggested-by: Nicolas Pitre <npitre@baylibre.com>
Signed-off-by: Neil Armstrong <narmstrong@baylibre.com>
When CONFIG_USERSPACE is enabled, the ELF file from linker pass 1 is
used to create a hash table that identifies kernel objects by address.
We therefore can't allow the size of any object in the pass 2 ELF to
change in a way that would change those addresses, or we would create
a garbage hash table.
Simultaneously (and regardless of CONFIG_USERSPACE's value),
gen_handles.py must transform arrays of handles from their pass 1
values to their pass 2 values; see the file's docstring for more
details on that transformation.
The way this works is that gen_handles.py just pads out each pass 2
array so its length is the same as its pass 1 value. The padding value
is a repeated run of DEVICE_HANDLE_ENDS values. This value is the
terminator which we look for at runtime in places like
device_required_handles_get(), so there must be at least one, and we
error out in gen_handles.py if there's no room in the pass 2 array for
at least one such value. (If there is extra room, we just keep
inserting extra DEVICE_HANDLE_ENDS values to pad the array to its
original length.)
However, it is possible that a device has more direct dependencies in
the pass 2 handles array than its corresponding devicetree node had in
the pass 1 array. When this happens, users have no recourse, so that's
a potential showstopper.
To work around this possibility for now, add a new config option,
CONFIG_DEVICE_HANDLE_PADDING, whose value defaults to 0.
When nonzero, it is a count of padding handles that are inserted into
each device handles array. When gen_handles.py errors out due to lack
of room, its error message now tells the user how much to increase
CONFIG_DEVICE_HANDLE_PADDING by to work around the problem.
It looks like a real fix for this is to allocate kernel objects whose
addresses are required for hash tables in CONFIG_USERSPACE=y
configurations *before* the handle arrays. The handle arrays could
then be resized as needed in pass 2, which saves ROM by avoiding
unnecessary padding, and would avoid the need for
CONFIG_DEVICE_HANDLE_PADDING altogether.
However, this 'real fix' is not available and we are facing a deadline
to get a temporary solution in for Zephyr v2.7.0, so this is a good
enough workaround for now.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This reverts commit b01e41ccdd.
It's not clear that the supported devices are being properly computed,
so let's revert this for v2.7.0 until we've had more time to think
it through.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
before running timer's timeout function, we need to make
sure that those threads waiting on this timer have been
added into the timer's wait queue, so add operations to
use timer lock to mask interrupts in z_timer_expiration_handler
function to synchronize timer's wait queue.
Signed-off-by: Chen Peng1 <peng1.chen@intel.com>
Some SMP applications have threading designs where every thread
created is always assigned to a specific CPU, and never want to
schedule them symmetrically across CPUs under any circumstance.
In this situation, it's possible to optimize the run queue design a
bit to put a separate queue in each CPU struct instead of having a
single global one. This is probably good for a few cycles per
scheduling event (maybe a bit more on architectures where cache
locality can be exploited) in circumstances where there is more than
one runnable thread. It's a mild optimization, but a basically simple
one.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Split "init_ready_q()" into a separate function that operates on the
queue pointer and not the global kernel object. Pure refactoring.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Similar to the previous patch, the various _priq_run_*() functions are
always passed a first argument that is the singleton system run queue
(this is because the same backend functions are used by wait queues).
Refactor into a simpler API that places the access to the run queue in
just a single spot.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Pure refactoring. For historical reasons these two functions took a
first argument (a pointer to the run queue) that was always the same.
Eliminate it.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Adding missing parenthesis. Without them wrong results
appeared when k_cycle_get_32 wrapped.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
add a bitarray into struct osThreadDef_t to indicate whether the
thread is used or not, then we can get the first available thread
by searching this array when creating a new thread, and update this
array to add a free thread when terminating a thread.
Signed-off-by: Chen Peng1 <peng1.chen@intel.com>
Cadence XCC is based off of a very old 4.2 gcc compiler, which didn't
perfectly support C99 "inline" semantics with respect to
cross-translation-unit inline linkage (which Zephyr does not use, our
inlines are static only) and declaration order.
Fix the one spot where we were calling an inline before its
ALWAYS_INLINE definition, and add a flag to suppress the warning so
CI's trying to build with XCC and -Werror don't flip out.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Some architectures already returns -ENOTSUP when these functions
are called. So add this return value to the API doc.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Add a SOC API to allow for application control over deep idle power
states. Note that the hardware idle entry happens out of the WAITI
instruction, so the application has to be responsibile for ensuring
the CPU to be halted actually reaches idle deterministically. Lots of
warnings in the docs to this effect.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
This commit removes the `timeout_q` from the `struct z_kernel` since it
is no longer used.
Note that the new kernel timeout implementation introduced in the
commit 987c0e5fc1 uses `timeout_list`
global variable in place of it.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
To support arm-ds / armlink it is required that the weak main is located
in an object externally to the object using the weak symbol.
If the weak symbol is inside the object referring to it, then the weak
symbol will be used and this will result in
```
Error: L6200E: Symbol __ARM_use_no_argv multiply defined
(by init.o and main.o).
```
as both the weak and strong symbols are used.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Cleanup and preparation commit for linker script generator.
Zephyr linker scripts provides start and end symbols for each section,
and sometimes even size and LMA start symbols.
Generally, start and end symbols uses the following pattern, as:
Section name: foo
Section start symbol: __foo_start
Section end symbol: __foo_end
However, this pattern is not followed consistently.
To allow for linker script generation and ensure consistent naming of
symbols then the following pattern is introduced consistently to allow
for cleaner linker script generation.
Section name: foo
Section start symbol: __foo_start
Section end symbol: __foo_end
Section size symbol: __foo_size
Section LMA start symbol: __foo_load_start
This commit aligns the symbols for __itcm_load_start and
__dtcm_data_load_start to other symbols and in such a way they follow
consistent pattern which allows for linker script and scatter file
generation.
The symbols are named according to the section name they describe.
Section names are itcm and dtcm.
The following symbols are aligned in this commit:
- __itcm_rom_start -> __itcm_load_start
- __dtcm_data_rom_start -> __dtcm_data_load_start
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Cleanup and preparation commit for linker script generator.
Zephyr linker scripts provides start and end symbols for each section,
and sometimes even size and LMA start symbols.
Generally, start and end symbols uses the following pattern, as:
Section name: foo
Section start symbol: __foo_start
Section end symbol: __foo_end
However, this pattern is not followed consistently.
To allow for linker script generation and ensure consistent naming of
symbols then the following pattern is introduced consistently to allow
for cleaner linker script generation.
Section name: foo
Section start symbol: __foo_start
Section end symbol: __foo_end
Section size symbol: __foo_size
Section LMA start symbol: __foo_load_start
This commit aligns the symbols for _ramfunc_ram/rom to other symbols and
in such a way they follow consistent pattern which allows for linker
script and scatter file generation.
The symbols are named according to the section name they describe.
Section name is `ramfunc`
The following symbols are aligned in this commit:
- _ramfunc_ram_start -> __ramfunc_start
- _ramfunc_ram_end -> __ramfunc_end
- _ramfunc_ram_size -> __ramfunc_size
- _ramfunc_rom_start -> __ramfunc_load_start
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Cleanup and preparation commit for linker script generator.
Zephyr linker scripts provides start and end symbols for each section,
and sometimes even size and LMA start symbols.
Generally, start and end symbols uses the following pattern, as:
Section name: foo
Section start symbol: __foo_start
Section end symbol: __foo_end
However, this pattern is not followed consistently.
To allow for linker script generation and ensure consistent naming of
symbols then the following pattern is introduced consistently to allow
for cleaner linker script generation.
Section name: foo
Section start symbol: __foo_start
Section end symbol: __foo_end
Section size symbol: __foo_size
Section LMA start symbol: __foo_load_start
This commit aligns the symbols for _data_ram/rom to other symbols and in
such a way they follow consistent pattern which allows for linker script
and scatter file generation.
The symbols are named according to the section name they describe.
Section name is `data`
A new group named data_region is introduced which instead spans all the
input and output sections that was previously covered by
__data_ram_start, __data_ram_end, and __data_rom_start.
The following symbols are aligned in this commit:
- __data_ram_start -> __data_region_start
- __data_ram_end -> __data_region_end
- __data_rom_start -> __data_region_load_start
The following new symbols are introduced so that the data section is
aligned with other sections:
- __data_end
- __data_start
value identical to __data_region_start but describes start of
the section.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
The initialization of the struct pm_device pm field found in the device
state can be statically initialized without the need of doing it at
runtime.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
With demand paging, the heap object and its backing memory
may not be in physical memory. So initialize those heaps
in pinned region at PRE_KERNEL_1 and the remaining heaps
once paging mechanism has been initialized.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds the kconfig to allow reserving a number of page frames
which do not count towards free memory. This is to ensure that
there are enough page frames available for paging code and data.
Or else, it would be possible to exhaust all page frames via
anonymous memory mappings.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This allows memory partitions to be put into the pinned
section so they are available during boot. For example,
the stack guard (in libc partition) is needed during boot
but before the paging mechanism is initialized. Without
pinning it in physical memory, it would fault in early
boot process.
A new cmake property app_smem,pinned_partitions is
introduced so that additional partitions can be pinned
if needed.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
During boot process, the boot sections need to be pinned in
memory to prevent them from being paged out (to avoid
pages being paged out and immediately paged in again).
Once the boot process is completed (just before calling main()),
the boot sections can be unpinned so the memory can be
used for demand paging for paging in data pages.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
If BSS section is not present in memory at boot, it would not
have been cleared as the data pages are not in physical memory.
Manipulating those pages would result in page faults.
In this scenario, zeroing BSS can only be done once the paging
mechanism has been initialized. So do it there.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The beginning of code in do_page_fault() is to pin the page
in memory if it is already present in physical memory.
It is there so that if a page is not present, it can proceed
to perform page-in and then pin it. So the counting of
page faults needs to be moved after the pinning code so
it actually counts page faults, and not counting pinning
operations when the page is already present.
Also clarify the comment on the goto statement as it is not
correct.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The z_main_stack is needed before paging mechanism is initialized
so put the stack into the pinned section to avoid page faults.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
In do_page_fault(), the incoming page fault address is not
aligned, and it was unconditionally assigned to the page
frame virtual address field. If the backing store simply
returns the virtual address without processing in
k_mem_paging_backing_store_location_get(), this unaligned
address will be passed to arch_mem_page_out(). On x86,
it is further passed to range_map() which asserts if
the physical address is not page aligned. So align
the address to page size before assigning it to the page
frame virtual address field.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
k_work_queue_start receives a struct that is expected to be
uninitialized (zeroed). Otherwise the behavior is undefined.
Following the Zephyr semantics, this pr introduce a new init function
for this struct.
Fixes#36865
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Adds an API to query and visit supported devices. Follows the example
set by the required devices API.
Implements #37793.
Signed-off-by: Jordan Yates <jordan.yates@data61.csiro.au>
This adds a very primitive logic to allow linking a prebuilt
static library of kernel code instead of building the kernel
from source. Note that the library is built with a specific
set of kconfigs, and they must match when building applications,
or else there would be mysterious crashes.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
While reading the code, found some typos in the code comments,
line 226 and 668.
Fix comments to make it more solid.
Signed-off-by: Naiyuan Tian <naiyuan.tian@intel.com>
Coding scanning tool raises a violation that happens dereferencing
of the "work" pointer in the expression "work->handler"
As were discussed before in PR #35664 it is not true.
Add explanation comment, because static code analysis tool
raised false-positive violation there.
Signed-off-by: Maksim Masalski <maksim.masalski@intel.com>
There are no reference for either K_NUM_PRIORITIES or
K_NUM_PRIO_BITMAPS, with the former being dropped in:
1acd8c2996 kernel: Scheduler rewrite
Dropping both of these, and also the two comments about extra priorities
taking extra RAM space, as those do not seem to apply either.
Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
This migrates all the current iterable section usages to the external
API, dropping the "Z_" prefix:
Z_ITERABLE_SECTION_ROM
Z_ITERABLE_SECTION_ROM_GC_ALLOWED
Z_ITERABLE_SECTION_RAM
Z_ITERABLE_SECTION_RAM_GC_ALLOWED
Z_STRUCT_SECTION_ITERABLE
Z_STRUCT_SECTION_ITERABLE_ALTERNATE
Z_STRUCT_SECTION_FOREACH
Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
Introduce a new API to allow devices capable of wake up the system
register themselves was wake up sources. This permits applications to
select the most appropriate way to wake up the system when it is
suspended.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
The constructors of static objects are stored in ".ctors"
section. In case of MWDT toolchain we have incompatible
".ctors" section format with GNU toolchain. So let's use
initialization code provided by MWDT instead of Zephyr one
in case of MWDT toolchain usage.
As it is done for GNU toolchain We call constructors of
static objects but we don't call destructors for them.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
Signed-off-by: Evgeniy Paltsev <PaltsevEvgeniy@gmail.com>
The device PM subsystem already holds the device state, so there is no
need to keep duplicates inside the device. The pm_device_state_get has
been refactored to just return the device state. Note that this is still
not safe, but the same applied to the previous implementation. This
problem will be addressed later.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
We cache the current thread ID in a thread-local variable
at thread entry, and have k_current_get() return that,
eliminating system call overhead for this API.
DL: changed _current to use z_current_get() as it is
being used during boot where TLS is not available.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The following device busy APIs:
- device_busy_set()
- device_busy_clear()
- device_busy_check()
- device_any_busy_check()
were used for device PM, so they have been moved to the pm subsystem.
This means they are now prefixed with `pm_` and are defined in
`pm/device.h`.
If device PM is not enabled dummy functions are now provided that do
nothing or return `-ENOSYS`, meaning that the functionality is not
available.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
z_smp_init() is only available if CONFIG_SMP is defined,
smp_timer_init() also depends on two Kconfig parameters. Also make it
conditional in cavs_timer.c. Also clarify some SMP-related comments
there.
Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
The z_interrupt_stacks was declared extern in the kernel internal
header file using the same macro which defines the same stack
array but with an added "extern" in front. This macro adds
alignment and section attribute which are actually not the same
as the actual stack array defined in kernel/init.c. The section
name used in the section attribute contains the file name where
the stack array is defined or extern declared. So the same
symbol, in this case z_interrupt_stacks, has different
attributes in two places, and GCC 11 starts to complain about
this. So use the newly introduced macro to extern declare
the stack array without adding/replacing any symbol attributes.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
On the native_posix board global object constructors
are run by the underlying OS runtime init prior to
Zephyr kernel init. Thus Zephyr should not run global
object constructors a second time. Doing so breaks
application behavior that relies on global
constructors doing work that must be done only once.
See bug #36858 for more information.
Signed-off-by: David Palchak <palchak@google.com>
If you try to unlock an unlocked mutex, it will incorrectly
succeeds and decreases the lock count to -1.
Fixes#36572
Signed-off-by: Chih Hung Yu <chyu313@gmail.com>
The final else {} in the if...else if is missing required
comment (non-empty, ';' is not sufficient). This adds a comment
to comply with CG 15.7.
Signed-off-by: Jennifer Williams <jennifer.m.williams@intel.com>
According to the Zephyr Coding Guideline all switch statements
shall be well-formed. Add a default case with break and comment
to avoid static analysis tool to raise a violation that there is no
default case.
Also, I think, in all cases above no need to use "break",
because they already are using "return".
Found as a coding guideline violation (MISRA R16.1) by static
coding scanning tool.
Signed-off-by: Maksim Masalski <maksim.masalski@intel.com>
Function types shall be in prototype form with named parameters
Found as a coding guideline violation (MISRA R8.2) by static
coding scanning tool.
Signed-off-by: Maksim Masalski <maksim.masalski@intel.com>
Changed location of the last k_mutex_unlock trace hook since it was
being called after k_sched_unlock, which could result in tracing
scenarios (other thread waiting for lock) where it appeared that a
mutex was being locked again before becoming unlocked.
Signed-off-by: Torbjörn Leksell <torbjorn.leksell@percepio.com>
Add a dependency on MULTITHREADING for the
STACK_SENTINEL feature, so it may not get
enabled in single-thread Zephyr builds.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
These functions are those that need be implemented by backing
store outside kernel. Promote them from z_* so these can be
included in documentation.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
These functions and data structures are those that need
to be implemented by eviction algorithm and application
outside kernel. Promote them from z_* so these can be
included in documentation.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Fixes calculation of remaining ticks returned from z_tick_sleep
so that it takes absolute timeouts into account.
Fixes#32506
Signed-off-by: Lauren Murphy <lauren.murphy@intel.com>
If single thread builds are not supported by the
architecture, the MULTITHREADING option should be
prompt-less to block any modifications to it. We
also introduce an explicit ARCH-level Kconfig that
reflects whether the ARCH is capable of single-thread
Zephyr builds.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
Protect critical sections using the mutex.
The mutex is required to use the conditional variable and since we
need to atomically check the pm state and the workqueue before wait
the condition, it is necessary to protect them using the same mutex.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Add a function that properly uses a mutex to check a condition before
wait on the conditional variable.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
In file include/kernel/thread.h in "struct _thread_base" is a member
called "_wait_q_t *pended_on"
At the same time in file kernel/sched.c is function called
"static _wait_q_t *pended_on()"
Coding scanning tool assigns violation (MISRA R5.9) that static
object reused, because thread.h is included in struct.c file.
I think we can rename function to avoid misreading in the future.
Signed-off-by: Maksim Masalski <maksim.masalski@intel.com>
Correct the way the relative ticks value is calculated for an absolute
timeout. Previously, elapsed() was called twice and the returned value
was first subtracted from and then added to the ticks value. It could
happen that the HW counter value read by elapsed() changed between the
two calls to this function. This caused the test_timeout_abs test case
from the timer_api test suite to occasionally fail, e.g. on certain nRF
platforms.
Signed-off-by: Andrzej Głąbek <andrzej.glabek@nordicsemi.no>
The scheduler has historically had an API where an application can
inform the kernel that it will never create a thread that can be
preempted, and the kernel and architecture layer would use that as an
optimization hint to eliminate some code paths.
Those optimizations have dwindled to almost nothing at this point, and
they're now objectively a smaller impact than the special casing that
was required to handle the idle thread (which, obviously, must always
be preemptible).
Fix this by eliminating the idea of "cooperative only" and ensuring
that there will always be at least one preemptible priority with value
>=0. CONFIG_NUM_PREEMPT_PRIORITIES now specifies the number of
user-accessible priorities other than the idle thread.
The only remaining workaround is that some older architectures (and
also SPARC) use the CONFIG_PREEMPT_ENABLED=n state as a hint to skip
thread switching on interrupt exit. So detect exactly those platforms
and implement a minimal workaround in the idle loop (basically "just
call swap()") instead, with a big explanation.
Note that this also fixes a bug in one of the philosophers samples,
where it would ask for 6 cooperative priorities but then use values -7
through -2. It was assuming the kernel would magically create a
cooperative priority for its idle thread, which wasn't correct even
before.
Fixes#34584
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
File userspace.c contains dead code in function char *otype_to_str()
Remove "return NULL" and replace with "ret = NULL".
Found as a coding guideline violation (MISRA R2.1) by static
coding scanning tool.
Signed-off-by: Maksim Masalski <maksim.masalski@intel.com>
If this call receives an invalid device pointer as argument it
assumes that the `device` is not ready for usage.
This routine is currently called by two device specific APIs:
- device_usable_check(const struct device *dev)
- device_is_ready(const struct device *dev)
The device-specific APIs documentation claims that these two
routines must be called with a device pointer captured from
DEVICE_DT_GET(). So passing NULL is a violation of the rule.
Nevertheless, is quite common in drivers to assign NULL to
a device pointer if the corresponding DT property has not been
found (e.g. a not used gpio interrupt declaration for a given
device instance) and seems legit to interpret this condition
same as the device is not ready for usage.
Signed-off-by: Armando Visconti <armando.visconti@st.com>
The original state management solution involved separate locks for a
work queue and each work item. To avoid inter-lock dependencies a
window was left between the point where the work item was removed from
the queue (protected by queue lock) and the point where the work item
state was updated to mark the work item running.
This introduced a bug: If a cancellation was issued during this window
it would succeed, and the work item would appear to be idle even
though in fact the work queue thread was about to run it.
Since there is now only one lock, move the work item state updates
into the mutex regions associated with dequeuing the work item and
clearing the work queue busy flag.
Note that removing the window between queue and work mutex regions
eliminates the potential of having a dequeued work item be cancelled
before its QUEUED flag is cleared, simplifying the work item state
update.
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
Violation of the [MISRAC2012-RULE_14_3-j]:
Boolean operations whose results are invariant
shall not be permitted
Probably in that part of code is a misprint.
Added to check _OBJ_INIT_FALSE case explicitly
Signed-off-by: Maksim Masalski <maksim.masalski@intel.com>
Our z_swap() API takes a key returned from arch_irq_lock() and
releases it atomically with the context switch. Make sure that the
action of the unlocking is to unmask interrupts globally. If
interrupts would still be masked then that means there is an OUTER
interrupt lock still held, and the code that locked it surely doesn't
expect the thread to be suspended and interrupts unmasked while it's
held!
Unfortunately, this kind of mistake is very easy to make. We should
catch that with a simple assertion. This is essentially a crude
Zephyr equivalent of the extremely common "BUG: scheduling while
atomic" error in Linux drivers (just google it).
The one exception made is the circumstance where a thread has already
aborted itself. At that stage, whatever upthread lock state might
have existed will have already been messed up, so there's no value in
our asserting here. We can't catch all bugs, and this can actually
happen in error handling and/or test frameworks.
Fixes#33319
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
When marking the reserved region at the end of virtual address
space, call virt_to_bitmap_offset() is not needed as we already
know the offset. So remove it.
Coverity-CID: 235930
Fixes#35160
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds the necessary bits for linker scripts and source code
to specify which symbols need to be pinned in memory. This is
needed for demand paging as some functions and data must reside
in memory all the time and cannot be paged out (e.g. paging,
scheduler, and interrupt routines for functionality).
This is up to the arch/SoC/board to define the sections in
their linker scripts as the pinned section may need special
alignment which cannot be done in common script snippets.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds the necessary bits for linker scripts and source code
to specify which symbols are needed for boot process so they
can be grouped together.
One use of this is to group boot related code and data so these
won't interval with other kernel and application for better
caching.
This is a must for demand paging as some functions and data
must be available during the boot process and before the memory
manager is initialized. During this time, paging cannot be used
so symbols linked in virtual memory space are unavailable.
This is up to the arch/SoC/board to define the sections in
their linker scripts as section may need special alignment
which cannot be done in common script snippets.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The cache API currently shipped in Zephyr is assuming that the cache
controller is always on-core thus managed at the arch level. This is not
always the case because many SoCs rely on external cache controllers as
a peripheral external to the core (for example PL310 cache controller
and the L2Cxxx family). In some cases you also want a single driver to
control a whole set of cache controllers.
Rework the cache code introducing support for external cache
controllers.
Signed-off-by: Carlo Caione <ccaione@baylibre.com>
Remove this intrusive tracing feature in favor of the new object tracing
using the main tracing feature in zephyr. See #33603 for the new tracing
coverage for all objects.
This will allow for support in more tools and less reliance on GDB for
tracing objects.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add Poll API and Work Poll tracing, default hooks, and documentation.
Signed-off-by: Torbjörn Leksell <torbjorn.leksell@percepio.com>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Device pm runtime was using semaphore to protect critical section but
enable / disable functions were waiting on the semaphore. So, just
replace it with a spin lock.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
The sync API was using k_poll_signal and in certain conditions is
possible multiple threads waiting on a signal leading to an undefined
behavior.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
This uses bitarrays for allocating and deallocating virtual
addresses with k_mem_map() and k_mem_unmap(). This will
allow us to reuse virtual addresses.
Fixes#28900
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds a new function prototype for arch_page_phys_get()
which will be used to translate mapped virtual addresses back
to physical memory addresses. This is needed for the future
k_mem_unmap() function which requires this to find
the corresponding page frame. It is faster to look through
the page tables instead of doing linear search of the page
frame array.
A weak function is provided in case arch_page_phys_get()
is not implemented at the arch level. This simply goes
through all the page frame and find the one which has
mapped to the virtual address.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When we start allowing unmapping of memory region, there is no
exact way to know if k_mem_map() is called with guard page option
specified or not. So just unconditionally enable guard pages on
both sides of the memory region to hopefully catch access
violations.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This provides a counterpart to z_phys_map() which can be used
to temporary map memory region during boot process, and
subsequently discards the mapping.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
work_timeout() is a function, a statement like "(void)work_timeout;"
has no effect.
Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Remove the config BOOT_TIME_MEASUREMENT and corresponding #ifdef'd code
throughout (kernel/init.c, idle.c, core/common.S , reset.S, ... ) which
hold the extern hooks for z_timestamp_main and z_timestamp_idle in the
removed boot_time test suite.
Signed-off-by: Jennifer Williams <jennifer.m.williams@intel.com>
smp.c only has to be built if CONFIG_SMP is enabled. Remove
preprocessor checks from the file itself and update cmake rules
instead.
Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Usually Zephyr boots all secondary CPUs as a part of system
boot. Some applications however need an ability to boot on
the main CPU only and enable secondary CPUs selectively at
run-time. Add a Kconfig option to support this behaviour.
When booting CPUs on demand applications also need helpers
to initialise a dummy thread and begin threaded execution
on those CPUs, add two such helpers.
Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Avoid fetching files which use scheduler. By explicitly avoiding
including RTOS specific files we ensure that it is not fetched
accidently.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
Ensure that k_heap is not attempt to block the thread when
timeout is set and space cannot be allocated.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
Mem_slab supports allocation with timeout which blocks the context
if no slab is available. Updated to treat every timeout as K_NO_WAIT
when multithreading is disabled.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
Updated timer to not touch thread/scheduler code when multithreading
is off.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
_kernel struct can be used when multithreading is disabled.
In that case sched.c may not be compiled.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
K_busy_wait is the only function from thread.c that is used when
CONFIG_MULTITHREADING=n. Moving to timeout since it fits better there
as it requires sys clock to be present.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
Return NULL instead of return numeric zero for pointer type.
Current usage violates MISRA rule 11.9.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This renames the obj_list element in struct dyn_obj to
dobj_list, to avoid identifier collision with the static
obj_list defined in userspace.c.
Violation of MISRA rule 5.9.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
work_queue_main() was missing final else statement
in the if else if construct. This commit adds else {}
to comply with coding guideline 15.7. Includes a
context-specific description of why this branch is empty.
Signed-off-by: Jennifer Williams <jennifer.m.williams@intel.com>
z_timeout_end_calc() was missing final else statement
in the if else if construct. This commit pulls the last
condition into a final else {} to comply with guideline
15.7.
Signed-off-by: Jennifer Williams <jennifer.m.williams@intel.com>
register_events() and signal_poll_event() missing final
else statement in the if else if construct. This commit adds
else {} to comply with coding guideline 15.7.
Signed-off-by: Jennifer Williams <jennifer.m.williams@intel.com>
Devices that do not require PM should just use NULL.
`device_pm_control_nop` is still kept as an alias to NULL untill all
in-tree usage is replaced with NULL.
Code relying on device_pm_control function now returns -ENOTSUP
(equivalent to calling device_pm_control_nop).
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Due to the use of gperf to generate hash table for kobjects,
the addresses of these kobjects cannot change during the last
few phases of linking (especially between zephyr_prebuilt.elf
and zephyr.elf). Because of this, the gperf generated data
needs to be placed at the end of memory to avoid pushing symbols
around in memory. This prevents moving these generated blocks
to earlier sections, for example, pinned data section needed
for demand paging. So create placeholders for use in
intermediate linking to reserve space for these generated blocks.
Due to uncertainty on the size of these blocks, more space is
being reserved which could result in wasted space. Though, this
retains the use of hash table for faster lookup.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The return value is documented to be true if the work was pending, but
the implementation returned true only if the work was actually running
(i.e. the caller had to wait). It should also return true if
scheduled or submitted work was cancelled.
Note that this means the return value cannot be used to determine
whether the call slept.
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
Add the ability to define architecture specific structures, notably
the ability to extend struct _cpu with per-CPU arch-specific stuff that
can be accessed with _current_cpu->arch.* similarly to _current->arch.*
for per-thead architecture data.
This is opt-in for architectures that want to benefit from this,
otherwise empty defaults are provided. A placeholder for ARM64 is
included to show the pattern.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
There's a typedef for non-pointer values compatible with atomic
non-pointer objects. Add a similar typedef for pointer values, and
the corresponding macro for initializing atomic pointer types.
This also will simplify replacing the Zephyr atomic API with one
based on C11 atomics, should that be desirable. C11 atomic pointer
values are not void*.
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
This commit adds the ability to use a message queue as a
k_poll object. It follows the same pattern as polling on
FIFOs.
This change has been proven in practice at Samsara.
Fixes: #26728
Signed-off-by: Nick Graves <nicholas.graves@samsara.com>
This avoids contention between unrelated slabs and allows for
userspace accessible slabs when located in memory partitions.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Both operands of an operator in which the usual arithmetic
conversions are performed shall have the same essential
type category.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Currently _curr_cpu is only used by the get_cpu macro to quickly access
the cpu struct. This is not really necessary because we can access to
the struct by directly referencing &(_kernel.cpus[cpu_num]) in assembly
Signed-off-by: Carlo Caione <ccaione@baylibre.com>
This adds bits to the paging timing histogram collection routines
so they can use timing functions to collect execution time data.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds a new kconfig CONFIG_TIMING_FUNCTIONS_NEED_AT_BOOT so
that the timing subsystem can be initialized at boot, instead of
being #ifdef under thread runtime statistics. This will allow
other part of kernel and other subsystems to utilize the timing
functions.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds the bits to record execution time of eviction selection,
and backing store page-in/page-out in histograms.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds more bits to gather statistics on demand paging,
e.g. clean vs dirty pages evicted, # page faults with
IRQ locked/unlocked, etc.
Also extends this to gather per-thread demand paging
statistics.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Tests of a value against zero should be made explicit, unless the
operand is effectively Boolean. This is based on MISRA rule 14.4.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Tests of a value against zero should be made explicit, unless the
operand is effectively Boolean. This is based on MISRA rule 14.4.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add a 'U' suffix to values when computing and comparing against
unsigned variables and other related fixes of the same MISRA rule (10.4)
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
k_work_schedule() is supposed to be a no-op if the work item is
already scheduled or submitted: the previous schedule is left
unchanged. The check incorrectly inhibited the schedule operation
when the work item was neither scheduled nor submitted, but was
running.
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
The identifiers used in the declaration and definition of a function
shall be identical [MISRAC2012-RULE_8_3-b]
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This patch replaces ENOSYS into ENOTSUP to keep consistency with
the return value specification of k_float_enable().
Signed-off-by: Katsuhiro Suzuki <katsuhiro@katsuster.net>
This patch introduce new API to enable FPU of thread. This is pair of
existed k_float_disable() API. And also add empty arch_float_enable()
into each architectures that have arch_float_disable(). The arc and
riscv already implemented arch_float_enable() so I do not touch
these implementations.
Motivation: Current Zephyr implementation does not allow to use FPU
on main and other system threads like as work queue. Users need to
create an other thread with K_FP_REGS for floating point programs.
Users can use FPU more easily if they can enable FPU on running
threads.
Signed-off-by: Katsuhiro Suzuki <katsuhiro@katsuster.net>
This symbol is reserved and usage of reserved symbols violates the
coding guidelines. (MISRA 21.2)
NAME
signal - ANSI C signal handling
SYNOPSIS
#include <signal.h>
sighandler_t signal(int signum, sighandler_t handler);
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This symbol is reserved and usage of reserved symbols violates the
coding guidelines. (MISRA 21.2)
NAME
remove - remove a file or directory
SYNOPSIS
#include <stdio.h>
int remove(const char *pathname);
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
both thread monitor and thread names are not EXPERIMENTAL any more. They
have been used across the tree and lots depend on those features
already.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Removed k_pipe_block_put and static functions only related to it.
After all the old usage of k_mem_block has been replaced by k_heap,
k_pipe_block_put still taking a deprecated k_mem_block as argument
becomes dead code. All APIs that hook it from kernel.h have been
confirmed to be removed. Since an asynchronous message descriptor
is only allocated in k_pipe_block_put, static functions for pipe_
async are removed as well.
Signed-off-by: Shihao Shen <shihao.shen@intel.com>
This feature predated the tickless kernel and has been in legacy mode
for a while. We now have no drivers or systems that do not support
tickless, so remove this option and cleanup the code to only use
tickless.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The clock/timer APIs are not application facing APIs, however, similar
to arch_ and a few other APIs they are available to implement drivers
and add support for new hardware and are documented and available to be
used outside of the clock/kernel subsystems.
Remove the leading z_ and provide them as clock_* APIs for someone
writing a new timer driver to use.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The only user of arch_mem_domain_destroy was the deprecated
k_mem_domain_destroy function which has now been removed. So remove
arch_mem_domain_destroy as well.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
Remove k_mem_domain_destroy and k_mem_domain_remove_thread as they've
been deprecated for at least 2 releases now.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
The internal function z_smp_reacquire_global_lock() has not used by
anywhere inside zephyr code, so remove it.
Fixes#33273.
Signed-off-by: Enjia Mai <enjiax.mai@intel.com>
The static device dependencies from devicetree are not the only ones
that might be present at runtime. Add API that allows visiting
required devices without assuming that handles for or pointers to them
can be accessed as a static contiguous sequence.
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
Wrap arch_sched_ipi() call in z_thread_abort() with ifdef checking for
hardware support of IPI.
Fixes#32723
Signed-off-by: Lauren Murphy <lauren.murphy@intel.com>
Previously, a racing write to the provided string could result
in up to CONFIG_THREAD_MAX_NAME_LEN-2 bytes after the end
of user-accessible memory being leaked into the thread name.
For now, make a temporary copy. In an ideal world this could
copy directly from userspace into the thread name, but that
violates the current vrfy / impl split.
Signed-off-by: James Harris <james.harris@intel.com>
The xtensa atomics layer was written with hand-coded assembly that had
to be called as functions. That's needlessly slow, given that the low
level primitives are a two-instruction sequence. Ideally the compiler
should see this as an inline to permit it to better optimize around
the needed barriers.
There was also a bug with the atomic_cas function, which had a loop
internally instead of returning the old value synchronously on a
failed swap. That's benign right now because our existing spin lock
does nothing but retry it in a tight loop anyway, but it's incorrect
per spec and would have caused a contention hang with more elaborate
algorithms (for example a spinlock with backoff semantics).
Remove the old implementation and replace with a much smaller inline C
one based on just two assembly primitives.
This patch also contains a little bit of refactoring to address the
scheme has been split out into a separate header for each, and the
ATOMIC_OPERATIONS_CUSTOM kconfig has been renamed to
ATOMIC_OPERATIONS_ARCH to better capture what it means.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Commit 6b84ab3830 ("kernel/sched: Adjust locking in z_swap()") moved
the call to arch_cohere_stacks() out of the scheduler lock while doing
some reorgnizing. On further reflection, this is incorrect. When
done outside the lock, the two arch_cohere_stacks() calls will race
against each other.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Zephyr docs state that timers will act as one-shot timers when started
with a period of K_NO_WAIT or K_FOREVER. However the code adjusting
period was setting K_FOREVER timeout ticks to 1 which caused the timer
to expire every tick. This adds a check to not adjust K_FOREVER periods
Signed-off-by: Eric Johnson <eric@liveathos.com>
pm_system_resume is always implemented when PM is enabled. There is no
need to have this weak function under an ifdef PM.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
pm_system_resume_from_deep_sleep is not implemented or used
anywhere. Just remove it and keep the code base cleaner.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
This function is useless and the state variable that it was
controlling is also not necessary because the same logic is being
handled by the variable post_ops_done.\
This reasonably simplifies idle thread logic.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
pm_system_suspend is called only from the idle thread and should
not be exported as a public API.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Previously, a k_sem_reset with any outstanding waiting threads would
result in the semaphore in an inconsistent state, with more threads
waiting in the wait_q than the count would indicate.
Explicitly -EAGAIN any waiting threads upon k_sem_reset, to
ensure safety here.
Signed-off-by: James Harris <james.harris@intel.com>
Currently there is no way to distinguish between a caller
explicitly asking for a semaphore with a limit that
happens to be `UINT_MAX` and a semaphore that just
has a limit "as large as possible".
Add `K_SEM_MAX_LIMIT`, currently defined to `UINT_MAX`, and akin
to `K_FOREVER` versus just passing some very large wait time.
In addition, the `k_sem_*` APIs were type-confused, where
the internal data structure was `uint32_t`, but the APIs took
and returned `unsigned int`. This changes the underlying data
structure to also use `unsigned int`, as changing the APIs
would be a (potentially) breaking change.
These changes are backwards-compatible, but it is strongly suggested
to take a quick scan for `k_sem_init` and `K_SEM_DEFINE` calls with
`UINT_MAX` (or `UINT32_MAX`) and replace them with `K_SEM_MAX_LIMIT`
where appropriate.
Signed-off-by: James Harris <james.harris@intel.com>
Due to the recent changes to scheduler z_find_first_thread_to_unpend
& z_remove_thread_from_ready_q are not used anymore. So removing the
dead code.
fixes: #32691
Signed-off-by: Spoorthy Priya Yerabolu <spoorthy.priya.yerabolu@intel.com>
While I'm in the idle code, let's clean this loop up. It was a really
bad #ifdef hell:
* Remove the CONFIG_TICKLESS_IDLE_THRESH logic (and the kconfig),
which never did anything but needlessly increase latency.
* Move the needed timeout logic from the main loop into
pm_save_idle(), which eliminates the special case for
!SYS_CLOCK_EXISTS.
Behavior (modulo that one kconfig) should be completely unchanged, and
now the inner part of the idle loop looks like:
while (true) {
(void) arch_irq_lock();
if (IS_ENABLED(CONFIG_PM)) {
pm_save_idle();
} else {
k_cpu_idle();
}
}
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
The removal of the abort handling also absconded with an IRQ lock that
is required for reliable operation in the idle loop. Put it back.
Once the idle loop has made a decision to enter idle, any interrupt
that arrives needs to be masked and delivered AFTER the system enters
idle. Otherwise we run the risk of races where the system accepts and
processes an interrupt that should have prevented idle, but then goes
to sleep anyway having already made the decision.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Now that the old API has been reimplemented with the new API remove
the old implementation and its tests.
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
Switch the default and clean up some test workarounds. This will enable
final conversions necessary to transition to the new API.
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
This commit provides a complete reimplementation of the work queue
infrastructure intended to eliminate the race conditions and feature
gaps in the existing implementation.
Both bare and delayable work structures are supported. Items can be
submitted; delayable items can be scheduled for submission at a future
time. Items can be delayed, queued, and running all at the same time.
A running item can also be canceling.
The new implementation:
* replaces "pending" with "busy" which identifies the active states;
* supports canceling delayed and submitted items;
* prevents resubmission of a item being canceled until cancellation
completes;
* supports waiting for cancellation to complete;
* supports flushing a work item (waiting for the last submission to
complete without preventing resubmission);
* supports waiting for a queue to drain (only allows resubmission from
the work thread);
* supports stopping a work queue in conjunction with draining it;
* prevents handler-reentrancy during resubmission.
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
Attempts to reimplement the existing work API using a new work
implementation failed, primarily due to heavy use of whitebox testing
in validating the original API. Add a temporary Kconfig that will
select between the two implementations so we can use the same
identifiers but select which implementation they reference.
This commit just adds the selection infrastructure and uses it to
conditionalize the existing implementation in anticipation of the new
one in the next commit.
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
These functions are a subset of proposed public APIs to clean up
several issues related to safely handling waking of threads. They
have been made private as they interface may change, but their use
will simplify the reimplementation of the k_work functionality.
See: https://github.com/zephyrproject-rtos/zephyr/pull/29668
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
Several internal APIs wrote thread attributes (return value, mainly)
_after_ calling `z_ready_thread`. This is unsafe, at least in SMP,
because another core could have already picked up and run the thread.
Fixes#32800.
Signed-off-by: James Harris <james.harris@intel.com>
`z_impl_k_yield` unlocked sched_spinlock, only to lock it again
immediately, do a little bit more work, then unlock it again.
This causes performance issues on SMP, where `sched_spinlock`
is often fairly highly contended and cores often end up spinning
for quite a while waiting to retake the lock in `z_swap_unlocked`.
Instead directly pass the spinlock key to `z_swap` and avoid the
extra lock+unlock.
Signed-off-by: James Harris <james.harris@intel.com>
`z_is_t1_higher_prio_than_t2` was being called twice in both the
context-switch fastpath and in `z_priq_rb_lessthan`, just to
dealing with priority ties. In addition, the API was error-prone
(and too much in the fastpath to be able to assert its invarients)
- see also #32710 for a previous example of this API breaking
and returning a>b but also b>a.
Replacing this with a direct 3-way comparison `z_cmp_t1_prio_with_t2`
sidesteps most of these issues. There is still a concern that
`sgn(z_cmp_t1_prio_with_t2(a,b)) != -sgn(z_cmp_t1_prio_with_t2(b,a))`
but I don't see any way to alleviate this aside from adding an
assert to the fastpath.
Signed-off-by: James Harris <james.harris@intel.com>
Previously two tasks with the same deadline and priority would
always have `z_is_t1_higher_prio_than_t2` `true` in both directions.
This is logically inconsistent, and results in `k_yield` not actually
yielding between identical threads.
Signed-off-by: James Harris <james.harris@intel.com>
Add a newer, much smaller and simpler implementation of abort and
join. No need to involve the idle thread. No need for a special code
path for self-abort. Joining a thread and waiting for an aborting one
to terminate elsewhere share an implementation. All work in both
calls happens under a single locked path with no unexpected
synchronization points.
This fixes a bug with the current implementation where the action of
z_sched_single_abort() was nonatomic, releasing the lock internally at
a point where the thread to be aborted could self-abort and confuse
the state such that it failed to abort at all.
Note that the arm32 and native_posix architectures, which have their
own thread abort implementations, now see a much simplified
"z_thread_abort()" internal API.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
THIS COMMIT DELIBERATELY BREAKS BISECTABILITY FOR EASE OF REVIEW.
SKIP IF YOU LAND HERE.
Remove the existing implementatoin of k_thread_abort(),
k_thread_join(), and the attendant facilities in the thread subsystem
and idle thread that support them.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
This function would correctly suppress attempts to set timeouts that
were too soon for the driver or farther out than what was already set,
but when it actually set the timeout it would use the requested value
and not clamp it to the minimum of it and the current timeout
expiration, leading to "too-long" timeouts being set at the driver.
In uniprocessor configurations, that turns out to have been benign
because something else would always come back along when timeout state
changed and fix the broken value before the expiration.
But in SMP, this opens up races. For example, the idle thread on one
CPU can see that there are no active threads and schedule a maximum
value timeout at the same time as the other thread adds a new timeout
that expects a near-term expiration. The broken code here would see
that the new timeout exists, decide that yes it needs to override, but
then set the K_TICKS_FOREVER value it got from the idle thread!
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
When the kernel is TICKLESS, timeouts are set as needed, and drivers
all have some minimum amount of time before which they can reliably
schedule an interrupt. When this happens, drivers will kick the
requested interrupt out by one tick. This means that it's not
reliably possible to get a timeout set for "one tick in the
future"[1].
And attempting to do that is dangerous anyway. If the driver will
delay a one-tick interrupt, then code that repeatedly tries to
schedule an imminent interrupt may end up in a state where it is
constantly pushing the interrupt out into the future, and timer
interrupts stop arriving! The timeout layer actually has protection
against this case.
Finally getting to the point: in recent changes, the timeslice layer
lost its integration with the "imminent" test in the timeout code, so
it's now able to run into this situation: very rapidly context
switching code (or rapidly arriving interrupts) will have the effect
of infinitely[2] delaying timeouts and stalling the whole timeout
subsystem.
Don't try to be fancy. Just clamp timeslice duration such that a
slice is 2 ticks at minimum and we'll never hit the problem. Adjust
the two tests that were explicitly requesting very short slice rates.
[1] Of course, the tradeoff is that the tick rate can be 100x higher
or more, so on balance tickless is a huge win.
[2] Actually it only lasts until a 31 bit signed rollover in the HPET
cycle count in practice.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Recent work to normalize use of the thread QUEUED state bit means that
we never attempt to remove unqueued threads from the low-level run
queue. So the old workaround for SWAP_NONATOMIC that was trying to
detect this condition isn't necessary anymore.
Which is serendipitous, because it was written to encode some very
specific logic about the circumstances where _current could be
dequeued that I'd like to be able to break.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
This is part of the scheduler API, and was always just a synchronized
wrapper around the internal ready_thread() function. But where the
internal users seem to be careful not to call it on threads that are
not known to be already queued or running, the general users in the
IPC code seem to be less strict.
Add a simple test to detect the case where a thread is already
running. Right now this just loops over the array of CPUs, so is O(N)
in the CPU count even though N is never more than four for us
currently. But this is possible without modifying data structures. A
more scalable way to do this if we ever need to run on very parallel
systems would be to use another state bit for RUNNING, or to keep a
backpointer in the thread struct to the CPU it's running on, etc...
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Swap was originally written to use the scheduler lock just to select a
new thread, but it would be nice to be able to rely on scheduler
atomicity later in the process (in particular it would be nice if the
assignment to cpu.current could be seen atomically). Rework the code
a bit so that swap takes the lock itself and holds it until just
before the call to arch_switch().
Note that the local interrupt mask has always been required to be held
across the swap, so extending the lock here has no effect on latency
at all on uniprocessor setups, and even on SMP only affects average
latency and not worst case.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Aborted threads will cancel their timeouts, but the timeout subsystem
isn't protected under the same lock so it's possible for a timeout to
fire just as a thread is being aborted and wake it up unexpectedly.
Check the state before blowing anything up.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
This got missed, leaving garbage there for restarted threads to trip
on. Actually I see multiple uninitialized fields, which seems odd.
This code deserves some rework, thread initialization isn't a
performance path and we should probably be zeroing the struct out.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Remove duplication in the code by moving macro LOCKED() to the correct
kernel_internal.h header.
Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
This adds a new kconfig CONFIG_SRAM_OFFSET to specify the offset
from beginning of SRAM where the kernel begins. On x86 and
PC compatible platforms, the first 1MB of RAM is reserved and
Zephyr should not link anything there. However, this 1MB still
needs to be mapped by the MMU to access various platform related
information. CONFIG_SRAM_OFFSET serves similar function as
CONFIG_KERNEL_VM_OFFSET and is needed for proper phys/virt
address translations.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The Z_BOOT_VIRT_TO_PHYS() and Z_BOOT_PHYS_TO_VIRT() address
translation macros are flipped in their calculations.
The calculation is supposed to be:
virt = phys + ((KERNEL_VM_BASE + KERNEL_VM_OFFSET) -
SRAM_BASE_ADDRESS)
So fix the them.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The computation was using the already-adjusted input value that
assumed relative timeouts and not the actual argument the user passed.
Absolute timeouts were consistently waking up one tick early.
Fixes#32499
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Following the idiom used for system calls, add script support to read
the initial application binary to identify which devices are defined,
and to use their offset in the device array as their unique handle
rather than the externally-defined ordinal from devicetree. The
device dependency arrays are updated to use these handles.
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
Move the busy status from a global atomic bit sequence to atomic flags
in the device PM state. While this temporarily adds 4 bytes to each
PM structure the whole device PM infrastructure will be refactored and
it's likely the extra memory can be recovered.
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
Separate the state indicator of whether the initialization function
has been invoked from the success or failure of the initialization.
This allows precise confirmation that the device is ready (i.e. it has
been initialized, and that initialization succeeded).
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
This avoids the need for distinct object that uses flash to store its
initializer. Instead the state is initialized when the kernel is
starting up, before anything can reference it. In future refactoring
the PM state could be accessed directly without storing an extra
pointer in the static device state.
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
Initialize all device objects in a batch before invoking any code that
might try to reference data in them. This eliminates a race condition
enabled by the ability to resolve a device structure at build time,
and reference it from one device's initialization routine before the
device itself has been initialized.
While the device is pulled from the sys_init records rather than
static devices, all in-tree init_entry records that are associated
with devices are produced via Z_DEVICE_DEFINE(), so there should be no
static devices that would be missed by instead iterating over the
device records.
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
Some recent changes exposed some common "arch_switch() anti-patterns"
in various architectures. The documentation technically described
this all correctly, but probably wasn't as clear as it should have
been. Rewrite, making clear exactly what needs to happen and how the
fields should be interpreted.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
It was possible with pathological timing (see below) for the scheduler
to pick a cycle of threads on each CPU and enter the context switch
path on all of them simultaneously.
Example:
* CPU0 is idle, CPU1 is running thread A
* CPU1 makes high priority thread B runnable
* CPU1 reaches a schedule point (or returns from an interrupt) and
decides to run thread B instead
* CPU0 simultaneously takes its IPI and returns, selecting thread A
Now both CPUs enter wait_for_switch() to spin, waiting for the context
switch code on the other thread to finish and mark the thread
runnable. So we have a deadlock, each CPU is spinning waiting for the
other!
Actually, in practice this seems not to happen on existing hardware
platforms, it's only exercisable in emulation. The reason is that the
hardware IPI time is much faster than the software paths required to
reach a schedule point or interrupt exit, so CPU1 always selects the
newly scheduled thread and no deadlock appears. I tried for a bit to
make this happen with a cycle of three threads, but it's complicated
to get right and I still couldn't get the timing to hit correctly. In
qemu, though, the IPI is implemented as a Unix signal sent to the
thread running the other CPU, which is far slower and opens the window
to see this happen.
The solution is simple enough: don't store the _current thread in the
run queue until we are on the tail end of the context switch path,
after wait_for_switch() and going to reach the end in guaranteed time.
Note that this requires changing a little logic to handle the yield
case: because we can no longer rely on _current's position in the run
queue to suppress it, we need to do the priority comparison directly
based on the existing "swap_ok" flag (which has always meant
"yielded", and maybe should be renamed).
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
The QUEUED state flag was managed separately from the run queue
insertion/deletion, and the logic (while AFAICT perfectly correct) was
tangled in a few places trying to keep them in sync. Put the
management of both behind a queue_thread()/dequeue_thread() API for
clarity. The ALWAYS_INLINE usage seems to be working to get the
compiler to condense the resulting multiple assignments. No behavior
change.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
The "null out the switch handle and put it back" code in the swap
implementation is a holdover from some defensive coding (not wanting
to break the case where we picked our current thread), but it hides a
subtle SMP race: when that field goes NULL, another CPU that may have
selected that thread (which is to say, our current thread) as its next
to run will be spinning on that to detect when the field goes
non-NULL. So it will get the signal to move on when we revert the
value, when clearly we are still running on the stack!
In practice this was found on x86 which poisons the switch context
such that it crashes instantly.
Instead, be firm about state and always set the switch handle of a
currently running thread to NULL immediately before it starts running:
right before entering arch_switch() and symmetrically on the interrupt
exit path.
Fixes#28105
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Some legacy spots in our IPC layer (legally) pass a NULL wait queue to
pend(). Allow this in the coherence assertion.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
The poll code uses a dummy wait queue so the threads have something to
block on, but the previous coherence pass (which rearranged things to
put the _poller data elsewhere) missed that this was on the stack,
which is not allowed. It actually has no use except as a list, so
make it a global static instead.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
The z_swap_unlocked() function used a dummy spinlock for simplicity.
But this runs afouls of checking for stack-resident spinlocks
(forbidden when KERNEL_COHERENCE is set). And it's executing needless
code to release the lock anyway. Replace with a compile time NULL,
which will improve performance, correctness and code size.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
The two calls to unpend a thread from a wait queue were inexplicably*
unsynchronized, as James Harris discovered. Rework them to call the
lowest level primities so we can wrap the process inside the scheduler
lock.
Fixes#32136
* I took a brief look. What seems to have happened here is that these
were originally synchronized via an implicit from an outer caller
(remember the original Uniprocessor irq_lock() API is a recursive
lock), and they were mostly implemented in terms of middle-level
calls that were themselves locked. So those got ported over to the
newer spinlock but the outer wrapper layer got forgotten.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
This lets the linker tell us what kind of alignment is required
for both tdata and tbss data when copying them into stack.
If they are not aligned as expected by the toolchain, generated
code would be accessing incorrect location for thread variables.
Fixes#32015
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The linker script defines `z_mapped_size` as follows:
```
z_mapped_size = z_mapped_end - z_mapped_start;
```
This is done with the belief that precomputed values at link time will
make the code smaller and faster.
On Aarch64, symbol values are relocated and loaded relative to the PC
as those are normally meant to be memory addresses.
Now if you have e.g. `CONFIG_SRAM_BASE_ADDRESS=0x2000000000` then
`z_mapped_size` might still have a reasonable value, say 0x59334.
But, when interpreted as an address, that's very very far from the PC
whose value is in the neighborhood of 0x2000000000. That overflows the
4GB relocation range:
```
kernel/libkernel.a(mmu.c.obj): in function `z_mem_manage_init':
kernel/mmu.c:527:(.text.z_mem_manage_init+0x1c):
relocation truncated to fit: R_AARCH64_ADR_PREL_PG_HI21
```
The solution is to define `Z_KERNEL_VIRT_SIZE` in terms of
`z_mapped_end - z_mapped_start` at the source code level. Given this
is used within loops that already start with `z_mapped_start` anyway,
the compiler is smart enough to combine the two occurrences and
dispense with a size counter, making the code effectively
slightly better for all while avoiding the Aarch64 relocation
overflow:
```
text data bss dec hex filename
1216 8 294936 296160 484e0 mmu.c.obj.arm64.before
1212 8 294936 296156 484dc mmu.c.obj.arm64.after
1110 8 9244 10362 287a mmu.c.obj.x86-64.before
1106 8 9244 10358 2876 mmu.c.obj.x86-64.after
```
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
The SYS_CLOCK_TICKS_PER_SEC default may depend on the kernel config
for tickless, rather than the capability.
Signed-off-by: Martin Åberg <martin.aberg@gaisler.com>
Activating K_FP_REGS flags introduces stack memory
overhead for the main thread in Cortex-M architecture.
Several ARM platforms experience main thread stack
overflows when building with FPU_SHARING=y.
Enabling FPU sharing in main thread should not be
the default configuration. Users are welcome to
enable FP sharing on the main thread in the
application code, in main().
This reverts commit 8453a73ede.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
The call to arch_mem_coherent() inside spinlock.h
when spinlock validation and memory coherence enabled
is causing build error as spinlock.h does not include
kernel_arch_func.h directly. However, simply including
that file does not work either as this creates
the chicken-or-egg in the chain of include files.
In order to make spin validation work with kernel
coherence enabled, a separate function is created
to break the circular dependencies of include files.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
There was an edge case in the timeout handling (exposed by, but not
strictly related to, the recent timeslice fix): the next_timeout()
computation would include time slice expiration as a clamp on the
result, but this would be invoked also on the z_set_timeout_expiry()
path which gets hooked on entry to a new thread which is needed to set
the timeout in the first place. So if no other timer interrupt was
scheduled, it was possible to miss the first timeslice interrupt after
thread scheduling.
The explanation is much longer than the fix (use <= as the comparator
instead of <).
In practice this was only being hit in the existing test suite on
riscv miv running under renode using non-default clock rates.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Fix an edge case that snuck in with the recent fix: if timeslicing is
enabled, the CPU's slice_ticks will be zero, and thus match a timeout
object's dticks value of zero, and thus get suppressed (because "we
already have a timeout scheduled for that") incorrectly.
Fixes#31789
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
There are more and more tests that fail due to stackoverflow.
Increasing MAIN_STACK_SIZE to fix those issues.
Signed-off-by: Alexandre Bourdiol <alexandre.bourdiol@st.com>
Time slices don't have a timeout struct associated and stored in
timeout_list. Time slice timeout is direct programmed in the system
clock and tracked in _current_cpu->slice_ticks.
There is one issue where the time slice timeout can be missed because
the system clock is re-programmed to a longer timeout. To this happens,
it is only necessary that the timeout_list is empty (any timeout set)
and a new timeout longer than remaining time slice is set. This is cause
because z_add_timeout does not check for the slice ticks.
The following example spots the issue:
K_THREAD_STACK_DEFINE(tstack, STACK_SIZE);
K_THREAD_STACK_ARRAY_DEFINE(tstacks, NUM_THREAD, STACK_SIZE);
K_SEM_DEFINE(sema, 0, NUM_THREAD);
static inline void spin_for_ms(int ms)
{
uint32_t t32 = k_uptime_get_32();
while (k_uptime_get_32() - t32 < ms) {
}
}
static void thread_time_slice(void *p1, void *p2, void *p3)
{
printk("thread[%d] - Before spin\n", (int)(uintptr_t)p1);
/* Spinning for longer than slice */
spin_for_ms(SLICE_SIZE + 20);
/* The following print should not happen before another
* same priority thread starts.
*/
printk("thread[%d] - After spinning\n", (int)(uintptr_t)p1);
k_sem_give(&sema);
}
void main(void)
{
k_tid_t tid[NUM_THREAD];
struct k_thread t[NUM_THREAD];
uint32_t slice_ticks = k_ms_to_ticks_ceil32(SLICE_SIZE);
int old_prio = k_thread_priority_get(k_current_get());
/* disable timeslice */
k_sched_time_slice_set(0, K_PRIO_PREEMPT(0));
for (int j = 0; j < 2; j++) {
k_sem_reset(&sema);
/* update priority for current thread */
k_thread_priority_set(k_current_get(), K_PRIO_PREEMPT(j));
/* synchronize to tick boundary */
k_usleep(1);
/* create delayed threads with equal preemptive priority */
for (int i = 0; i < NUM_THREAD; i++) {
tid[i] = k_thread_create(&t[i], tstacks[i], STACK_SIZE,
thread_time_slice, (void *)i, NULL,
NULL, K_PRIO_PREEMPT(j), 0,
K_NO_WAIT);
}
/* enable time slice (and reset the counter!) */
k_sched_time_slice_set(SLICE_SIZE, K_PRIO_PREEMPT(0));
/* Spins for while to spend this thread time but not longer */
/* than a slice. This is important */
spin_for_ms(100);
printk("before sleep\n");
/* relinquish CPU and wait for each thread to complete */
k_sleep(K_TICKS(slice_ticks * (NUM_THREAD + 1)));
for (int i = 0; i < NUM_THREAD; i++) {
k_sem_take(&sema, K_FOREVER);
}
/* test case teardown */
for (int i = 0; i < NUM_THREAD; i++) {
k_thread_abort(tid[i]);
}
/* disable time slice */
k_sched_time_slice_set(0, K_PRIO_PREEMPT(0));
}
k_thread_priority_set(k_current_get(), old_prio);
}
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Some arches like x86 need all memory mapped so that they can
fetch information placed arbitrarily by firmware, like ACPI
tables.
Ensure that if this is the case, the kernel won't accidentally
clobber it by thinking the relevant virtual memory is unused.
Otherwise this has no effect on page frame management.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
If we evict enough pages to completely fill the backing store,
through APIs like k_mem_map(), z_page_frame_evict(), or
z_mem_page_out(), this will produce a crash the next time we
try to handle a page fault.
The backing store now always reserves a free storage location
for actual page faults.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
This will enable testing of the implementation until the
critical set of pages is identified and known to the
kernel.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Implement runtime APIs for pinning, paging in, and evicting
memory, as well as the page fault hook called from architecture
code.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Architecture layer hooks for demand paging. See
doxygen for these API definitions for more details.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Page tables created at build time may not include the
gperf data at the very end of RAM. Ensure this is mapped
properly at runtime to work around this.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Pre-allocation of paging structures is now required, such that
no allocations are ever needed when mapping memory.
Instantiation of new memory domains may still require allocations
unless a common page table is used.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Allows applications to increase the data space available to Zephyr
via anonymous memory mappings. Loosely based on mmap().
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
The strategy used in z_heap_aligned_alloc() was to allocate an extra
align-sized memory block for storing a pointer to the memory heap.
This is wasteful in terms of memory usage when alignment is larger
than a pointer width. A loop is needed to find the initial memory
start when freeing it which isn't optimal either.
Instead, let's have sys_heap_aligned_alloc() rewind a pointer after
it is aligned to make just enough room for storing our heap reference.
This way the heap reference is always located immediately before the
aligned memory and any unused memory is returned to the heap.
The rewind and alignment values may coincide in which case only
the alignment is necessary anyway.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Remove conditionals (PM_DEEP_SLEEP_STATES and PM_SLEEP_STATES) from
power management code. Now these features are always available when
power management is enabled.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Migrate the whole pm subsystem to use new power states information
from power_state.h and get states and residency properties from
device tree.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
The internal API to measure time until a delay expires does not modify
the referenced timeout. Make the functions that call it take pointers
to const objects, so that they can be used with pointer to
const-qualified containers.
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
This removes the z_ prefix those (functions, enums, etc.) that
are being used outside the coredump subsys. This aligns better
with the naming convention.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
If we evict enough pages to completely fill the backing store,
through APIs like k_mem_map(), z_page_frame_evict(), or
z_mem_page_out(), this will produce a crash the next time we
try to handle a page fault.
The backing store now always reserves a free storage location
for actual page faults.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
This will enable testing of the implementation until the
critical set of pages is identified and known to the
kernel.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Implement runtime APIs for pinning, paging in, and evicting
memory, as well as the page fault hook called from architecture
code.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Architecture layer hooks for demand paging. See
doxygen for these API definitions for more details.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Page tables created at build time may not include the
gperf data at the very end of RAM. Ensure this is mapped
properly at runtime to work around this.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Pre-allocation of paging structures is now required, such that
no allocations are ever needed when mapping memory.
Instantiation of new memory domains may still require allocations
unless a common page table is used.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Allows applications to increase the data space available to Zephyr
via anonymous memory mappings. Loosely based on mmap().
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Following the idiom used for system calls, add script support to read
the initial application binary to identify which devices are defined,
and to use their offset in the device array as their unique handle
rather than the externally-defined ordinal from devicetree. The
device dependency arrays are updated to use these handles.
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
The only two supported operations for data caches in the cache framework
are currently arch_dcache_flush() and arch_dcache_invd().
This is quite restrictive because for some architectures we also want to
control i-cache and in general we want a finer control over what can be
flushed, invalidated or cleaned. To address these needs this patch
expands the set of operations that can be performed on data and
instruction caches, adding hooks for the operations on the whole cache,
a specific level or a specific address range.
Signed-off-by: Carlo Caione <ccaione@baylibre.com>
register_event always returns 0, so the conditional will
always take the first branch and code in the else part
is never reached.
Fixes#31282
Signed-off-by: Ningx Zhao <ningx.zhao@intel.com>
1. Exclude the CODE UNREACHABLE line while generating coverage report.
2. Exclude the memory domain deprecated API when calculating code
coverage.
Signed-off-by: Enjia Mai <enjiax.mai@intel.com>
First, the maximum heap size must fit in 31 bits worth of chunks
because the internal 32-bit field holding the size is shared with
the `used` bit.
Then the mention of a 256-byte block in the doc is no longer
relevant. That pertained to the previous allocator implementation.
And ditto for the HEAP_MEM_POOL_MIN_SIZE kconfig option.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Needing to check the current cycle time (which involves a spinlock and
register read on most architectures) is wasteful in the scheduler
priority predicate, which is a hot path. If we "burn" one bit of
precision (and document the rule), we can do the comparison without
knowing the current time.
2^31 cycles is still far longer than a live deadline thread in any
legitimate realtime app should ever live before being scheduled.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Adds a linker section for Cortex-M instruction tightly coupled memory
(ITCM), similar to the existing section for DTCM. A new executable MPU
region is not added as there isn't currently a need to make this section
accessible to user mode. This section can be enabled by setting a device
tree chosen node zephyr,itcm.
Signed-off-by: Maureen Helm <maureen.helm@nxp.com>
This allows allocating dynamic kernel objects with memory alignment
requirements. The first candidate is for thread objects where,
on some architectures, it must be aligned for saving/restoring
registers.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
PM depends on SYS_CLOCK_EXISTS in Kconfig but several boards have
Kconfig overrides that allow the dependency to be ignored, so
CONFIG_PM=y even though CONFIG_SYS_CLOCK_EXISTS=n. Fix the code so
that the true dependency is reflected in the generated code.
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
This change adds z_heap_aligned_alloc() and k_aligned_alloc()
and changes z_heap_malloc() and k_malloc() to be small wrappers around
the aligned variants.
Fixes#29519
Signed-off-by: Christopher Friedt <chrisfriedt@gmail.com>
Ticks should be assigned directly to timeout value in case of
CONFIG_LEGACY_TIMEOUT_API=y, just as they were before referenced patch.
Fixes: 7a815d5d99 ("kernel: sched: Use k_ticks_t in z_tick_sleep")
Signed-off-by: Marcin Niestroj <m.niestroj@grinn-global.com>
Renamed to make its semantics clearer; this function maps
*physical* memory addresses and is not equivalent to
posix mmap(), which might confuse people.
mem_map test case remains the same name as other memory
mapping scenarios will be added in the fullness of time.
Parameter names to z_phys_map adjusted slightly to be more
consistent with names used in other memory mapping functions.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Inside the idle loop, in some configuration, IRQ is unlocked and
then immediately locked again. There is a side effect:
1. IRQ is unlocked in middle of the loop.
2. Another thread (A) can now run so idle thread is un-scheduled.
3. Thread A runs to its end and going through the thread
self-abort path.
4. Idle thread is rescheduled again, and continues to run
the remaining loop when it eventuall calls k_cpu_idle().
The "pending abort" path is not being executed on thread A
at this point.
5. Now, thread A is suspended, and the CPU is in idle waiting
for interrupts (e.g. timeouts).
6. Thread B is waiting to join on thread A. Since thread A has
not been terminated yet so thread B is waiting until
the idle thread runs again and starts executing from
the beginning of while loop.
7. Depending on how many threads are running and how active
the platform is, idle thread may not run again for a while,
resulting in thread B appearing to be stuck.
To avoid this situation, the unlock/lock pair in middle of
the loop is removed so no rescheduling can be done mid-loop.
When there is no thread abort pending, it simply locks IRQ
and calls k_cpu_idle(). This is almost identical to the idle
loop before the thread abort code was introduced (except
the check for cpu->pending_abort).
Fixes#30573
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
In order to release irq_offload semaphore outside kernel/thread.c, we
make it visible by modifying it non-static under ztest. This would be
needed such as when call irq_offload() to enter interrupt context and
a fatal error happened, then you have to release it in your fatal
handler, or the irq_offload will still be locked and no longer be
using again.
Signed-off-by: Enjia Mai <enjiax.mai@intel.com>
Cleanup code for power management and remove some duplication and
isolate power management code from the kernel code.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
- Remove SYS_ prefix
- shorten POWER_MANAGEMENT to just PM
- DEVICE_POWER_MANAGEMENT -> PM_DEVICE
and use PM_ as the prefix for all PM related Kconfigs
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
k_heap did not have an aligned alloc function, even though
this is supported by the internal sys_heap.
Signed-off-by: Maximilian Bachmann <m.bachmann@acontis.com>
