Commit graph

3196 commits

Author SHA1 Message Date
Andy Ross
6c82745cb3 kernel/poll: Use a separate thread backpointer for triggered work
The triggered work scheme uses a trick where it overloads the thread
pointer field of the struct poller (which normally stores the ID of
the thread that is blocked in k_poll()) to be able to find the work
queue to which it will submit.

Give it its own pointer field to break this false dependency.

Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2020-11-11 10:59:52 -05:00
Andy Ross
0c7af40d61 kernel/poll: Make the poll callback a single-byte enum instead
Somewhat weirdly, k_poll() can do one of two things when the set of
events is signaled: it can wake up a sleeping thread, or it can submit
an unrelated work item to a work queue.  The difference in behaviors
is currently captured by a callback, but as there are only two it's
cleaner to put this into a "mode" enumerant.  That also shrinks the
size of the data such that the poller struct can be moved somewhere
other than the calling stack.

Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2020-11-11 10:59:52 -05:00
Andrew Boie
ea6e4ad098 kernel: support non-identity RAM mapping
Some platforms may have multiple RAM regions which are
dis-continuous in the physical memory map. We really want
these to be in a continuous virtual region, and we need to
stop assuming that there is just one SRAM region that is
identity-mapped.

We no longer use CONFIG_SRAM_BASE_ADDRESS and CONFIG_SRAM_SIZE
as the bounds of kernel RAM, and no longer assume in the core
kernel that these are identity mapped at boot.

Two new Kconfigs, CONFIG_KERNEL_VM_BASE and
CONFIG_KERNEL_RAM_SIZE now indicate the bounds of this region
in virtual memory.

We are currently only memory-mapping physical device driver
MMIO regions so we do not need virtual-to-physical calculations
to re-map RAM yet. When the time comes an architecture interface
will be defined for this.

Platforms which just have one RAM region may continue to
identity-map it.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-11-09 20:19:13 -05:00
Trond Einar Snekvik
86c793af3f sys: util: Replace MIN(MAX(a, b), c) with CLAMP
Replaces all existing variants of value clamping with the MIN and MAX
macros with the CLAMP macro.

Signed-off-by: Trond Einar Snekvik <Trond.Einar.Snekvik@nordicsemi.no>
2020-11-05 12:12:17 +01:00
Maximilian Bachmann
b3c5fe6720 kernel: return error instead of misaligned k_thread object
k_object_alloc(K_OBJ_THREAD) returns a usable struct k_thread pointer.
This pointer is 4 byte aligned. On x86 and x86_64 struct _thread_arch
has a member which requires alignment. Since this is currently not
supported k_object_alloc(K_OBJ_THREAD) now returns an error instead of
a misaligned pointer.

Signed-off-by: Maximilian Bachmann <m.bachmann@acontis.com>
2020-11-04 14:03:43 -08:00
Daniel Leung
2d152ab019 kernel: limit thread local storage to Zephyr SDK
Toolchains other than Zephyr SDK may not support generating
code with thread local storage. So limit TLS to Zephyr SDK
for now, and only enable TLS on other toolchains as needed.

Fixes #29541

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2020-10-30 15:59:06 +01:00
iva kik
687f799b7f kernel/queue: fix queue append/get race
k_queue_append and k_queue_alloc_append are in a race
    condition.

    Scenario:
    Using either of the above mentioned functions from a
    preemptive context, can lead to the element being
    appended to the queue to be lost. The element loss happens
    when the k_queue_append is interrupted at a critical point,
    when the list only contains one element and k_queue_get is
    called, before the k_queue_append is allowed to complete.

    Fix:
    Move sys_sflist_peek_tail inside queue_insert(). Add additional
    bool paramenter to queue_insert(), to indicate append/prepend
    operation.
    Fixes #29257

Signed-off-by: iva kik <megatheriumiva@gmail.com>
2020-10-28 17:01:41 +01:00
Andrew Boie
0e30c6a0fa kernel: wipe TLS when dropping to user mode
For threads that run in supervisor mode for some time before
synchronously dropping to user mode, re-initialize the TLS
area to prevent leakage of potentially sensitive information.

We did this already for CONFIG_THREAD_USERSPACE_LOCAL_DATA
but not the new CONFIG_THREAD_LOCAL_STORAGE.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-10-26 13:06:16 +01:00
Daniel Leung
62cf1960ad kernel: support using thread local storage for errno
This enables storing errno in the thread local storage area.
With this enabled, a syscall to access errno can be avoided
when userspace is also enabled.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2020-10-24 10:52:00 -07:00
Daniel Leung
388725870f arm: cortex_m: add support for thread local storage
Adds the necessary bits to initialize TLS in the stack
area and sets up CPU registers during context switch.

Note that since Cortex-M does not have the thread ID or
process ID register needed to store TLS pointer at runtime
for toolchain to access thread data, a global variable is
used instead.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2020-10-24 10:52:00 -07:00
Daniel Leung
02b20351cd kernel: add common bits to support TLS
This adds the common struct fields and functions to support
the implementation of thread local storage in individual
architecture. This uses the thread stack to store TLS data.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2020-10-24 10:52:00 -07:00
Daniel Leung
240beb42af kconfig: add configs for thread local storage
Add kconfigs to indicate whether an architecture has support
for thread local storage (TLS), and to enable TLS in kernel.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2020-10-24 10:52:00 -07:00
Andrew Boie
ad110d495e kernel: fatal: check if _current is NULL
Arches which have custom swap to main can have _current be
NULL very, very early in the boot process. Check this to
avoid an infinite loop of fatal errors.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-10-24 12:54:32 -04:00
Andrew Boie
f5c3fc498b kernel: add arch_mem_unmap() interface
The core kernel does not use this yet, but it will be later used
as part of infrastructure for memory-mapping stacks, as detailed
in #28899.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-10-23 22:02:47 -04:00
Carlo Caione
47c592cd32 mmu: Fix mapping_pos calculation
In the MMU code mapping_pos is miscalculated when for example
SRAM_BASE_ADDRESS==0x40000000 and KERNEL_VM_SIZE==0xc0000000 getting a
mapping_pos of 0x0.

The problem is that we must cast the two values to uintptr_t before
casting the result to avoid the rollover to 0.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-10-23 16:24:04 -04:00
Andrew Boie
933b420235 kernel: add context pointer to thread->fn_abort
For compatibility layers like CMSIS where thread objects
are drawn from a pool, provide a context pointer to the
exited thread object so it may be freed.

This is somewhat obscure and has no supporting APIs or
overview documentation and should be considered a private
kernel feature. Applications should really be using
k_thread_join() instead.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-10-22 23:32:37 -04:00
Andrew Boie
fa6db61eb2 userspace: do nothing if added to same domain
If we call k_mem_domain_add_thread() to a memory domain
the thread already belongs to, do nothing.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-10-22 16:47:07 -07:00
Anas Nashif
d2c71796af kernel: document k_sleep with K_FOREVER
When calling k_sleep with K_FOREVER as the timeout value, we consider
this as a suspend call.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-10-22 07:00:15 -04:00
Anas Nashif
081605ee23 kernel: do not queue a thread that is already queued
Do not add a thread to the run queue if it was already added.

Fixes #29244

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-10-22 07:00:15 -04:00
Anas Nashif
bf69afcdae kernel: only resume suspended threads
Do not try to resume a thread that was not suspended.

Fixes #28694

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-10-22 07:00:15 -04:00
Andrew Boie
52bf482471 userspace: fix k_mem_domain_init()
The synchronous arch hook wasn't being invoked when
k_mem_domain_init() was called with a partition list.
This could result with regions not being programmed in
the page tables as expected.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-10-21 10:20:49 -07:00
Carlo Caione
f161223637 userspace: Fix thread index type in z_thread_perms_all_clear()
The type for the thread index returned by thread_index_get() must be
casted to int when comparing with (-1). Directly using uintptr_t is
breaking the ARMv8 implementation where where the check (index != -1) is
verified also when no thread index is returned.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-10-21 08:00:35 -04:00
Andy Ross
a8d5437799 soc/xtensa: Misc. checkpatch fixups
Code style fixes.  Kept separate from the original changes to permit
easier rebasing.

Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2020-10-21 06:38:53 -04:00
Andy Ross
4a8b3d194c kernel/poll: Mark incompatibility with KERNEL_COHERENCE
The k_poll implementation places a struct _poller on the stack and
shares it with other threads, which is incompatible with the
KERNEL_COHERENCE model of cached stacks.

Make this a hard build failure instead of a kconfig dependency for
clarity.  The failures if a user actually enables both are subtle and
difficult to debug.

Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2020-10-21 06:38:53 -04:00
Andy Ross
f6d32ab0a4 kernel: Add cache coherence management framework
Zephyr SMP kernels need to be able to run on architectures with
incoherent caches.  Naive implementation of synchronization on such
architectures requires extensive cache flushing (e.g. flush+invalidate
everything on every spin lock operation, flush on every unlock!) and
is a performance problem.

Instead, many of these systems will have access to separate "coherent"
(usually uncached) and "incoherent" regions of memory.  Where this is
available, place all writable data sections by default into the
coherent region.  An "__incoherent" attribute flag is defined for data
regions that are known to be CPU-local and which should use the cache.
By default, this is used for stack memory.

Stack memory will be incoherent by default, as by definition it is
local to its current thread.  This requires special cache management
on context switch, so an arch API has been added for that.

Also, when enabled, add assertions to strategic places to ensure that
shared kernel data is indeed coherent.  We check thread objects, the
_kernel struct, waitq's, timeouts and spinlocks.  In practice almost
all kernel synchronization is built on top of these structures, and
any shared data structs will contain at least one of them.

Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-10-21 06:38:53 -04:00
Andrew Boie
348a0fda62 userspace: make mem domain lock non-static
Strictly speaking, any access to a mem domain or its
containing partitions should be serialized on this lock.

Architecture code may need to grab this lock if it is
using this data during, for example, context switches,
especially if they support SMP as locking interrupts
is not enough.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-10-20 09:37:49 -07:00
Andrew Boie
9f87deafd2 userspace: fix k_mem_domain_destroy()
This deprecated API won't be removed for one more release,
ensure it doesn't put the kernel into a bad state as it
currently sets all the member thread domain assignment to
NULL which is not what we want.

Have it reassign all member threads to the default domain.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-10-20 09:37:49 -07:00
Andrew Boie
b5a71f74a8 userspace: remove threads from domain on abort
When threads exited we were leaving dangling references to
them in the domain's mem_domain_q.

z_thread_single_abort() now calls into the memory domain
code via z_mem_domain_exit_thread() to take it off.

The thread setup code now invokes z_mem_domain_init_thread(),
avoiding extra checks in k_mem_domain_add_thread(), we know
the object isn't currently a member of a doamin.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-10-20 09:37:49 -07:00
David Komel
211e5b73c0 kernel: k_busy_wait() should return immediately on zero timeout
When CONFIG_ARCH_HAS_CUSTOM_BUSY_WAIT is not defined, cycles_to_wait
is calculated using a division operation. This calculation could take a
significant amount of time (a few microseconds on some architectures,
depending on the system clock).
In the special case of zero usec_to_wait, the function should return
immediately rather than spend time on calculations.
For example, in spi driver (spi_context.h, _spi_context_cs_control()),
k_busy_wait() can be called with zero delay. This can increase spi
transaction time significantly.
Another improvement, is moving the start_cycles initialization
before cycles_to_wait calculation, so the time it takes to calculate
cycles_to_wait will be taken into account.

Signed-off-by: David Komel <a8961713@gmail.com>
2020-10-14 14:52:06 -04:00
Peter Bigot
923caadf3b kernel: delayed_work: update k_delayed_work_cancel documentation
Although the documentation states that only work items that have not
completed the delay can be cancelled, in fact pending items can also
be removed from a work queue.  Document that behavior

Also document the specific return value that will be obtained based on
the state of the work item at the time cancellation was attempted
using the current implementation.

Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
2020-10-09 11:48:00 +02:00
Peter Bigot
f69f935865 kernel: delayed_work: fix race condition in k_delayed_work_cancel
The implementation checks the work queue pointer before taking a lock
and entering code that, depending on build options, may fault if that
pointer is null.  Move the check under the lock to ensure condition is
preserved.  Then the check in work_cancel is no longer necessary since
condition is verified under lock at all call sites.

Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
2020-10-09 11:48:00 +02:00
Jennifer Williams
4d33007486 kernel: stack: fix stack_push spinlock and return
The z_impl_k_stack_push() has a spinlock that is set after
stack member access, which could cause race conditions and
used multiple return statements. This commit
- moves the lock before the CHECKIF
- implements goto for flow of lock, reschedule, and unlock
- uses ret for single return at the end

Signed-off-by: Jennifer Williams <jennifer.m.williams@intel.com>
2020-10-07 17:10:36 -04:00
Peter A. Bigot
f1b86caff3 kernel: timer: update k_timer API for const correctness
API that takes k_timer structures but doesn't change data in them is
updated to const-qualify the underlying object, allowing information
to be retrieved from contexts where the containing object is
immutable.

Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2020-10-02 11:29:14 +02:00
Peter A. Bigot
16a4081520 kernel: timer: update _timeout API for const correctness
API that takes _timeout structures but doesn't change data in them is
updated to const-qualify the underlying object, allowing information
to be retrieved from contexts where the containing object is
immutable.

Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2020-10-02 11:29:14 +02:00
Aastha Grover
83b9f69755 code-guideline: Fixing code violation 10.4 Rule
Both operands of an operator in the arithmetic conversions
performed shall have the same essential type category.

Changes are related to converting the integer constants to the
unsigned integer constants

Signed-off-by: Aastha Grover <aastha.grover@intel.com>
2020-10-01 17:13:29 -04:00
Andrew Boie
e0ca403f4c kernel: add assert for mis-used k_thread_create()
k_thread_create() works as expected on both uninitialized memory,
or threads that have completely exited.

However, horrible and difficult to comprehend things can happen if a
thread object is already being used by the kernel and
k_thread_create() is called on it.

Historically this has been a problem with test cases trying to be
parsimonious with thread objects and not properly cleaning up
after themselves. Add an assertion for this which should catch
both the illegal creation of a thread already active, or threads
racing to create the same thread object.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-09-30 14:11:59 -04:00
Andrew Boie
f5a7e1a108 kernel: handle thread self-aborts on idle thread
Fixes races where threads on another CPU are joining the
exiting thread, since it could still be running when
the joiners wake up on a different CPU.

Fixes problems where the thread object is still being
used by the kernel when the fn_abort() function is called,
preventing the thread object from being recycled or
freed back to a slab pool.

Fixes a race where a thread is aborted from one CPU while
it self-aborts on another CPU, that was currently worked
around with a busy-wait.

Precedent for doing this comes from FreeRTOS, which also
performs final thread cleanup in the idle thread.

Some logic in z_thread_single_abort() rearranged such that
when we release sched_spinlock, the thread object pointer
is never dereferenced by the kernel again; join waiters
or fn_abort() logic may free it immediately.

An assertion added to z_thread_single_abort() to ensure
it never gets called with thread == _current outside of an ISR.

Some logic has been added to ensure z_thread_single_abort()
tasks don't run more than once.

Fixes: #26486
Related to: #23063 #23062

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-09-30 14:11:59 -04:00
Carles Cufi
28cb9dab64 kernel: Deprecate CONFIG_MULTITHREADING
Deprecate the Kconfig option for the time being. Unless a contributor
volunteers to take over the work to maintain the option, it will be
removed after 2 releases.

Relates to #27415.

Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
2020-09-23 15:50:32 -05:00
Lauren Murphy
f29a2d1ccc doc: Clarify semantics of k_msgq_put
Amend Doxygen documentation for k_msgq_put to note that the message
content will not be modified as a result of the function call and
constify data parameter in function prototype.

Fixes #22301

Signed-off-by: Lauren Murphy <lauren.murphy@intel.com>
2020-09-23 13:21:07 -05:00
Daniel Leung
b5c2ff9397 kernel: add kconfig CONFIG_KERNEL_MEM_POOL
Adds a kconfig CONFIG_KERNEL_MEM_POOL to decide whether
kernel memory pool related code is compiled. This option
can be disabled to shrink code size. If k_heap is not
being used at all, kheap.c will also not be compiled,
resulting in further smaller code size.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2020-09-19 05:49:13 -04:00
Peter Bigot
332b7df61b kernel: avoid implementation-defined behavior in timeout calculation
When to->dticks is an int64_t it may happen that the calculated
remaining time is a value that cannot be exactly represented in the
destination int32_t, producing an implementation-defined result which
can include a signal (interrupt).  Cap the maximum delay to the
largest value suported by the int32_t result.

Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
2020-09-17 22:19:35 -04:00
Watson Zeng
37f75d2d1f kernel: sched: bug fix for trace and monitor
sys_trace_thread_abort and z_thread_monitor_exit in
z_thread_single_abort also need to be protected by
sched_spinlock, otherwise when after the spinlock
release, if there is an pending interrupt, it will cause an
sched in the interrrupt exit, and those trace and monitor
function will never reach.

Signed-off-by: Watson Zeng <zhiwei@synopsys.com>
2020-09-17 09:30:22 +02:00
Andrew Boie
a8775ab8cb sched: don't use local lock in z_tick_sleep()
We're modifying thread_state. Use sched_spinlock instead.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-09-16 13:11:12 -05:00
Andrew Boie
8e0f6a5936 sched: hold spinlock in z_time_slice()
We are checking thread->base members like thread_state and prio
and making decisions based on it, hold the sched_spinlock to
avoid potential concurrency problems if these members are modified
on another CPU or nested interrupt.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-09-16 13:11:12 -05:00
Andrew Boie
83d7770de4 sched: check if runnable in sliceable()
We need to check if a thread is runnable at all before we
contemplate putting it on the end of the priority queue,
it might not be on the queue at all if it was suspended.

Replaces the less comprehensive check to see if the thread
was pending a timeout.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-09-16 13:11:12 -05:00
Andrew Boie
ffc5bdffbb sched: hold spinlock in z_thread_timeout()
We are checking and modifying members of thread->base
(in particular it's waitq and thread_state) which are
nominally protected by sched_spinlock. Hold it while
doing this to avoid concurrent changes on another CPU
or ISR preeemption.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-09-16 13:11:12 -05:00
Peter Bigot
1a7cd6ddbd kernel: device: invert sense of ready bit
The device status ready bit replaced the previous hack of clearing the
device API pointer when initialization of the device failed.  It
inadvertently changed the behavior of device_get_binding() when
invoked before the device was initialized: previously that succeeded
for uninitialized devices, after the change it failed.

Multiple driver initializations rely on being able to get a device
pointer for something they're going to depend on in their init
function, even if that device has not yet been initialized.  Although
this is wrong, and would cause faults if the device failed to
initialize before use, in practice it has been working.

It's not feasible to identify all the situations where this has
occurred, nor to add code to diagnose such cases without changing the
state associated with a device to distinguish initialized from
initialization success/failure.  Restore the previous behavior until a
more holistic solution is developed.

Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
2020-09-15 18:22:38 +02:00
Anas Nashif
6e27478c3d benchmarking: remove execution benchmarking code
This code had one purpose only, feed timing information into a test and
was not used by anything else. The custom trace points unfortunatly were
not accurate and this test was delivering informatin that conflicted
with other tests we have due to placement of such trace points in the
architecture and kernel code.

For such measurements we are planning to use the tracing functionality
in a special mode that would be used for metrics without polluting the
architecture and kernel code with additional tracing and timing code.

Furthermore, much of the assembly code used had issues.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2020-09-05 13:28:38 -05:00
Daniel Leung
0ffcfa9633 timing: introduce timing functions as a generic feature
Add timing functions and APIs.  This is now used with some of the tests
we have for performance and metrics and will be used whereever timing
informations are needed, for example for tracing, profiling and other
operations where timing info is critical.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-09-05 13:28:38 -05:00
Watson Zeng
1dddbecb35 tracing: swap: bug fix and enhancement for ARC
* Move switched_in into the arch context switch assembly code,
  which will correctly record the switched_in information.

* Add switched_in/switched_out for context switch in irq exit.

Signed-off-by: Watson Zeng <zhiwei@synopsys.com>
2020-09-03 21:54:15 +02:00