Commit graph

5615 commits

Author SHA1 Message Date
Ulf Magnusson
cf89ba33ea global: Fix up leading/trailing blank lines in files
To make the updated test in
https://github.com/zephyrproject-rtos/ci-tools/pull/121 clean, though it
only checks modified files.

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2020-01-27 17:41:55 -06:00
Peter Bigot
ab7773dae8 sanitycheck: leave non-tty terminal unmodified on exit
The terminal configuration should not be reset on exit if
sanitycheck's output is being directed to something that mediates the
terminal control, such as a pager.  Only do the reset if stdout is
interactive.

Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
2020-01-26 17:54:41 +01:00
Ulf Magnusson
08484d0ac4 dts: gen_defines.py: Convert to use f-strings
Use f-strings instead of .format() to makes all the macro identifier and
multiline comment building easier to read.

f-strings were added in Python 3.6, which is required by Zephyr now.

Convert some

    ... + "_" + ...

string building to f-strings as well.

f-strings are a bit faster too (because they avoid format()/str()
lookups), though it's not likely to be noticeable here.

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2020-01-24 21:33:28 -05:00
Ulf Magnusson
a5f6233428 kconfig.py: Convert to use f-strings
Use f-strings instead of .format() to make things more readable. They
were added in Python 3.6, which Zephyr requires now.

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2020-01-24 15:57:23 +01:00
Ulf Magnusson
ba2f95ab39 kconfig: Show unsatisfied deps. when assignments don't take
Show which dependencies are unsatisfied when symbols don't get their
assigned value.

For example, assume that FOO below is assigned with CONFIG_FOO=y. Note
that BAR, BAZ, and STR = "hmm" are 'n'.

    config FOO
           bool "foo"
           depends on BAR && BAZ && QAZ && STR = "hmm"

    config BAR
           def_bool n

    config BAZ
           def_bool n

    config QAZ
           def_bool y

    config STR
           def_string "zmh"

This now prints this warning:

    warning: FOO (defined at /home/ulf/z/z/Kconfig:10) was assigned the
    value 'y' but got the value 'n'. Check these unsatisfied
    dependencies: BAR (=n), BAZ (=n), STR = "hmm" (=n). See ...

Fixes: #21888

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2020-01-23 23:04:09 +01:00
Anas Nashif
dd65f7c38a sanitycheck: change distribution of tests per set
Sort by testcase name and path and not by platforms. This way we do not
have the same platform executing on CI system causing failures to
overload.

Right now we have almost all qemu targets executing on 1 or 2 nodes,
causing those systems to be overloaded. Instead of taking the default
sorting by platforms, we now sort by testcases, so the distribution is
more "random", but still reproducible.

Remove the ordering function that would put native_posix in the first
set, it is not being used anymore.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-01-23 14:02:59 -05:00
Rihards Skuja
f59b92b3ee runners: openocd: don't throw when no config is provided
In some cases the config is already provided for us e.g. when setting
OPENOCD_NRF5_SUBFAMILY. Also it looks like config was always supposed to
be optional (there are checks for it on do_run()).

Signed-off-by: Rihards Skuja <rihardssk@mikrotik.com>
2020-01-23 10:38:57 -05:00
Flavio Ceolin
661035a6b8 west: runners: stm32: Remove unecessary assignment
msg_text is unconditionally assigned later and this inital value is
never used.

Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
2020-01-22 17:37:41 -05:00
Flavio Ceolin
9dbb50bac8 scripts: sanitycheck: Simplify if statement
Outer if is checking if the variable is in a list with two options
which means that we can just do an if / else and remove an assignment.

Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
2020-01-22 17:37:41 -05:00
Ulf Magnusson
45050dda48 kconfig/cmake: Improve reconfiguration behavior
There are some issues with the behavior when rerunning CMake in an
already initialized build directory:

 1. The check for assignments to promptless symbols in configuration
    fragments isn't run when reconfiguring, because it only runs if
    zephyr/.config doesn't exist

 2. As outlined in
    https://github.com/zephyrproject-rtos/zephyr/issues/9573, you can
    get into situations where zephyr/.config is invalid (e.g. due to
    being outdated), but menuconfig/guiconfig can't be run to fix it

 3. If kconfig.py fails while merging fragments during reconfiguration,
    it will ignore the fragments during the next reconfiguration and use
    the existing zephyr/.config instead, because the fragment checksum
    is calculated and saved before running kconfig.py

(Footnote: The input configuration file(s) to kconfig.py can be either a
list of configuration fragments, when merging fragments, or just
zephyr/.config, if the merged configuration is up-to-date. The output
configuration file is always zephyr/.config.)

To fix the first two issues, explicitly tell kconfig.py when it's
dealing with handwritten configuration input (fragments), via a new
--handwritten-input-configs flag. This is more robust than checking
whether zephyr/.config exists, which was the old logic.

When dealing with handwritten input, there should be no assignments to
promptless symbols. Assignments to promptless symbols is expected in
zephyr/.config however, because it doubles as configuration output.

When running menuconfig/guiconfig, the input configuration is
zephyr/.config rather than configuration fragments, so this change also
makes sure that menuconfig can always be run as long as zephyr/.config
exists and is up-to-date.

To fix the last issue, only write the checksum for the configuration
fragments if kconfig.py succeeds (which means it wrote a
zephyr/.config).

Also improve naming a bit, add help texts for the command-line
parameters to kconfig.py, and simplify write_kconfig_filenames() by
moving logic into it.

Partial fix for
https://github.com/zephyrproject-rtos/zephyr/issues/9573, without the
part in #issuecomment-469701831. Can still run into issues when e.g.
when CMake files can't make sense of settings.

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2020-01-22 18:28:07 +01:00
Anas Nashif
e337882189 sanitycheck: fix reporting of timeouts
Timeouts were not reported correctly in the XML file as failures,
causing some confusion in the shippable results.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-01-22 01:39:28 -05:00
Anas Nashif
9fe1e2a663 sanitycheck: support excluding platforms
It is often needed to run sanitycheck and exclude a specific platform or
multiple platforms. Add an option to provide this using
--exclude-platform. The option can be used multiple times to exclude
multiple platforms at the same time.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-01-21 21:10:49 -05:00
Ulf Magnusson
7419f4f46f scripts: kconfig.py: Simplify write_kconfig_filenames()
The assert has never hit and probably won't be useful. Shorten the code
a bit.

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2020-01-17 19:39:14 +01:00
Ulf Magnusson
7a531ea22b scripts: kconfig.py: Use err() helper for warnings-turned-errors
De-spams the code a bit.

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2020-01-17 19:39:14 +01:00
Ulf Magnusson
7a148ef9da scripts: kconfig.py: Rename verify_*() functions to check_*()
Bit shorter, matches the devicetree code.

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2020-01-17 19:39:14 +01:00
Ulf Magnusson
bb4a4e79dc scripts: kconfig.py: Detect all assignments to promptless symbols
Assigning to promptless symbols has no effect.

Previously, the only check was for whether the value assigned to a
symbol matched its final value. This misses cases where a promptless
symbol is assigned to and just happens to get the assigned-to value as
its final value.

Instead, detect whether configuration files are being merged (by
checking if zephyr/.config already exists), and explicitly check for
assignments to promptless symbols in that case. We can't do it when
zephyr/.config already exists (and is being loaded), because it includes
values for promptless symbols as well.

With the no-prompt check moved out, also use a more specific message for
it, and remove stuff related to prompts elsewhere. Shorten messages a
bit at the same time, and add two warn() and err() helpers.

Fixes: #20697

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2020-01-17 19:39:14 +01:00
Ulf Magnusson
4e85006ba4 dts: Rename generated_dts_board*.{h,conf} to devicetree*.{h,conf}
generated_dts_board.h is pretty redundant and confusing as a name. Call
it devicetree.h instead.

dts.h would be another option, but DTS stands for "devicetree source"
and is the source code format, so it's a bit confusing too.

The replacement was done by grepping for 'generated_dts_board' and
'GENERATED_DTS_BOARD'.

Two build diagram and input-output SVG files were updated as well, along
with misc. documentation.

hal_ti, mcuboot, and ci-tools updates are included too, in the west.yml
update.

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2020-01-17 17:57:59 +01:00
Anas Nashif
d6d0b6454b ci: do not display timestamps
With timestamps enabled, the log in shippable is very difficult to read,
so remove those and enable them when needed for debugging only.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-01-17 08:30:23 -06:00
Anas Nashif
1342d21fc0 sanitycheck: support runtime artifact cleanup
Using --runtime-artifact-cleanup, the script will now remove all
artifacts of passing tests. This is useful for running sanitycheck on
systems with little disk space.

Fixes #21922

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-01-16 19:19:07 -05:00
Ulf Magnusson
d0f7652858 dts: gen_defines.py: Rename *_dev() to *_node()
out_dev(), etc., as a name is a holdover from before commit 73ac1466fb
("scripts: edtlib: Call nodes "nodes" instead of "devices""). It's
confusing for the same reasons as mentioned in that commit.

Renamings:

 - dev_ident()            -> node_ident()
 - out_dev()              -> out_node()
 - out_dev_s()            -> out_node_s()
 - dev_aliases()          -> node_aliases()
 - dev_path_aliases()     -> node_path_aliases()
 - dev_instance_aliases() -> node_instance_aliases()

Also clean up comments to talk about nodes instead of devices where it
makes sense.

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2020-01-16 18:32:58 +01:00
Martí Bolívar
60db6a6e2f scripts: runners: nrfjprog: sort SNR list before prompting
This guarantees muscle memory will work over multiple runs when the
same boards are connected. (The "nrfjprog --help" output for --ids
does not guarantee an order.)

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-01-16 10:20:39 -05:00
Martí Bolívar
e4768ad7e7 scripts: runners: nrfjprog: restrict when we guess snr
Right now, the nrfjprog runner will prompt the user for which board to
use if there are multiple possibilities and the --snr command line was
not given to specify one ahead of time.

Tweak this so it only happens if standard input is connected to a
terminal. This should avoid stalling the process on board farms when
this runner is used in automation.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-01-16 10:20:39 -05:00
Martí Bolívar
ae2f27cf16 scripts: runners: defer ensure_snr() to run in nrfjprog
The create() classmethod should not be doing any I/O -- its only job
is to create the ZephyrBinaryRunner instance. It's currently trying to
figure out what the serial number of the board is, though. Let's defer
that work to do_run(), so it gets handled by run_common's exception
handler.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-01-16 10:20:39 -05:00
Martí Bolívar
8837469492 scripts: runners: improve error handling CLI
Catch RuntimeError when calling runner.run() and print a message
instead of dumping stack unless in verbose mode.

This improves the command line interface when runners raise exceptions.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-01-16 10:20:39 -05:00
Andrew Boie
75bd9b6ce0 sanitycheck: improve summary output
Show how many tests passed out of those that were actually
performed, not including skipped tests (which we also report
a count of)

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-14 23:26:06 -05:00
Anas Nashif
cd1f017468 sanitycheck: set extra_args from command line
Missed setting extra_args coming from command line in the testsuite.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-01-14 13:39:46 -05:00
Ulf Magnusson
75621c3b45 dts: gen_defines.py: Add should_write() helper for properties
should_write() returns False for properties that are skipped in
write_properties(). Shortens write_properties() a bit.

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2020-01-14 17:31:59 +01:00
Ulf Magnusson
776de86dc7 menuconfig: Update to work around resizing crash on macOS
Update menuconfig (and kconfiglib and guiconfig, just to sync) to
upstream revision 424d0d38e7, to get this commit in, which works around
a crash on some macOS Python installations.

    menuconfig: Work around crash on resize on some macOS systems

    get_wch() has started raising curses.error when resizing the
    terminal on some macOS Python installations. Work around for now by
    falling back on getch(), which still works.

    See https://github.com/ulfalizer/Kconfiglib/issues/84. Needs more
    investigation, but I don't have a Mac handy.

    Based on https://github.com/ulfalizer/Kconfiglib/pull/85, but with
    some more comments.

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2020-01-13 14:03:12 +01:00
Ulf Magnusson
02dab5a5a8 dts: gen_defines.py: Add helper for outputting {} initializers
The

    "{" + ", ".join(...) + "}"

pattern is in a bunch of places now. Add an out_dev_init() helper for
it, with the same parameters as out_dev().

Also makes "{" vs. "{ " consistent, picking the first one.

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2020-01-13 13:50:15 +01:00
Ulf Magnusson
405daa0081 dts: edtlib: Add helper for testing error messages and test _slice()
Add a helper function verify_error() to testedtlib.py that takes a DTS
source and verifies that it generates a particular EDTError.

Use it to test the hints that the previous commit added to errors
generated by _slice().

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2020-01-13 13:49:33 +01:00
Ulf Magnusson
57b2d2749c dts: edtlib: Improve error for wrong reg/ranges/interrupts size
Show how the element size was calculated in the error message when a
'reg', 'ranges', or 'interrupts' property has the wrong size. This
should help with debugging. Also mention that #*-cells properties come
from the parent node, which can be subtle.

Came up in https://github.com/zephyrproject-rtos/zephyr/issues/21607
(though it seems the comment disappeared).

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2020-01-13 13:49:33 +01:00
Ulf Magnusson
b4e18072a8 kconfig: Fix wrong max argument count for some preprocessor functions
dt_chosen_reg() and dt_node_reg() take between 1 and 3 arguments, not
between 1 and 4 arguments. The implicit name argument isn't included in
the count.

These functions implement $(dt_chosen_reg_*) and $(dt_node_reg_*).

Giving the right max argument count makes Kconfiglib generate error
messages that give the location of the call, instead of getting a
cryptic generic Python error.

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2020-01-13 13:48:17 +01:00
Erwan Gouriou
a11405df52 scripts: requirements: Request version 0.24.0 for pyocd
pyocd 0.24.0 provides support for more user options.
This enables flashing of additional boards using pyocd.

Signed-off-by: Erwan Gouriou <erwan.gouriou@linaro.org>
2020-01-13 11:49:57 +01:00
Anas Nashif
e7656ea831 sanitycheck: do not log stderr from BinaryHandler
stderr from the binary handler (native_posix for example) was redirected
to the logger as errors, this is confusing the console and users, so
remove this.

Fixes #21784

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-01-09 14:28:32 -05:00
Andrew Boie
9b34ecd0c8 userspace: fix incorrect linker routing
The _thread_idx_map bitfield which has '1' set for free
thread indexes really needs to live in the
data section reserved for kernel object metadata, as this
is a part of memory that is allowed to shift addresses
between zephyr_prebuilt.elf and zephyr.elf.

However, if an application defines enough static threads
that there are no free indexes, the entire bitfield will
be zeroed and the bitfield will end up in the main BSS
section.

Force this data to always be in the .kobject_data.data
section regardless of its contents.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-09 09:13:23 -08:00
Ulf Magnusson
5372dedf2f scripts: kconfig: lint.py: Improve error reporting
If the working directory for a command was missing (usually due to
forgetting to run 'west update'), you'd get a FileNotFoundError
exception along with a cryptic error like

    'git' not found

Only catch OSError instead (which is a base class of FileNotFoundError),
and always show the exception message. It makes it clear that it's the
working directory that's missing.

Add some other misc. improvements too:

 - Turn stderr output from external commands into a warning instead of
   an error

 - Add err() and warn() helpers

 - Include the command name in messages

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2020-01-09 11:19:57 -05:00
Ulf Magnusson
23a5b4963b dts: edtlib: Add 'type: path' for path references
Add binding support for a 'path' property type, for properties that are
assigned node paths. Usually, paths are assigned with path references
like 'foo = &label' (common in /chosen), but plain strings are accepted
as well as long as they're valid paths.

The 'path' type is mostly for completeness at this point, but might be
useful for https://github.com/zephyrproject-rtos/zephyr/issues/21623.
The support is there already in dtlib.

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2020-01-08 08:02:00 -06:00
Olof Johansson
8067aca7e9 gen_defines: close files before exit
In some cases, we've seen the output files be truncated when the
python script has been rebuilt into a .pex before running it --
likely due to buffering.

Closing files explicitly is the right thing to do anyway, so let's
do it.

Signed-off-by: Olof Johansson <olof@lixom.net>
2020-01-07 21:06:58 +01:00
Ulf Magnusson
def1f0e2d5 devicetree: Remove DT_SRAM_{BASE_ADDRESS,SIZE}, use CONFIG_* versions
The SRAM address and size are currently available as both
DT_SRAM_{BASE_ADDRESS,SIZE} and as CONFIG_SRAM_{BASE_ADDRESS,SIZE} (via
the Kconfig preprocessor).

Use the CONFIG_SRAM_* versions everywhere, and remove generation of the
DT_SRAM_* versions from gen_defines.py.

The Kconfig symbols currently depend on 'ARC || ARM || NIOS2 || X86'.
Not sure why, so I removed it.

It looks like no configuration files set CONFIG_SRAM_* at the moment, so
another option might be to use the DT_* symbols everywhere instead. Some
Kconfig.defconfig.series files add defaults to them though.

Also improve the help texts for CONFIG_SRAM_* to say that they normally
come from devicetree rather than configuration files.

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2020-01-07 17:19:36 +01:00
Ulf Magnusson
95c18f1e6b dts: gen_defines.py: Clean up 'reg' code
Reorganize write_regs() to reuse more code for *_BASE_ADDRESS and
*_SIZE, and get rid of reg_addr_ident() and reg_size_ident().

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2020-01-07 16:42:33 +01:00
Anas Nashif
e9450065de sanitycheck: additional scripting hook after flashing
Support custom hooks after flashing is completed.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-01-06 13:58:10 -05:00
Anas Nashif
4522e10d65 sanitycheck: report overflow on SRAM overflow
Instead of failing, report overflow when the SRAM overflows.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-01-06 13:58:10 -05:00
Anas Nashif
bdffb88950 sanitycheck: close pipe file handles
Make sure we close the pipe file handles after we are done, otherwise we
will end up with too many open file descriptors and crash...

Fixes #20974
Fixes #21637

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-01-06 13:58:10 -05:00
Anas Nashif
64c84b15ce sanitycheck: more debug messages
More debug messages to help locate issues and debug bugs.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-01-06 13:58:10 -05:00
Anas Nashif
1b1a7e242b sanitycheck: run pre/post script using dedicated function
Reduce duplicated code by introducing a dedicated function for running
custom scripts.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-01-06 13:58:10 -05:00
Anas Nashif
32e924459d sanitycheck: log errors when executing binaries
Add some verbosity into the log file to help with debugging.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-01-06 13:58:10 -05:00
Anas Nashif
52b66c6ad1 sanitycheck: when timing out in handler, show correct log file
Fix issue where we timeout in handler and show the device.log file
instead.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-01-06 13:58:10 -05:00
Anas Nashif
813deb7967 sanitycheck: use Unknown instead of N/A for fail reason
Instead of N/A, use Unknown which is more appropriate, N/A does not
really apply here.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-01-06 13:58:10 -05:00
Anas Nashif
5c331f95ad sanitycheck: add Digilent to device discovery
Some platforms use this USB device for serial, detect it when generating
hardware map.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-01-06 13:58:10 -05:00
Andrew Boie
bd13710524 sanitycheck: use abs path for logs
Makes it much easier to inspect them in a different terminal
from where the test was run. These paths tend to be long
anyway, even if relative.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-03 08:40:34 -05:00