doc: genrest: Use a separate index page for all symbols

Add an option --separate-all-index that makes genrest.py generate a
separate index-all.rst index page that lists all symbols, instead of
listing all symbols in index.rst. index-all.rst is linked from
index.rst.

This was originally motivated by an external project where index.rst
becomes the top-level page, which runs into a Sphinx bottleneck with
sphinx_rtd_theme. See https://github.com/sphinx-doc/sphinx/issues/6909.

This turned out pretty nice after some feedback from Ruth Fuchss, so use
it for Zephyr too.

Also unclutter the generated documentation a bit by removing some
headings. This makes the navigation menu on the left nicer too.

Piggyback making genrest.py executable, which is handy when running it
manually.

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>

doc/CMakeLists.txt

@@ -206,6 +206,7 @@ add_custom_target(
   KCONFIG_TURBO_MODE=${KCONFIG_TURBO_MODE}
   KCONFIG_DOC_MODE=1
     ${PYTHON_EXECUTABLE} scripts/genrest.py ${RST_OUT}/doc/reference/kconfig/
+      --separate-all-index
       --keep-module-paths
       --modules Architecture,arch,${ZEPHYR_BASE}/arch
                 Driver,drivers,${ZEPHYR_BASE}/drivers

doc/scripts/genrest.py

@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+
 """
 Generates an alphabetical index of Kconfig symbols with links in index.rst, and
 a separate CONFIG_FOO.rst file for each Kconfig symbol.
@@ -88,6 +90,9 @@ def init():
     #   <path> is an absolute pathlib.Path instance, which is handy for robust
     #   path comparisons.
     #
+    # separate_all_index:
+    #   True if --separate-all-index was passed
+    #
     # strip_module_paths:
     #   True unless --keep-module-paths was passed
 
@@ -95,6 +100,7 @@ def init():
     global out_dir
     global index_desc
     global modules
+    global separate_all_index
     global strip_module_paths
 
     args = parse_args()
@@ -102,6 +108,7 @@ def init():
     kconf = kconfiglib.Kconfig(args.kconfig)
     out_dir = args.out_dir
     index_desc = args.index_desc
+    separate_all_index = args.separate_all_index
     strip_module_paths = args.strip_module_paths
 
     modules = []
@@ -180,16 +187,17 @@ deep). The title of the index is "<title> Configuration
 Options", and a 'configuration_options_<suffix>' RST link
 target is inserted at the top of the index page.
 
-If <index description path> is given, it should be the
+If <index description path> is given, it should be the path
-path to an RST file. The contents of this file will appear
+to an RST file. The contents of this file will appear under
-under the Overview heading on the symbol index page for the
+at the top of the symbol index page for the module,
-module. If no <index description path> is given, a generic
+underneath the heading. If no <index description path> is
-description is used instead.
+given, a generic description is used instead.
 
 The top-level index.rst index page contains a TOC tree that
 links to the index-*.rst pages for any modules. It also
 includes a list of all symbols, including symbols that do
-not appear in any module.
+not appear in any module. Pass --separate-all-index to use a
+separate index for the list of all symbols.
 
 If a symbol is defined in more than one module, it will be
 listed on several index pages.
@@ -200,6 +208,23 @@ symbol information pages, showing
 within modules. This behavior can be disabled by passing
 --keep-module-paths.""")
 
+    parser.add_argument(
+        "--separate-all-index",
+        action="store_true",
+        help="""\
+Instead of listing all symbols in index.rst, use a separate
+index-all.rst index page, which is linked from index.rst.
+Probably only useful in combination with --modules.
+
+index-all.rst has a 'configuration_options_all' RST link
+target.
+
+This option can make the documentation build orders of
+magnitude faster when the index.rst generated by this script
+is the top-level page, because Sphinx currently runs into a
+bottleneck with large top-level pages with some themes. See
+https://github.com/sphinx-doc/sphinx/issues/6909.""")
+
     parser.add_argument(
         "--keep-module-paths",
         dest="strip_module_paths",
@@ -227,26 +252,42 @@ def write_index_pages():
 
 def write_main_index_page():
     # Writes the main index page, which lists all symbols. In --modules mode,
-    # links to the module index pages are included.
+    # links to the module index pages are included. If --separate-all-index was
+    # passed, a separate index-all.rst index is generated as well.
 
     rst = index_header(title="Configuration Options",
                        link="configuration_options",
                        desc_path=index_desc)
 
+    if separate_all_index:
+        rst += """
+
+This index page lists all symbols, regardless of where they are defined:
+
+.. toctree::
+   :maxdepth: 1
+
+   index-all.rst\
+"""
+        write_if_updated("index-all.rst",
+                         index_header(title="All Configuration Options",
+                                      link="configuration_options_all",
+                                      desc_path=None) +
+                         sym_table_rst("Configuration Options",
+                                       kconf.unique_defined_syms))
+
     if modules:
         rst += """
 
-Subsystems
+These index pages only list symbols defined within a particular subsystem:
-**********
-
-These index pages list symbols defined within a particular subsystem, while the
-list below includes all configuration symbols.
 
 .. toctree::
    :maxdepth: 1
 
 """ + "\n".join("   index-" + suffix for _, suffix, _, _, in modules)
 
+    if not separate_all_index:
+        # Put index of all symbols in index.rst
         rst += sym_table_rst("All configuration options",
                              kconf.unique_defined_syms)
 
@@ -332,8 +373,8 @@ def index_header(title, link, desc_path):
     #   Link target string
     #
     # desc_path:
-    #   Path to file with RST file put into the Overview section at the
+    #   Path to file with RST to put at the of the page, underneath the
-    #   beginning of the page. If None, a generic description is used.
+    #   heading. If None, a generic description is used.
 
     if desc_path is None:
         desc = DEFAULT_INDEX_DESC
@@ -351,9 +392,6 @@ def index_header(title, link, desc_path):
 {}
 {}
 
-Overview
-********
-
 {}
 
 This documentation is generated automatically from the :file:`Kconfig` files by