More editing and reorganization of extension docs.

docs/source/extension/extension_dev.rst

@@ -9,16 +9,15 @@ The JupyterLab application is comprised of a core application object and a set o
 Getting Started
 ---------------
 
-The documentation in this section covers the basic and advanced concepts for writing extensions. If you would rather get hands-on practice or more in-depth reference documentation, we have a number of tutorials, examples, cookiecutters, and generated reference documentation.
+The documentation in this section covers the basic and advanced concepts for writing extensions. To get hands-on practice or more in-depth reference documentation, we have tutorials, examples, cookiecutters, and generated reference documentation.
 
 Tutorials
 ^^^^^^^^^
 
 We provide a set of guides to get started writing extensions for JupyterLab:
 
-- :ref:`extension_tutorial`: An in-depth tutorial to learn how to make a simple JupyterLab extension.
-- The `JupyterLab Extension Examples Repository <https://github.com/jupyterlab/extension-examples>`_: A short tutorial series
-  to learn how to develop extensions for JupyterLab, by example.
+- :ref:`extension_tutorial`: A tutorial to learn how to make a simple JupyterLab extension.
+- The `JupyterLab Extension Examples Repository <https://github.com/jupyterlab/extension-examples>`_: A short tutorial series to learn how to develop extensions for JupyterLab by example.
 - :ref:`developer-extension-points`: A list of the most common JupyterLab extension points.
 - Another common pattern for extending JupyterLab document widgets with application plugins is covered in :ref:`documents`.
 
@@ -30,49 +29,41 @@ We provide several cookiecutters to create JupyterLab extensions:
 - `extension-cookiecutter-ts <https://github.com/jupyterlab/extension-cookiecutter-ts>`_: Create a JupyterLab extension in TypeScript
 - `extension-cookiecutter-js <https://github.com/jupyterlab/extension-cookiecutter-js>`_: Create a JupyterLab extension in JavaScript
 - `mimerender-cookiecutter-ts <https://github.com/jupyterlab/mimerender-cookiecutter-ts>`_: Create a MIME Renderer JupyterLab extension in TypeScript
-- `theme-cookiecutter <https://github.com/jupyterlab/theme-cookiecutter>`_: Create a new theme for JupyterLab
+- `theme-cookiecutter <https://github.com/jupyterlab/theme-cookiecutter>`_: Create a theme extension for JupyterLab
 
 API Reference Documentation
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-If you are looking for generated reference API documentation on the JupyterLab and Lumino API:
+Here is some autogenerated API documentation for JupyterLab and Lumino packages:
 
 - `JupyterLab API Documentation <https://jupyterlab.github.io/jupyterlab/>`_
 - `Lumino API Documentation <https://jupyterlab.github.io/lumino/>`_
 
 
-JupyterLab Extensions
----------------------
+Overview of Extensions
+----------------------
 
-A plugin is the basic unit of extensibility in JupyterLab. JupyterLab supports several types of plugins:
+A JupyterLab extension is a package that contains JupyterLab plugins. A plugin is the basic unit of extensibility in JupyterLab. JupyterLab supports several types of plugins:
 
 -  **application plugins:** Application plugins are the fundamental building block of JupyterLab functionality. Application plugins interact with JupyterLab and other plugins by requiring services provided by other plugins, and optionally providing their own service to the system.
 -  **mime renderer plugins:** Mime renderer plugins are simplified, restricted ways to extend JupyterLab to render custom mime data in notebooks and files. These plugins are automatically converted to equivalent application plugins by JupyterLab when they are loaded.
 -  **theme plugins:** Theme plugins provide a way to customize the appearance of JupyterLab by changing themeable values (i.e., CSS variable values) and providing additional fonts and graphics to JupyterLab.
 
-Plugins are distributed in JupyterLab extensions—one extension can contain multiple plugins. An extension can be distributed in several ways:
+An extension is a package that contains one or more JupyterLab plugins. There are several types of extensions:
 
-- A "source" extension is a JavaScript (npm) package that exports one or more plugins. Installing a source extension requires a user to rebuild JupyterLab. This rebuilding step requires Node.js and may take a lot of time and memory, so some users may not be able to install the extension.
-- A "prebuilt" extension (new in JupyterLab 3.0) is a bundle of JavaScript code that can be loaded into JupyterLab without rebuilding JupyterLab. In this case, the extension developer uses tools provided by JupyterLab to compile a source extension into a JavaScript bundle that includes the non-JupyterLab JavaScript dependencies, then distributes the resulting bundle in, for example, a Python pip or conda package. Users installing prebuilt extensions do not have to have Node.js installed and can immediately use the extension without a JupyterLab rebuild.
+- A *source extension* is a JavaScript (npm) package that exports one or more plugins. Installing a source extension requires a user to rebuild JupyterLab. This rebuilding step requires Node.js and may take a lot of time and memory, so some users may not be able to install a source extension. See :ref:`deduplication` for the technical reasons for rebuilding JupyterLab when a source extension is installed.
+- A *prebuilt extension* (new in JupyterLab 3.0) distributes a bundle of JavaScript code prebuilt from a source extension that can be loaded into JupyterLab without rebuilding JupyterLab. In this case, the extension developer uses tools provided by JupyterLab to compile a source extension into a JavaScript bundle that includes the non-JupyterLab JavaScript dependencies, then distributes the resulting bundle in, for example, a Python pip or conda package. Installing a prebuilt extensions does not require Node.js.
 
-An extension may be distributed as both a source JavaScript package published on NPM and as a prebuilt extension bundle published in a Python package, giving users the choice of how to install it.
+An extension can be published both as a source extension on NPM and as a prebuilt extension (e.g., published as a Python package). In some cases, system administrators may even choose to install a prebuilt extension by directly copying the prebuilt bundle to an appropriate directory, circumventing the need to create a Python package. If a source extension and a prebuilt extension with the same name are installed in JupyterLab, the prebuilt extension takes precedence.
 
 Because prebuilt extensions do not require a JupyterLab rebuild, they have a distinct advantage in multiuser systems where JuptyerLab is installed at the system level. On such systems, only the system administrator has permissions to rebuild JupyterLab and install source extensions. Since prebuilt extensions can be installed at the per-user level, the per-environment level, or the system level, each user can have their own separate set of prebuilt extensions that are loaded dynamically in their browser on top of the system-wide JupyterLab.
 
 .. tip::
-   We recommend developing prebuilt extensions in Python packages for user convenience.
-
-
-Writing an Extension
---------------------
-
-We encourage extension authors to add the `jupyterlab-extension GitHub topic
-<https://github.com/search?utf8=%E2%9C%93&q=topic%3Ajupyterlab-extension&type=Repositories>`__
-to any GitHub extension repository.
+   We recommend publishing prebuilt extensions in Python packages for user convenience.
 
 
-Plugin metadata
-^^^^^^^^^^^^^^^
+Application Plugins
+-------------------
 
 A typical plugin is specified by the following metadata. The ``id`` and ``activate`` fields are required and the other fields may be omitted. For more information about the ``requires``, ``optional``, or ``provides`` fields, see :ref:`services`.
 
@@ -94,7 +85,7 @@ A typical plugin is specified by the following metadata. The ``id`` and ``activa
 - ``activate`` is the function called when your plugin is activated. The arguments are, in order, the Application object, the services corresponding to the ``requires`` tokens, then the services corresponding to the ``optional`` tokens (or ``null`` if that particular ``optional`` token is not registered in the system). The return value of the ``activate`` function (or resolved return value if a promise is returned) will be stored in the system as the service associated with the ``provides`` token.
 
 Application Object
-""""""""""""""""""
+^^^^^^^^^^^^^^^^^^
 
 A Jupyter front-end application object is given to each plugin in its
 ``activate()`` function. The application object has:
@@ -107,10 +98,152 @@ A Jupyter front-end application object is given to each plugin in its
 -  ``shell`` - a generic Jupyter front-end shell instance, which holds the user interface for the application.
 
 
+.. _services:
+
+Plugins Interacting with Each Other
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+One of the foundational features of the JupyterLab plugin system is that plugins can interact with other plugins by providing a service to the system and requiring services provided by other plugins. A service can be any JavaScript value, and typically is a JavaScript object with methods and data attributes. For example, the plugin that supplies the JupyterLab main menu provides a service object to the system with methods and attributes other plugins can use to interact with the main menu.
+
+In the following discussion, the plugin that is providing a service to the system is the *provider* plugin, and the plugin that is requiring and using the service is the *consumer* plugin.
+
+A service provided by a plugin is identified by a *token*, i.e., a concrete instance of the Lumino Token class. The provider plugin lists the token in its plugin metadata ``provides`` field, and returns the associated service from its ``activate`` function. Consumer plugins import the token (for example, from the provider plugin's extension JavaScript package) and list the token in their plugin metadata ``requires`` or ``optional`` fields. When JupyterLab instantiates the consumer plugin, it will pass in the service associated with the token. JupyterLab orders plugin activation to ensure that a provider of a service is activated before its consumers.
+
+A token defined in TypeScript can also define a TypeScript interface for the service associated with the token. If the provider or consumer uses TypeScript, the service will be type-checked against this interface.
+
+.. note::
+   JupyterLab uses tokens to identify services (instead of strings, for example) to prevent conflicts between identifiers and to enable type checking when using TypeScript.
+
+Publishing Tokens
+"""""""""""""""""
+
+Since consumers will need to import a token used by a provider, the token should be exported in a published JavaScript package. A pattern in core JupyterLab is to create and export tokens from a self-contained ``tokens`` JavaScript module in a package. This enables consumers to import a token directly from the package's ``tokens`` module (e.g., ``import { MyToken } from 'provider/tokens';``), thus enabling a tree-shaking bundling optimization to bundle only the tokens and not other code from the package.
+
+Another pattern in core JupyterLab is to create and export a token from a third package that both the provider and consumer extensions import, rather than defining the token in the provider's package. This enables a user to swap out the provider extension for a different extension that provides the same token with an alternative service implementation. For example, the core JupyterLab ``filebrowser`` package exports a token representing the file browser service (enabling interactions with the file browser). The ``filebrowser-extension`` package contains a plugin that implements the file browser in JupyterLab and provides the file browser service to JupyterLab (identified with the token imported from the ``filebrowser`` package). Extensions in JupyterLab that want to interact with the filebrowser thus do not need to have a JavaScript dependency on the ``filebrowser-extension`` package, but only need to import the token from the ``filebrowser`` package. This pattern enables users to seamlessly change the file browser in JupyterLab by writing their own extension that imports the same token from the ``filebrowser`` package and provides it to the system with their own alternative file browser service.
+
+.. _deduplication:
+
+Deduplication of Dependencies
+"""""""""""""""""""""""""""""
+
+..
+   TODO: Maybe put this part in the place where we talk about the sharedPackages metadata? It's an important implementation detail in JupyterLab that has consequences for extension metadata.
+
+One important concern and challenge in the JupyterLab extension system is deduplicating dependencies of extensions instead of having extensions use their own bundled copies of dependencies. For example, the Lumino widgets system on which JupyterLab relies for communication across the application requires all packages use the same copy of the ``@lumino/widgets`` package. Tokens identifying plugin services also need to be shared across the providers and consumers of the services, so dependencies that export tokens need to be deduplicated.
+
+Deduplication in JupyterLab happens in two ways. For source extensions, JupyterLab deduplicates dependencies when rebuilds itself to include the extension during the extension installation process. Deduplication is one of the main reasons JupyterLab needs to be rebuilt when installing source extensions. For prebuilt extensions, JupyterLab relies on the new Webpack module federation system to share dependencies across different bundles (including the core JupyterLab application bundle).
+
+To ensure that a consumer gets the same token instance as the provider provided to the sytem, any required tokens that are imported by a consumer extension should list the exporting extension as a singleton package in their ``jupyterlab.sharedPackages`` config. Required token packages should be listed as ``bundled: false`` - this will generate a JavaScript error if the package (and thus the token) is not present in the system at runtime. Optional token packages should be listed as singletons that are bundled (otherwise, if they are not present in the system, it will cause a js error when you try to import them).
+
+
+Plugin Settings
+^^^^^^^^^^^^^^^
+
+JupyterLab exposes a plugin settings system that can be used to provide
+default setting values and user overrides.
+
+An extension can specify user settings using a JSON Schema. The schema
+definition should be in a file that resides in the ``schemaDir``
+directory that is specified in the ``package.json`` file of the
+extension. The actual file name should use is the part that follows the
+package name of extension. So for example, the JupyterLab
+``apputils-extension`` package hosts several plugins:
+
+-  ``'@jupyterlab/apputils-extension:menu'``
+-  ``'@jupyterlab/apputils-extension:palette'``
+-  ``'@jupyterlab/apputils-extension:settings'``
+-  ``'@jupyterlab/apputils-extension:themes'``
+
+And in the ``package.json`` for ``@jupyterlab/apputils-extension``, the
+``schemaDir`` field is a directory called ``schema``. Since the
+``themes`` plugin requires a JSON schema, its schema file location is:
+``schema/themes.json``. The plugin's name is used to automatically
+associate it with its settings file, so this naming convention is
+important. Ensure that the schema files are included in the ``"files"``
+metadata in ``package.json``.
+
+See the
+`fileeditor-extension <https://github.com/jupyterlab/jupyterlab/tree/master/packages/fileeditor-extension>`__
+for another example of an extension that uses settings.
+
+A system administrator or user can override default values of extension settings with the :ref:`overrides.json <overridesjson>` file.
+
+.. _rendermime:
+
+Mime Renderer Plugins
+----------------------
+
+Mime Renderer plugins are a convenience for creating an plugin
+that can render mime data and potentially render files of a given type.
+We provide an extension cookiecutter for mime renderer plugins in TypeScript
+`here <https://github.com/jupyterlab/mimerender-cookiecutter-ts>`__.
+
+Mime renderer plugins are more declarative than standard plugins.
+The extension is treated the same from the command line perspective
+(``jupyter labextension install`` ), but it does not directly create
+JupyterLab plugins. Instead it exports an interface given in the
+`rendermime-interfaces <https://jupyterlab.github.io/jupyterlab/interfaces/_rendermime_interfaces_src_index_.irendermime.iextension.html>`__
+package.
+
+The JupyterLab repo has an example mime renderer extension for
+`pdf <https://github.com/jupyterlab/jupyterlab/tree/master/packages/pdf-extension>`__
+files. It provides a mime renderer for pdf data and registers itself as
+a document renderer for pdf file types.
+
+The JupyterLab organization also has a mime renderer extension tutorial
+which adds mp4 video rendering to the application
+`here <https://github.com/jupyterlab/jupyterlab-mp4>`__.
+
+The ``rendermime-interfaces`` package is intended to be the only
+JupyterLab package needed to create a mime renderer extension (using the
+interfaces in TypeScript or as a form of documentation if using plain
+JavaScript).
+
+The only other difference from a standard extension is that has a
+``jupyterlab`` key in its ``package.json`` with ``"mimeExtension"``
+metadata. The value can be ``true`` to use the main module of the
+package, or a string path to a specific module (e.g. ``"lib/foo"``).
+
+The mime renderer can update its data by calling ``.setData()`` on the
+model it is given to render. This can be used for example to add a
+``png`` representation of a dynamic figure, which will be picked up by a
+notebook model and added to the notebook document. When using
+``IDocumentWidgetFactoryOptions``, you can update the document model by
+calling ``.setData()`` with updated data for the rendered MIME type. The
+document can then be saved by the user in the usual manner.
+
+
+Theme plugins
+-------------
+
+A theme is a JupyterLab plugin that uses a ``ThemeManager`` and can
+be loaded and unloaded dynamically. The package must include all static
+assets that are referenced by ``url()`` in its CSS files. Local URLs can
+be used to reference files relative to the location of the referring sibling CSS files. For example ``url('images/foo.png')`` or
+``url('../foo/bar.css')``\ can be used to refer local files in the
+theme. Absolute URLs (starting with a ``/``) or external URLs (e.g.
+``https:``) can be used to refer to external assets. The path to the
+theme asset entry point is specified ``package.json`` under the ``"jupyterlab"``
+key as ``"themePath"``. See the `JupyterLab Light
+Theme <https://github.com/jupyterlab/jupyterlab/tree/master/packages/theme-light-extension>`__
+for an example. Ensure that the theme files are included in the
+``"files"`` metadata in ``package.json``.  Note that if you want to use SCSS, SASS, or LESS files,
+you must compile them to CSS and point JupyterLab to the CSS files.
+
+The theme extension is installed in the same way as a regular extension (see
+`extension authoring <#extension-authoring>`__).
+
+It is also possible to create a new theme using the
+`TypeScript theme cookiecutter <https://github.com/jupyterlab/theme-cookiecutter>`__.
+
+
+Writing a Source Extension
+--------------------------
 
 package.json metadata
 ^^^^^^^^^^^^^^^^^^^^^
 
+We provide a schema for the valid ``jupyterlab`` metadata for an extension's ``package.json`` describing the available options
 
 Custom webpack config
 """""""""""""""""""""
@@ -156,8 +289,6 @@ when building the federated extension with ``jlpm run build``.
 Disabling other extensions
 """"""""""""""""""""""""""
 
-Prebuilt data
-"""""""""""""""
 
 Sharing configuration
 """""""""""""""""""""
@@ -248,200 +379,83 @@ Currently supported package managers are ``pip`` and ``conda``.
 Packaging extensions
 ^^^^^^^^^^^^^^^^^^^^
 
-Prebuilt Extensions
-^^^^^^^^^^^^^^^^^^^
-
-``install.json``
+Most extensions are single JavaScript packages, and can be shipped on npmjs.org.
+This makes them discoverable by the JupyterLab extension manager, provided they
+have the ``jupyterlab-extension`` keyword  in their ``package.json``.  If the package also
+contains a server extension (Python package), the author has two options.
+The server extension and the JupyterLab extension can be shipped in a single package,
+or they can be shipped separately.
 
-How prebuilt extensions work
-"""""""""""""""""""""""""""""
+The JupyterLab extension can be bundled in a package on PyPI and conda-forge so
+that it ends up in the user's application directory.  Note that the user will still have to run ``jupyter lab build``
+(or build when prompted in the UI) in order to use the extension.
+The general idea is to pack the Jupyterlab extension using ``npm pack``, and then
+use the ``data_files`` logic in ``setup.py`` to ensure the file ends up in the
+``<jupyterlab_application>/share/jupyter/lab/extensions``
+directory.
 
-Steps for building
-""""""""""""""""""
+Note that even if the JupyterLab extension is unusable without the
+server extension, as long as you use the companion package metadata it is still
+useful to publish it to npmjs.org so it is discoverable by the JupyterLab extension manager.
 
-Directory walkthrough
-"""""""""""""""""""""
+The server extension can be enabled on install by using ``data_files``.
+an example of this approach is `jupyterlab-matplotlib <https://github.com/matplotlib/jupyter-matplotlib/tree/ce9cc91e52065d33e57c3265282640f2aa44e08f>`__.  The file used to enable the server extension is `here <https://github.com/matplotlib/jupyter-matplotlib/blob/ce9cc91e52065d33e57c3265282640f2aa44e08f/jupyter-matplotlib.json>`__.   The logic to ship the JS tarball and server extension
+enabler is in `setup.py <https://github.com/matplotlib/jupyter-matplotlib/blob/ce9cc91e52065d33e57c3265282640f2aa44e08f/setup.py>`__.  Note that the ``setup.py``
+file has additional logic to automatically create the JS tarball as part of the
+release process, but this could also be done manually.
 
-Plugins
--------
 
-.. _rendermime:
+Prebuilt Extensions
+-------------------
 
-Mime Renderer Plugins
+package.json metadata
 ^^^^^^^^^^^^^^^^^^^^^
 
-Mime Renderer plugins are a convenience for creating an plugin
-that can render mime data and potentially render files of a given type.
-We provide an extension cookiecutter for mime renderer plugins in TypeScript
-`here <https://github.com/jupyterlab/mimerender-cookiecutter-ts>`__.
+In addition to the package metadata given above, prebuilt extensions have extra metadata for where the prebuilt assets should go.
 
-Mime renderer plugins are more declarative than standard plugins.
-The extension is treated the same from the command line perspective
-(``jupyter labextension install`` ), but it does not directly create
-JupyterLab plugins. Instead it exports an interface given in the
-`rendermime-interfaces <https://jupyterlab.github.io/jupyterlab/interfaces/_rendermime_interfaces_src_index_.irendermime.iextension.html>`__
-package.
 
-The JupyterLab repo has an example mime renderer extension for
-`pdf <https://github.com/jupyterlab/jupyterlab/tree/master/packages/pdf-extension>`__
-files. It provides a mime renderer for pdf data and registers itself as
-a document renderer for pdf file types.
-
-The JupyterLab organization also has a mime renderer extension tutorial
-which adds mp4 video rendering to the application
-`here <https://github.com/jupyterlab/jupyterlab-mp4>`__.
-
-The ``rendermime-interfaces`` package is intended to be the only
-JupyterLab package needed to create a mime renderer extension (using the
-interfaces in TypeScript or as a form of documentation if using plain
-JavaScript).
-
-The only other difference from a standard extension is that has a
-``jupyterlab`` key in its ``package.json`` with ``"mimeExtension"``
-metadata. The value can be ``true`` to use the main module of the
-package, or a string path to a specific module (e.g. ``"lib/foo"``).
-
-The mime renderer can update its data by calling ``.setData()`` on the
-model it is given to render. This can be used for example to add a
-``png`` representation of a dynamic figure, which will be picked up by a
-notebook model and added to the notebook document. When using
-``IDocumentWidgetFactoryOptions``, you can update the document model by
-calling ``.setData()`` with updated data for the rendered MIME type. The
-document can then be saved by the user in the usual manner.
-
-
-Theme plugins
-^^^^^^^^^^^^^
-
-A theme is a JupyterLab plugin that uses a ``ThemeManager`` and can
-be loaded and unloaded dynamically. The package must include all static
-assets that are referenced by ``url()`` in its CSS files. Local URLs can
-be used to reference files relative to the location of the referring sibling CSS files. For example ``url('images/foo.png')`` or
-``url('../foo/bar.css')``\ can be used to refer local files in the
-theme. Absolute URLs (starting with a ``/``) or external URLs (e.g.
-``https:``) can be used to refer to external assets. The path to the
-theme asset entry point is specified ``package.json`` under the ``"jupyterlab"``
-key as ``"themePath"``. See the `JupyterLab Light
-Theme <https://github.com/jupyterlab/jupyterlab/tree/master/packages/theme-light-extension>`__
-for an example. Ensure that the theme files are included in the
-``"files"`` metadata in ``package.json``.  Note that if you want to use SCSS, SASS, or LESS files,
-you must compile them to CSS and point JupyterLab to the CSS files.
-
-The theme extension is installed in the same way as a regular extension (see
-`extension authoring <#extension-authoring>`__).
-
-It is also possible to create a new theme using the
-`TypeScript theme cookiecutter <https://github.com/jupyterlab/theme-cookiecutter>`__.
-
-
-.. _services:
-
-Plugins Interacting with Each Other
------------------------------------
-
-One of the foundational features of the JupyterLab plugin system is that plugins can interact with other plugins by providing a service to the system and requiring services provided by other plugins. A service can be any JavaScript value, and typically is a JavaScript object with methods and data attributes. For example, the plugin that supplies the JupyterLab main menu provides a service object to the system with methods and attributes other plugins can use to interact with the main menu.
-
-In the following discussion, the plugin that is providing a service to the system is the *provider* plugin, and the plugin that is requiring and using the service is the *consumer* plugin.
-
-A service provided by a plugin is identified by a *token*, i.e., a concrete instance of the Lumino Token class. The provider plugin lists the token in its plugin metadata ``provides`` field, and returns the associated service from its ``activate`` function. Consumer plugins import the token (for example, from the provider plugin's extension JavaScript package) and list the token in their plugin metadata ``requires`` or ``optional`` fields. When JupyterLab instantiates the consumer plugin, it will pass in the service associated with the token. JupyterLab orders plugin activation to ensure that a provider of a service is activated before its consumers.
-
-A token defined in TypeScript can also define a TypeScript interface for the service associated with the token. If the provider or consumer uses TypeScript, the service will be type-checked against this interface.
-
-.. note::
-   JupyterLab uses tokens to identify services (instead of strings, for example) to prevent conflicts between identifiers and to enable type checking when using TypeScript.
-
-Publishing Tokens
-^^^^^^^^^^^^^^^^^
-Since consumers will need to import a token used by a provider, the token should be exported in a published JavaScript package. A pattern in core JupyterLab is to create and export tokens from a self-contained ``tokens`` JavaScript module in a package. This enables consumers to import a token directly from the package's ``tokens`` module (e.g., ``import { MyToken } from 'provider/tokens';``), thus enabling a tree-shaking bundling optimization to bundle only the tokens and not other code from the package.
-
-Another pattern in core JupyterLab is to create and export a token from a third package that both the provider and consumer extensions import, rather than defining the token in the provider's package. This enables a user to swap out the provider extension for a different extension that provides the same token with an alternative service implementation. For example, the core JupyterLab ``filebrowser`` package exports a token representing the file browser service (enabling interactions with the file browser). The ``filebrowser-extension`` package contains a plugin that implements the file browser in JupyterLab and provides the file browser service to JupyterLab (identified with the token imported from the ``filebrowser`` package). Extensions in JupyterLab that want to interact with the filebrowser thus do not need to have a JavaScript dependency on the ``filebrowser-extension`` package, but only need to import the token from the ``filebrowser`` package. This pattern enables users to seamlessly change the file browser in JupyterLab by writing their own extension that imports the same token from the ``filebrowser`` package and provides it to the system with their own alternative file browser service.
-
-
-Deduplication of Dependencies
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-..
-   TODO: Maybe put this part in the place where we talk about the sharedPackages metadata? It's an important implementation detail in JupyterLab that has consequences for extension metadata.
-
-One important concern and challenge in the JupyterLab extension system is deduplicating dependencies of extensions instead of having extensions use their own bundled copies of dependencies. For example, the Lumino widgets system on which JupyterLab relies for communication across the application requires all packages use the same copy of the ``@lumino/widgets`` package. Tokens identifying plugin services also need to be shared across the providers and consumers of the services, so dependencies that export tokens need to be deduplicated.
-
-Deduplication in JupyterLab happens in two ways. For source extensions, JupyterLab deduplicates dependencies when rebuilds itself to include the extension during the extension installation process. Deduplication is one of the main reasons JupyterLab needs to be rebuilt when installing source extensions. For prebuilt extensions, JupyterLab relies on the new Webpack module federation system to share dependencies across different bundles (including the core JupyterLab application bundle).
-
-To ensure that a consumer gets the same token instance as the provider provided to the sytem, any required tokens that are imported by a consumer extension should list the exporting extension as a singleton package in their ``jupyterlab.sharedPackages`` config. Required token packages should be listed as ``bundled: false`` - this will generate a JavaScript error if the package (and thus the token) is not present in the system at runtime. Optional token packages should be listed as singletons that are bundled (otherwise, if they are not present in the system, it will cause a js error when you try to import them).
-
-
-Advanced Plugins
-----------------
-
-Plugin Settings
-^^^^^^^^^^^^^^^
-
-In addition to the file system that is accessed by using the
-``@jupyterlab/services`` package, JupyterLab exposes a plugin settings
-system that can be used to provide default setting values and user overrides.
-
-An extension can specify user settings using a JSON Schema. The schema
-definition should be in a file that resides in the ``schemaDir``
-directory that is specified in the ``package.json`` file of the
-extension. The actual file name should use is the part that follows the
-package name of extension. So for example, the JupyterLab
-``apputils-extension`` package hosts several plugins:
-
--  ``'@jupyterlab/apputils-extension:menu'``
--  ``'@jupyterlab/apputils-extension:palette'``
--  ``'@jupyterlab/apputils-extension:settings'``
--  ``'@jupyterlab/apputils-extension:themes'``
-
-And in the ``package.json`` for ``@jupyterlab/apputils-extension``, the
-``schemaDir`` field is a directory called ``schema``. Since the
-``themes`` plugin requires a JSON schema, its schema file location is:
-``schema/themes.json``. The plugin's name is used to automatically
-associate it with its settings file, so this naming convention is
-important. Ensure that the schema files are included in the ``"files"``
-metadata in ``package.json``.
-
-See the
-`fileeditor-extension <https://github.com/jupyterlab/jupyterlab/tree/master/packages/fileeditor-extension>`__
-for another example of an extension that uses settings.
-
-A system administrator or user can override default values of extension settings with the :ref:`overrides.json <overridesjson>` file.
-
-
-
-Development workflow
---------------------
+Packaging Information
+^^^^^^^^^^^^^^^^^^^^^
 
-We encourage extension authors to add the `jupyterlab-extension GitHub topic
-<https://github.com/search?utf8=%E2%9C%93&q=topic%3Ajupyterlab-extension&type=Repositories>`__
-to any GitHub extension repository.
+Since prebuilt extensions are distributed in many ways, there is an extra file, ``install.json`` that helps the user know how a prebuilt extension was installed. This file is put there by the packaging system distributing the prebuilt extension.
 
 
-Older Docs
-==========
+How prebuilt extensions work
+"""""""""""""""""""""""""""""
 
-Implementation
---------------
-- We provide a ``jupyter labextension build`` script that is used to build federated bundles
+Steps for building
+""""""""""""""""""
+- We provide a ``jupyter labextension build`` script that is used to build prebuilt bundles
    - The command produces a set of static assets that are shipped along with a package (notionally on ``pip``/``conda``)
    - It is a Python cli so that it can use the dependency metadata from the active JupyterLab
    - The assets include a module federation ``remoteEntry.*.js``, generated bundles, and some other files that we use
    - ``package.json`` is the original ``package.json`` file that we use to gather metadata about the package, with some included build metadata
    - we use the previously existing ``@jupyterlab/builder -> build`` to generate the ``imports.css``, ``schemas`` and ``themes`` file structure
-- We provide a schema for the valid ``jupyterlab`` metadata for an extension's ``package.json`` describing the available options
 - We provide a ``labextensions`` handler in ``jupyterlab_server`` that loads static assets from ``labextensions`` paths, following a similar logic to how ``nbextensions`` are discovered and loaded from disk
-- The ``settings`` and ``themes`` handlers in ``jupyterlab_server`` has been updated to load from the new ``labextensions`` locations, favoring the federated extension locations over the bundled ones
+- The ``settings`` and ``themes`` handlers in ``jupyterlab_server`` has been updated to load from the new ``labextensions`` locations, favoring the prebuilt extension locations over the bundled ones
 - A ``labextension develop`` command has been added to install an in-development extension into JupyterLab.  The default behavior is to create a symlink in the ``sys-prefix/share/jupyter/labextensions/package-name`` to the static directory of the extension
 - We provide a ``cookiecutter`` that handles all of the scaffolding for an extension author, including the shipping of ``data_files`` so that when the user installs the package, the static assets end up in ``share/jupyter/labextensions``
+
+Directory walkthrough
+"""""""""""""""""""""
+
+Runtime configuration
+---------------------
 - We handle disabling of lab extensions using a trait on the ``LabApp`` class, so it can be set by admins and overridden by users.  Extensions are automatically enabled when installed, and must be explicitly disabled.  The disabled config can consist of a package name or a plugin regex pattern
-- Extensions can provide ``disabled`` metadata that can be used to replace an entire extension or individual plugins
 - ``page_config`` and ``overrides`` are also handled with traits so that admins can provide defaults and users can provide overrides
-- We provide a script to update extensions: ``python -m jupyterlab.upgrade_extension``
-- We update the ``extension-manager`` to target metadata on ``pypi``/``conda`` and consume those packages.
+
+
+
+
+Development workflow
+--------------------
+
 
 
 Extension Authoring
 -------------------
 
+
 An Extension is a valid `npm
 package <https://docs.npmjs.com/getting-started/what-is-npm>`__ that
 meets the following criteria:
@@ -466,6 +480,8 @@ meets the following criteria:
          "jupyterlab",
          "jupyterlab-extension"
        ],
+- We encourage extension authors to add the `jupyterlab-extension GitHub topic
+<https://github.com/search?utf8=%E2%9C%93&q=topic%3Ajupyterlab-extension&type=Repositories>`__ to any GitHub extension repository.
 
 While authoring the extension, you can use the command:
 
@@ -629,38 +645,3 @@ Finally, you will need to configure babel with a ``babel.config.js`` file contai
        ]
      ]
    };
-
-
-
-
-
-
-
-Shipping Packages
------------------
-
-Most extensions are single JavaScript packages, and can be shipped on npmjs.org.
-This makes them discoverable by the JupyterLab extension manager, provided they
-have the ``jupyterlab-extension`` keyword  in their ``package.json``.  If the package also
-contains a server extension (Python package), the author has two options.
-The server extension and the JupyterLab extension can be shipped in a single package,
-or they can be shipped separately.
-
-The JupyterLab extension can be bundled in a package on PyPI and conda-forge so
-that it ends up in the user's application directory.  Note that the user will still have to run ``jupyter lab build``
-(or build when prompted in the UI) in order to use the extension.
-The general idea is to pack the Jupyterlab extension using ``npm pack``, and then
-use the ``data_files`` logic in ``setup.py`` to ensure the file ends up in the
-``<jupyterlab_application>/share/jupyter/lab/extensions``
-directory.
-
-Note that even if the JupyterLab extension is unusable without the
-server extension, as long as you use the companion package metadata it is still
-useful to publish it to npmjs.org so it is discoverable by the JupyterLab extension manager.
-
-The server extension can be enabled on install by using ``data_files``.
-an example of this approach is `jupyterlab-matplotlib <https://github.com/matplotlib/jupyter-matplotlib/tree/ce9cc91e52065d33e57c3265282640f2aa44e08f>`__.  The file used to enable the server extension is `here <https://github.com/matplotlib/jupyter-matplotlib/blob/ce9cc91e52065d33e57c3265282640f2aa44e08f/jupyter-matplotlib.json>`__.   The logic to ship the JS tarball and server extension
-enabler is in `setup.py <https://github.com/matplotlib/jupyter-matplotlib/blob/ce9cc91e52065d33e57c3265282640f2aa44e08f/setup.py>`__.  Note that the ``setup.py``
-file has additional logic to automatically create the JS tarball as part of the
-release process, but this could also be done manually.
-