wip update packaging

Finish packaging updates

wip refactor packaging, core, and dev modes

wip packaging updates

reorg

wip packaging setup

wip packaging

cleanup

Move mock packages out of jupyterlab dir

Always use yarn for developers

wip update dev and core mode

clean up commands and extension

cleanup

update docs

Update .yarnrc

update ci scripts

Update ensure repo

fix integrity check

Update gitignore

Switch back to jlpm

wip

Update scripts

cleanup

cleanup

cleanup

cleanup

no need for egg-info hack anymore

clean up handling of mock packages

update selenium check

Refactor travis

Cleanup

reorder matrix

Fix handling of link and unlinking extensions

Fix handling of postcss

Cache the miniconda dir

fix travis yml syntax

fix travis yml syntax

Update group names

fix miniconda handling

Update docs

Fix handling of miniconda path

reorder path

tweak karma settings

cleanup

Remove manifest in favor of package_data

fix path addition

more setup cleanup

Add staging data to package data

Make sure our data_files work

add comment

use glob2 to get proper files

defer using glob2

cleanup

Inline handling of ** globs

Add a node_modules blocker

use fresh environments on every build

fix removal of conda envs

use env create for requirements file

ensure selenium is installed

Fix #3231

front load the longer job

Fix #3227

Uniform handling of error messages for extension commands

Let process errors propagate

Fix ##3264

Fix validation and add tests

Fix property name

fix handling of app_dir

Remove pdb from py.test so builds don't time out

Use npm to run scripts in the re-usable build utils

Clean up build:update

Fix glob handling

Use declarative specs for data files and package files

Add detection of extension module shadowing

Fix package_data_spec

Add workaround for git clean behavior on windows

remove debug statement

Clean up setupbase

Address comments

Remove outdated file check

glob handling cleanup

Incorporate jupyter-packaging #20

clean up setup.py

Make sure build:src works

Reinstate build:src

Fix spelling

wip add test manager

fix pip install .

Clean up selenium check implementation and temp usage

Invert requirements

clean up handling of launcher versions in script

fix version spec

.gitignore

@@ -2,10 +2,12 @@ MANIFEST
 build
 dist
 lib
-jupyterlab/build
+jupyterlab/static
 jupyterlab/schemas
 jupyterlab/themes
 jupyterlab/geckodriver
+dev_mode/schemas
+dev_mode/themes
 
 node_modules
 .cache

.travis.yml

@@ -7,12 +7,20 @@ addons:
   firefox: latest
 cache:
   yarn: true
+  directories:
+    - $HOME/miniconda
+    - $HOME/.cache/pip
 env:
   matrix:
-    - GROUP=tests
-    - GROUP=coverage
-    - GROUP=other
+    - GROUP=py3
+    - GROUP=py2
+    - GROUP=js
+    - GROUP=integrity
+    - GROUP=cli
+    - GROUP=docs
+    - GROUP=js_cov
   global:
+  - MINICONDA_DIR="$HOME/miniconda"
   - GH_REF: github.com/jupyterlab/jupyterlab.git
   - secure: MWpTI6cj3/Bnmtrr0Oqlp2JeWqDneB9aEjlQDaRxLOkqVbxhqDcYW9qAgZZP+sq29vT5oVMWzyCirteKxJfG2vy3HQE1XNLhz82Sf/7sE6DQ51gohl0CcOeA/uA8hCXEw97hneFWsZgHKqSoch7nVDsE3qfYgO+930jHlnxYApJGP9hZFv2Q2NVa6+99kipEYS4BY/yBDYKy6/t4kXcnBrUlNaPtdjnXcrY9esLZ7EQtkaG5VqcQVIBaLJKGF5Q7Aufj5nCFaZ6hZDF1Bi/AbmIbVWFyiT+22i8DZK6YwenECckyzoWkl+bEhYepWsgBKh/BDgPBAmPWKHgU5V4apDaGqZBhF7FP6H02AdZYYuCwl47jyakqvWLZW7oDmorL+HsWG5HQ3m0tMT2ywdbwNOiD39tiPPXjsvROh5ys9vL6NzQvxILCeEOnzcZrFuxi2LGEZfnlqRIjkh1llUAvNc3mOycRLWDOwVQa2+U59qDRXCSY2RD+MOfcdFUGengVujTMaAPMBUa3E33/ZIOOKJtR5TIajYZvd9B2uDlz02QfvTK+hrTaNYJjRZ8WCaeSM/CIKdoLw+29MNO6eqtchw0/vNvM8c9EkhrhMQKcY04OecVhmZkemFhd4SD5l92VX3z3xSxLkmazfNkj3CigWDXNxfDd2ORoGjA46Pga8RM=
 install:

.yarnrc

@@ -1,3 +1,3 @@
-yarn-path "./jupyterlab/yarn.js"
+yarn-path "./jupyterlab/staging/yarn.js"
 workspaces-experimental true
 registry "https://registry.npmjs.org"

CONTRIBUTING.md

@@ -23,7 +23,7 @@ All source code is written in [TypeScript](http://www.typescriptlang.org/Handboo
 
 ## Setting Up a Development Environment
 
-### Installing Node.js and npm
+### Installing Node.js and jlpm
 
 Building the JupyterLab from its GitHub source code requires Node.js version
 4+.
@@ -82,14 +82,13 @@ jupyter serverextension enable --py jupyterlab
 
 Notes:
 
-* The `jlpm` command is a JupyterLab-provided, locked version of the [yarn](https://yarnpkg.com/en/).  If you have `yarn` installed already, you can use
-the `yarn` command when developing, and it will use the local version of `yarn`
- in `jupyterlab/yarn.js` when run in the repository or a built application
- directory.
+* The `jlpm` command is a JupyterLab-provided, locked version of the [yarn](https://yarnpkg.com/en/) package manager.  If you have `yarn` installed 
+already, you can use the `yarn` command when developing, and it will use the 
+local version of `yarn` in `jupyterlab/yarn.js` when run in the repository or 
+a built application directory.
 
-* At times, it may be necessary to clean your local repo with the command `npm run clean:slate`.  This will clean the repository, and re-install and
-rebuild.  Note that we use `npm` in this one instance because the `jlpm`
-command may not be available at the time.
+* At times, it may be necessary to clean your local repo with the command `jlpm run clean:slate`.  This will clean the repository, and re-install and
+rebuild.
 
 * If `pip` gives a `VersionConflict` error, it usually means that the installed
 version of `jupyterlab_launcher` is out of date. Run `pip install --upgrade
@@ -156,22 +155,13 @@ provides additional architecture information.
 The repository consists of many npm packages that are managed using the lerna
 build tool.  The npm package source files are in the `packages/` subdirectory.
 
-**Prerequisites**
-
-- [node](http://nodejs.org/) (preferably version 5 or later)
-- Jupyter notebook server version 4.3 or later (to run examples)
-
-```bash
-npm install --save jupyterlab
-```
-
 ### Build the NPM Packages from Source
 
 ```bash
 git clone https://github.com/jupyterlab/jupyterlab.git
 cd jupyterlab
 pip install -e .
-jlpm install
+jlpm
 jlpm run build:packages
 ```
 
@@ -211,11 +201,11 @@ the initial build if the assets are already built.
 ## Build Utilities
 
 There are a series of build utilities for maintaining the repository.
-To get a suggested version for a library use `npm run get:dependency foo`.
-To update the version of a library across the repo use `npm run update:dependency foo@^x.x`.
-To remove an unwanted dependency use `npm run remove:dependency foo`.
+To get a suggested version for a library use `jlpm run get:dependency foo`.
+To update the version of a library across the repo use `jlpm run update:dependency foo@^x.x`.
+To remove an unwanted dependency use `jlpm run remove:dependency foo`.
 
-The key utility is `npm run integrity`, which ensures the integrity of
+The key utility is `jlpm run integrity`, which ensures the integrity of
 the packages in the repo. It will:
 
 - Ensure the core package version dependencies match everywhere.
@@ -228,10 +218,10 @@ in the repository at once, instead of 50+ individual builds.
 
 The integrity script also allows you to automatically add a dependency for
 a package by importing from it in the TypeScript file, and then running:
-`npm run integrity && npm install` from the repo root.
+`jlpm run integrity` from the repo root.
 
 We also have scripts for creating and removing packages in `packages/`,
-`npm run create:package` and `npm run remove:package`.
+`jlpm run create:package` and `jlpm run remove:package`.
 
 
 ## Notes

MANIFEST.in

@@ -1,18 +0,0 @@
-recursive-include jupyterlab/build *
-recursive-include jupyterlab/schemas *
-recursive-include jupyterlab/themes *
-
-include package.json
-include LICENSE
-include CONTRIBUTING.md
-include README.md
-include package.json
-include setupbase.py
-
-include jupyterlab/package.app.json
-include jupyterlab/*.js
-include jupyterlab/.yarnrc
-include jupyterlab/yarn.app.lock
-include jupyterlab/yarn.lock
-
-prune jupyterlab/tests

README.md

@@ -56,15 +56,10 @@ Note: If installing using `pip install --user`, you must add the user-level
  `bin` directory to your `PATH` environment variable in order to launch
  `jupyter lab`.
 
-Note: JupyterLab can be installed from a git checkout using `pip`.  Note
-that you will have to build the application after installing to get the static 
-assets.  `pip` will warn you that the static assets are not included when 
-installing, and it will fail to build a cached wheel if it decides to try 
-and make one.  Example:
+JupyterLab can be installed from a git checkout using `pip`.  Example:
 
 ```bash
 pip install git+git://github.com/jupyterlab/jupyterlab.git
-jupyter lab build   # Requires nodejs
 jupyter serverextension enable --py jupyterlab --sys-prefix
 ```
 

appveyor.yml

@@ -12,7 +12,7 @@ environment:
 
 # build cache to preserve files/folders between builds
 cache:
-  - '%LOCALAPPDATA%\\Yarn' # yarn cache
+  - '%LOCALAPPDATA%\\Yarn' # jlpm cache
 
 # scripts that run after cloning repository
 install:
@@ -37,8 +37,8 @@ install:
   - 'jlpm versions'
   - 'jlpm config current'
   - 'jlpm cache list'
-  - 'jlpm install'
-  - 'jlpm run build:main'
+  - 'jlpm'
+  - 'jlpm run build'
   - 'jupyter lab build'
 
 build: off
@@ -47,9 +47,13 @@ build: off
 test_script:
   # Run integrity first we we see the message.
   - 'jlpm run integrity'
+  - 'python -m jupyterlab.selenium_check'
+  - 'pip install jupyterlab_launcher==0.6'
   - 'py.test'
   - 'git status'
   - 'jlpm run build'
   - 'jlpm run build:test'
   - 'jlpm test || jlpm test || jlpm test'
-  - 'python -m jupyterlab.selenium_check'
+  - 'python -m jupyterlab.selenium_check --dev-mode'
+
+

buildutils/src/create-package.ts

@@ -37,5 +37,6 @@ inquirer.prompt(questions).then(answers => {
   data.name = name;
   data.description = description;
   utils.writePackageData(jsonPath, data);
-  utils.run('jlpm run integrity');
+  // Use npm here so this file can be used outside of JupyterLab.
+  utils.run('npm run integrity');
 });

buildutils/src/ensure-repo.ts

@@ -113,7 +113,7 @@ function ensureJupyterlab(): string[] {
   let version = String(childProcess.execSync(cmd)).trim();
 
   let basePath = path.resolve('.');
-  let corePath = path.join(basePath, 'jupyterlab', 'package.json');
+  let corePath = path.join(basePath, 'dev_mode', 'package.json');
   let corePackage = utils.readJSONFile(corePath);
 
   corePackage.jupyterlab.extensions = {};
@@ -183,7 +183,14 @@ function ensureIntegrity(): boolean {
   let messages: { [key: string]: string[] } = {};
 
   // Pick up all the package versions.
-  utils.getLernaPaths().forEach(pkgPath => {
+  let paths = utils.getLernaPaths();
+
+  // These two are not part of the workspaces but should be kept
+  // in sync.
+  paths.push('./jupyterlab/tests/mock_packages/extension');
+  paths.push('./jupyterlab/tests/mock_packages/mimeextension');
+
+  paths.forEach(pkgPath => {
     // Read in the package.json.
     let data: any;
     try {

buildutils/src/make-release.ts

@@ -1,44 +0,0 @@
-/*-----------------------------------------------------------------------------
-| Copyright (c) Jupyter Development Team.
-| Distributed under the terms of the Modified BSD License.
-|----------------------------------------------------------------------------*/
-
-import * as fs from 'fs-extra';
-import * as path from 'path';
-import * as utils from './utils';
-import { ensureIntegrity } from './ensure-repo';
-
-// Start from a clean slate.
-utils.run('npm run clean:slate');
-
-// Ensure integrity.
-if (!ensureIntegrity()) {
-    process.exit(1);
-}
-
-// Build the packages.
-utils.run('npm run build:packages');
-
-// Change to the jupyterlab dir.
-process.chdir(path.join('.', 'jupyterlab'));
-
-// Run a production build with Typescript source maps.
-utils.run('jlpm run build:prod');
-
-// Update the package.app.json file.
-let data = utils.readJSONFile('./package.json');
-data['scripts']['build'] = 'webpack';
-data['scripts']['watch'] = 'webpack --watch';
-data['scripts']['build:prod'] = "webpack --define process.env.NODE_ENV=\"'production'\"";
-data['jupyterlab']['outputDir'] = '..';
-data['jupyterlab']['staticDir'] = '../static';
-data['jupyterlab']['linkedPackages'] = {};
-utils.writePackageData('./package.app.json', data);
-
-// Update our app index file.
-fs.copySync('./index.js', './index.app.js');
-
-// Add  the release metadata.
-let releaseData: any = { version: data.jupyterlab.version };
-let text = JSON.stringify(releaseData, null, 2) + '\n';
-fs.writeFileSync('./build/release_data.json', text);

buildutils/src/patch-release.ts

@@ -27,16 +27,20 @@ if (!fs.existsSync(packagePath)) {
 
 // Perform the patch operations.
 console.log('Patching', target, '...');
-utils.run('jlpm run build:packages');
-utils.run('jlpm version patch', { cwd: packagePath });
-utils.run('jlpm publish', { cwd: packagePath});
+// Use npm here so this file can be used outside of JupyterLab.
+utils.run('npm run build:packages');
+utils.run('npm version patch', { cwd: packagePath });
+utils.run('npm publish', { cwd: packagePath});
+
+// Update the static folder.
+utils.run('npm run build:update');
 
 // Extract the new package info.
 let data = utils.readJSONFile(path.join(packagePath, 'package.json'));
 let name = data.name;
 let version = data.version;
 
-utils.run('jlpm run integrity');
+utils.run('npm run integrity');
 utils.run('git commit -a -m "Release ' + name + '@' + version + '"');
 utils.run('git tag ' + name + '@' + version);
 

buildutils/src/remove-package.ts

@@ -45,7 +45,7 @@ fs.removeSync(path.dirname(packagePath));
 
 // Update the core jupyterlab build dependencies.
 try {
-  utils.run('jlpm run integrity');
+  utils.run('npm run integrity');
 } catch (e) {
   if (!process.env.TRAVIS_BRANCH) {
     console.error(e);

buildutils/src/update-core-mode.ts

@@ -0,0 +1,35 @@
+/*-----------------------------------------------------------------------------
+| Copyright (c) Jupyter Development Team.
+| Distributed under the terms of the Modified BSD License.
+|----------------------------------------------------------------------------*/
+
+import * as fs from 'fs-extra';
+import * as path from 'path';
+import * as utils from './utils';
+
+// Get the dev mode package.json file.
+let data = utils.readJSONFile('./dev_mode/package.json');
+
+// Update the values that need to change and write to staging.
+data['scripts']['build'] = 'webpack';
+data['scripts']['watch'] = 'webpack --watch';
+data['scripts']['build:prod'] = "webpack --define process.env.NODE_ENV=\"'production'\"";
+data['jupyterlab']['outputDir'] = '..';
+data['jupyterlab']['staticDir'] = '../static';
+data['jupyterlab']['linkedPackages'] = {};
+
+let staging = './jupyterlab/staging';
+utils.writePackageData(path.join(staging, 'package.json'), data);
+
+// Update our index file and webpack file.
+fs.copySync('./dev_mode/index.js', './jupyterlab/staging/index.js');
+fs.copySync('./dev_mode/webpack.config.js',
+            './jupyterlab/staging/webpack.config.js');
+
+
+// Update the jlpm.lock file.
+utils.run('jlpm', { cwd: staging });
+
+
+// Build the core assets.
+utils.run('jlpm run build:prod', { cwd: staging });

clean.py

@@ -0,0 +1,16 @@
+import os
+import subprocess
+
+here = os.path.abspath(os.path.dirname(__file__))
+
+
+# Workaround for https://github.com/git-for-windows/git/issues/607
+if os.name == 'nt':
+    for (root, dnames, files) in os.walk(here):
+        if 'node_modules' in dnames:
+            subprocess.check_call(['rmdir', '/s', '/q', 'node_modules'],
+                cwd=root, shell=True)
+            dnames.remove('node_modules')
+
+
+subprocess.check_call(['git', 'clean', '-dfx'], cwd=here)

docs/extensions_dev.md

@@ -93,29 +93,19 @@ jupyter lab --watch
 ```
 
 This will cause the application to incrementally rebuild when one of the
-linked packages changes.  You can open another terminal and run
-
-```bash
-npm run watch
-```
-
-to automatically recompile the TypeScript files while editing.  Note that
-only the compiled JavaScript files (and the CSS files) are watched by the 
-WebPack process.  
-
-The is also a `jupyter lab --fast-watch` that allows you to skip the
-initial build if the app is already built.  
+linked packages changes.  Note that only compiled JavaScript files (and the 
+CSS files) are watched by the WebPack process.  
 
 Note that the application is built against **released** versions of the
 core JupyterLab extensions.  If your extension depends on JupyterLab
 packages, it should be compatible with the dependencies in the
-`jupyterlab/package.app.json` file.  If you must
+`jupyterlab/static/package.json` file.  If you must
 install a extension into a development branch of JupyterLab, you
 have to graft it into the source tree of JupyterLab itself.
 This may be done using the command
 
 ```
-jlpm run add:sibling <path-or-url> && npm install
+jlpm run add:sibling <path-or-url>
 ```
 
 in the JupyterLab root directory, where `<path-or-url>` refers either to an

docs/extensions_user.md

@@ -4,9 +4,9 @@ JupyterLab extensions add functionality to the JupyterLab application.
 They can provide new file viewer types, launcher activities, and new Notebook
 output renderers for example.
 
-### Installing Node.js and npm
+### Installing Node.js
 
-Installing JupyterLab extensions requires Node.js version 6+ and Node's package manager, ``npm``.
+Installing JupyterLab extensions requires Node.js version 4+.
 
 If you use ``conda``, you can get them with:
 

jupyterlab/commands.py

@@ -13,7 +13,6 @@ import json
 import logging
 import os
 import os.path as osp
-from os.path import join as pjoin
 import re
 import shutil
 import site
@@ -34,6 +33,15 @@ from .process import Process, WatchHelper
 # The regex for expecting the webpack output.
 WEBPACK_EXPECT = re.compile(r'.*/index.out.js')
 
+# The dev mode directory.
+DEV_DIR = osp.realpath(os.path.join(HERE, '..', 'dev_mode'))
+
+
+def pjoin(*args):
+    """Join paths to create a real path.
+    """
+    return osp.realpath(osp.join(*args))
+
 
 def get_user_settings_dir():
     """Get the configured JupyterLab app directory.
@@ -72,13 +80,26 @@ def get_app_dir():
     return osp.realpath(app_dir)
 
 
-def watch_dev(cwd, logger=None):
+def ensure_dev(logger=None):
+    """Ensure that the dev assets are available.
+    """
+    parent = pjoin(HERE, '..')
+
+    if not osp.exists(pjoin(parent, 'node_modules')):
+        yarn_proc = Process(['node', YARN_PATH], cwd=parent, logger=logger)
+        yarn_proc.wait()
+
+    if not osp.exists(pjoin(parent, 'dev_mode', 'build')):
+        yarn_proc = Process(['node', YARN_PATH, 'build'], cwd=parent,
+                            logger=logger)
+        yarn_proc.wait()
+
+
+def watch_dev(logger=None):
     """Run watch mode in a given directory.
 
     Parameters
     ----------
-    cwd: string
-        The current working directory.
     logger: :class:`~logger.Logger`, optional
         The logger instance.
 
@@ -86,6 +107,12 @@ def watch_dev(cwd, logger=None):
     -------
     A list of `WatchHelper` objects.
     """
+    parent = pjoin(HERE, '..')
+
+    if not osp.exists(pjoin(parent, 'node_modules')):
+        yarn_proc = Process(['node', YARN_PATH], cwd=parent, logger=logger)
+        yarn_proc.wait()
+
     logger = logger or logging.getLogger('jupyterlab')
     ts_dir = osp.realpath(osp.join(HERE, '..', 'packages', 'metapackage'))
 
@@ -101,7 +128,8 @@ def watch_dev(cwd, logger=None):
 
     # Run webpack watch and wait for compilation.
     wp_proc = WatchHelper(['node', YARN_PATH, 'run', 'watch'],
-        cwd=HERE, logger=logger, startup_regex=WEBPACK_EXPECT)
+        cwd=DEV_DIR, logger=logger,
+        startup_regex=WEBPACK_EXPECT)
 
     return [ts_proc, tsf_proc, wp_proc]
 
@@ -143,7 +171,9 @@ def uninstall_extension(name, app_dir=None, logger=None):
 def clean(app_dir=None):
     """Clean the JupyterLab application directory."""
     app_dir = app_dir or get_app_dir()
-    if app_dir == HERE:
+    if app_dir == pjoin(HERE, 'dev'):
+        raise ValueError('Cannot clean the dev app')
+    if app_dir == pjoin(HERE, 'core'):
         raise ValueError('Cannot clean the core app')
     for name in ['static', 'staging']:
         target = pjoin(app_dir, name)
@@ -152,12 +182,12 @@ def clean(app_dir=None):
 
 
 def build(app_dir=None, name=None, version=None, logger=None,
-        updating=False, command='build:prod', kill_event=None,
+        command='build:prod', kill_event=None,
         clean_staging=False):
     """Build the JupyterLab application.
     """
     handler = _AppHandler(app_dir, logger, kill_event=kill_event)
-    handler.build(name=name, version=version, updating=updating,
+    handler.build(name=name, version=version,
                   command=command, clean_staging=clean_staging)
 
 
@@ -224,7 +254,7 @@ def get_app_version():
 class _AppHandler(object):
 
     def __init__(self, app_dir, logger=None, kill_event=None):
-        if app_dir == HERE:
+        if app_dir and app_dir.startswith(HERE):
             raise ValueError('Cannot run lab extension commands in core app')
         self.app_dir = app_dir or get_app_dir()
         self.sys_dir = get_app_dir()
@@ -272,15 +302,15 @@ class _AppHandler(object):
             if other['path'] != info['path'] and other['location'] == 'app':
                 os.remove(other['path'])
 
-    def build(self, name=None, version=None, updating=False,
-              command='build:prod', clean_staging=False):
+    def build(self, name=None, version=None, command='build:prod',
+            clean_staging=False):
         """Build the application.
         """
         # Set up the build directory.
         app_dir = self.app_dir
 
         self._populate_staging(
-            name=name, version=version, updating=updating, clean=clean_staging
+            name=name, version=version, clean=clean_staging
         )
 
         staging = pjoin(app_dir, 'staging')
@@ -381,7 +411,7 @@ class _AppHandler(object):
             return [msg % (static_version, core_version)]
 
         # Look for mismatched extensions.
-        new_package = self._get_package_template()
+        new_package = self._get_package_template(silent=fast)
         new_jlab = new_package['jupyterlab']
         new_deps = new_package.get('dependencies', dict())
 
@@ -462,15 +492,22 @@ class _AppHandler(object):
     def link_package(self, path):
         """Link a package at the given path.
         """
+        path = _normalize_path(path)
         if not osp.exists(path) or not osp.isdir(path):
-            raise ValueError('Can only link local directories')
+            msg = 'Can install "%s" only link local directories'
+            raise ValueError(msg % path)
 
         with TemporaryDirectory() as tempdir:
             info = self._extract_package(path, tempdir)
 
-        if _is_extension(info['data']):
+        messages = _validate_extension(info['data'])
+        if not messages:
             return self.install_extension(path)
 
+        # Warn that it is a linked package.
+        self.logger.warn('Installing %s as a linked package:', path)
+        [self.logger.warn(m) for m in messages]
+
         # Add to metadata.
         config = self._read_build_config()
         linked = config.setdefault('linked_packages', dict())
@@ -480,6 +517,7 @@ class _AppHandler(object):
     def unlink_package(self, path):
         """Link a package by name or at the given path.
         """
+        path = _normalize_path(path)
         config = self._read_build_config()
         linked = config.setdefault('linked_packages', dict())
 
@@ -551,8 +589,7 @@ class _AppHandler(object):
         info['disabled_core'] = disabled_core
         return info
 
-    def _populate_staging(self, name=None, version=None, updating=False,
-            clean=False):
+    def _populate_staging(self, name=None, version=None, clean=False):
         """Set up the assets in the staging directory.
         """
         app_dir = self.app_dir
@@ -567,7 +604,7 @@ class _AppHandler(object):
 
         # Look for mismatched version.
         pkg_path = pjoin(staging, 'package.json')
-        overwrite_lock = not updating
+        overwrite_lock = False
 
         if osp.exists(pkg_path):
             with open(pkg_path) as fid:
@@ -578,39 +615,35 @@ class _AppHandler(object):
             else:
                 overwrite_lock = False
 
-        for fname in ['index.app.js', 'webpack.config.js',
-                'yarn.app.lock', '.yarnrc', 'yarn.js']:
-            if fname == 'yarn.app.lock' and not overwrite_lock:
+        for fname in ['index.js', 'webpack.config.js',
+                'yarn.lock', '.yarnrc', 'yarn.js']:
+            if fname == 'yarn.lock' and not overwrite_lock:
                 continue
-            dest = pjoin(staging, fname.replace('.app', ''))
-            shutil.copy(pjoin(HERE, fname), dest)
+            shutil.copy(pjoin(HERE, 'staging', fname), pjoin(staging, fname))
 
-        # Ensure a clean linked packes directory.
+        # Ensure a clean linked packages directory.
         linked_dir = pjoin(staging, 'linked_packages')
         if osp.exists(linked_dir):
             shutil.rmtree(linked_dir)
         os.makedirs(linked_dir)
 
         # Template the package.json file.
-        if updating:
-            data = self.info['core_data']
-        else:
-            # Update the local extensions.
-            extensions = self.info['extensions']
-            for (key, source) in self.info['local_extensions'].items():
-                dname = pjoin(app_dir, 'extensions')
-                self._update_local(key, source, dname, extensions[key],
-                    'local_extensions')
-
-            # Update the linked packages.
-            linked = self.info['linked_packages']
-            for (key, item) in linked.items():
-                dname = pjoin(staging, 'linked_packages')
-                self._update_local(key, item['source'], dname, item,
-                    'linked_packages')
-
-            # Then get the package template.
-            data = self._get_package_template()
+        # Update the local extensions.
+        extensions = self.info['extensions']
+        for (key, source) in self.info['local_extensions'].items():
+            dname = pjoin(app_dir, 'extensions')
+            self._update_local(key, source, dname, extensions[key],
+                'local_extensions')
+
+        # Update the linked packages.
+        linked = self.info['linked_packages']
+        for (key, item) in linked.items():
+            dname = pjoin(staging, 'linked_packages')
+            self._update_local(key, item['source'], dname, item,
+                'linked_packages')
+
+        # Then get the package template.
+        data = self._get_package_template()
 
         if version:
             data['jupyterlab']['version'] = version
@@ -622,7 +655,7 @@ class _AppHandler(object):
         with open(pkg_path, 'w') as fid:
             json.dump(data, fid, indent=4)
 
-    def _get_package_template(self):
+    def _get_package_template(self, silent=False):
         """Get the template the for staging package.json file.
         """
         logger = self.logger
@@ -648,7 +681,8 @@ class _AppHandler(object):
                 msg = _format_compatibility_errors(
                     key, value['version'], errors
                 )
-                logger.warn(msg + '\n')
+                if not silent:
+                    logger.warn(msg + '\n')
                 continue
 
             data['dependencies'][key] = format_path(value['path'])
@@ -915,8 +949,10 @@ class _AppHandler(object):
         data = info['data']
 
         # Verify that the package is an extension.
-        if not _is_extension(data):
-            raise ValueError('Not a valid extension')
+        messages = _validate_extension(data)
+        if messages:
+            msg = '"%s" is not a valid extension:\n%s'
+            raise ValueError(msg % (extension, '\n'.join(messages)))
 
         # Verify package compatibility.
         core_data = _get_core_data()
@@ -946,13 +982,12 @@ class _AppHandler(object):
 
         info = dict(source=source, is_dir=is_dir)
 
-        self._run([which('npm'), 'pack', source], cwd=tempdir)
-
-        paths = glob.glob(pjoin(tempdir, '*.tgz'))
-        if not paths:
-            raise ValueError('Extension failed to pack')
+        ret = self._run([which('npm'), 'pack', source], cwd=tempdir)
+        if ret != 0:
+            msg = '"%s" is not a valid npm package'
+            raise ValueError(msg % source)
 
-        path = paths[0]
+        path = glob.glob(pjoin(tempdir, '*.tgz'))[0]
         info['data'] = _read_package(path)
         if is_dir:
             info['sha'] = sha = _tarsum(path)
@@ -970,6 +1005,8 @@ class _AppHandler(object):
 
     def _run(self, cmd, **kwargs):
         """Run the command using our logger and abort callback.
+
+        Returns the exit code.
         """
         if self.kill_event.is_set():
             raise ValueError('Command was killed')
@@ -977,7 +1014,7 @@ class _AppHandler(object):
         kwargs['logger'] = self.logger
         kwargs['kill_event'] = self.kill_event
         proc = Process(cmd, **kwargs)
-        proc.wait()
+        return proc.wait()
 
 
 def _normalize_path(extension):
@@ -995,20 +1032,58 @@ def _read_package(target):
     tar = tarfile.open(target, "r:gz")
     f = tar.extractfile('package/package.json')
     data = json.loads(f.read().decode('utf8'))
+    data['jupyterlab_extracted_files'] = [
+        f.path[len('package/'):] for f in tar.getmembers()
+    ]
     tar.close()
     return data
 
 
-def _is_extension(data):
+def _validate_extension(data):
     """Detect if a package is an extension using its metadata.
+
+    Returns any problems it finds.
     """
-    if 'jupyterlab' not in data:
-        return False
-    if not isinstance(data['jupyterlab'], dict):
-        return False
-    is_extension = data['jupyterlab'].get('extension', False)
-    is_mime_extension = data['jupyterlab'].get('mimeExtension', False)
-    return is_extension or is_mime_extension
+    jlab = data.get('jupyterlab', None)
+    if jlab is None:
+        return ['No `jupyterlab` key']
+    if not isinstance(jlab, dict):
+        return ['The `jupyterlab` key must be a JSON object']
+    extension = jlab.get('extension', False)
+    mime_extension = jlab.get('mimeExtension', False)
+    themeDir = jlab.get('themeDir', '')
+    schemaDir = jlab.get('schemaDir', '')
+
+    messages = []
+    if not extension and not mime_extension:
+        messages.append('No `extension` or `mimeExtension` key present')
+
+    if extension == mime_extension:
+        msg = '`mimeExtension` and `extension` must point to different modules'
+        messages.append(msg)
+
+    files = data['jupyterlab_extracted_files']
+    main = data.get('main', 'index.js')
+
+    if extension is True:
+        if main not in files:
+            messages.append('Missing extension module "%s"' % main)
+    elif extension and extension not in files:
+        messages.append('Missing extension module "%s"' % extension)
+
+    if mime_extension is True:
+        if main not in files:
+            messages.append('Missing mimeExtension module "%s"' % main)
+    elif mime_extension and mime_extension not in files:
+        messages.append('Missing mimeExtension module "%s"' % mime_extension)
+
+    if themeDir and not any(f.startswith(themeDir) for f in files):
+        messages.append('themeDir is empty: "%s"' % themeDir)
+
+    if schemaDir and not any(f.startswith(schemaDir) for f in files):
+        messages.append('schemaDir is empty: "%s"' % schemaDir)
+
+    return messages
 
 
 def _tarsum(input_file):
@@ -1033,7 +1108,7 @@ def _tarsum(input_file):
 def _get_core_data():
     """Get the data for the app template.
     """
-    with open(pjoin(HERE, 'package.app.json')) as fid:
+    with open(pjoin(HERE, 'staging', 'package.json')) as fid:
         return json.load(fid)
 
 

jupyterlab/extension.py

@@ -7,11 +7,12 @@
 #-----------------------------------------------------------------------------
 # Module globals
 #-----------------------------------------------------------------------------
+import os
 
-DEV_NOTE_NPM = """You're running JupyterLab from source.
+DEV_NOTE = """You're running JupyterLab from source.
 If you're working on the TypeScript sources of JupyterLab, try running
 
-    jlpm run --dev-mode watch
+    jupyter lab --dev-mode --watch
 
 
 to have the system incrementally watch and build JupyterLab for you, as you
@@ -28,94 +29,102 @@ def load_jupyter_server_extension(nbapp):
     """Load the JupyterLab server extension.
     """
     # Delay imports to speed up jlpmapp
-    import os
     from jupyterlab_launcher import add_handlers, LabConfig
     from notebook.utils import url_path_join as ujoin
     from tornado.ioloop import IOLoop
     from .build_handler import build_path, Builder, BuildHandler
     from .commands import (
-        get_app_dir, get_user_settings_dir, watch, watch_dev
+        get_app_dir, get_user_settings_dir, watch, ensure_dev, watch_dev,
+        pjoin, DEV_DIR, HERE, get_app_version
     )
     from ._version import __version__
 
-    # Print messages.
-    here = os.path.dirname(__file__)
-    nbapp.log.info('JupyterLab alpha preview extension loaded from %s' % here)
-
-    if hasattr(nbapp, 'app_dir'):
-        app_dir = nbapp.app_dir or get_app_dir()
-    else:
-        app_dir = get_app_dir()
-
     web_app = nbapp.web_app
+    logger = nbapp.log
     config = LabConfig()
 
+    # Print messages.
+    logger.info('JupyterLab alpha preview extension loaded from %s' % HERE)
+
+    app_dir = getattr(nbapp, 'app_dir', get_app_dir())
+
     config.name = 'JupyterLab'
-    config.assets_dir = os.path.join(app_dir, 'static')
-    config.settings_dir = os.path.join(app_dir, 'settings')
     config.page_title = 'JupyterLab Alpha Preview'
     config.page_url = '/lab'
-    config.dev_mode = False
 
     # Check for core mode.
-    core_mode = ''
-    if hasattr(nbapp, 'core_mode'):
-        core_mode = nbapp.core_mode
+    core_mode = False
+    if getattr(nbapp, 'core_mode', False) or app_dir.startswith(HERE):
+        core_mode = True
+
+    # Check for dev mode.
+    dev_mode = False
+    if getattr(nbapp, 'dev_mode', False) or app_dir.startswith(DEV_DIR):
+        dev_mode = True
 
     # Check for watch.
-    watch_mode = False
-    if hasattr(nbapp, 'watch'):
-        watch_mode = nbapp.watch
+    watch_mode = getattr(nbapp, 'watch', False)
 
-    # Check for an app dir that is local.
-    if app_dir == here or app_dir == os.path.join(here, 'build'):
-        core_mode = True
-        config.settings_dir = ''
+    if watch_mode and core_mode:
+        logger.warn('Cannot watch in core mode, did you mean --dev-mode?')
+        watch_mode = False
+
+    if core_mode and dev_mode:
+        logger.warn('Conflicting modes, choosing dev_mode over core_mode')
+        core_mode = False
 
     page_config = web_app.settings.setdefault('page_config_data', dict())
-    page_config['buildAvailable'] = not core_mode
-    page_config['buildCheck'] = not core_mode
+    page_config['buildAvailable'] = not core_mode and not dev_mode
+    page_config['buildCheck'] = not core_mode and not dev_mode
     page_config['token'] = nbapp.token
 
     if core_mode:
-        config.assets_dir = os.path.join(here, 'build')
-        config.version = __version__
-        if not os.path.exists(config.assets_dir) and not watch_mode:
+        config.assets_dir = pjoin(HERE, 'static')
+        config.schemas_dir = pjoin(HERE, 'schemas')
+        config.settings_dir = ''
+        config.themes_dir = pjoin(HERE, 'themes')
+        config.version = get_app_version()
+
+        logger.info(CORE_NOTE.strip())
+        if not os.path.exists(config.assets_dir):
             msg = 'Static assets not built, please see CONTRIBUTING.md'
-            nbapp.log.error(msg)
-        else:
-            sentinel = os.path.join(here, 'build', 'release_data.json')
-            config.dev_mode = not os.path.exists(sentinel)
+            logger.error(msg)
 
-    if config.dev_mode and not watch_mode:
-        nbapp.log.info(DEV_NOTE_NPM)
-    elif core_mode:
-        nbapp.log.info(CORE_NOTE.strip())
+    elif dev_mode:
+        config.assets_dir = pjoin(DEV_DIR, 'build')
+        config.schemas_dir = pjoin(DEV_DIR, 'schemas')
+        config.settings_dir = ''
+        config.themes_dir = pjoin(DEV_DIR, 'themes')
+        config.version = __version__
+
+        ensure_dev(logger)
+        if not watch_mode:
+            logger.info(DEV_NOTE)
 
-    if core_mode:
-        schemas_dir = os.path.join(here, 'schemas')
-        config.themes_dir = os.path.join(here, 'themes')
     else:
-        schemas_dir = os.path.join(app_dir, 'schemas')
-        config.themes_dir = os.path.join(app_dir, 'themes')
+        config.assets_dir = pjoin(app_dir, 'static')
+        config.schemas_dir = pjoin(app_dir, 'schemas')
+        config.settings_dir = pjoin(app_dir, 'settings')
+        config.themes_dir = pjoin(app_dir, 'themes')
+        config.version = get_app_version()
 
-    config.schemas_dir = schemas_dir
+    config.dev_mode = dev_mode
     config.user_settings_dir = get_user_settings_dir()
 
     if watch_mode:
         # Set the ioloop in case the watch fails.
         nbapp.ioloop = IOLoop.current()
         if config.dev_mode:
-            watch_dev(nbapp.log)
+            watch_dev(logger)
         else:
-            watch(app_dir, nbapp.log)
+            watch(app_dir, logger)
             page_config['buildAvailable'] = False
 
     add_handlers(web_app, config)
 
     base_url = web_app.settings['base_url']
     build_url = ujoin(base_url, build_path)
-    builder = Builder(nbapp.log, core_mode, app_dir)
+    builder = Builder(logger, core_mode, app_dir)
     build_handler = (build_url, BuildHandler, {'builder': builder})
 
     web_app.add_handlers(".*$", [build_handler])

jupyterlab/jlpmapp.py

@@ -14,7 +14,7 @@ except ImportError:
     import subprocess
 
 HERE = os.path.dirname(os.path.abspath(__file__))
-YARN_PATH = os.path.join(HERE, 'yarn.js')
+YARN_PATH = os.path.join(HERE, 'staging', 'yarn.js')
 
 
 def execvp(cmd, argv):

jupyterlab/labapp.py

@@ -12,7 +12,8 @@ from traitlets import Bool, Unicode
 from ._version import __version__
 from .extension import load_jupyter_server_extension
 from .commands import (
-    build, clean, get_app_dir, get_user_settings_dir, get_app_version
+    build, clean, get_app_dir, get_user_settings_dir, get_app_version,
+    ensure_dev
 )
 
 
@@ -98,7 +99,7 @@ lab_flags['core-mode'] = (
     "Start the app in core mode."
 )
 lab_flags['dev-mode'] = (
-    {'LabApp': {'core_mode': True}},
+    {'LabApp': {'dev_mode': True}},
     "Start the app in dev mode for running from source."
 )
 lab_flags['watch'] = (
@@ -122,9 +123,9 @@ class LabApp(NotebookApp):
       assets contained in the installed `jupyterlab` Python package. In core mode, no
       extensions are enabled. This is the default in a stable JupyterLab release if you
       have no extensions installed.
-    * Dev mode (`--dev-mode`): like core mode, but when the `jupyterlab` Python package
-      is installed from source and installed using `pip install -e .`. In this case
-      JupyterLab will show a red stripe at the top of the page.
+    * Dev mode (`--dev-mode`): uses the unpublished local JavaScript packages
+        in the `dev_mode` folder.  In this case JupyterLab will show a red stripe at the top of the page.  It can only be used if JupyterLab
+        is installed as `pip install -e .`.
     * App mode: JupyterLab allows multiple JupyterLab "applications" to be
       created by the user with different combinations of extensions. The `--app-dir` can
       be used to set a directory for different applications. The default application
@@ -152,7 +153,7 @@ class LabApp(NotebookApp):
     default_url = Unicode('/lab', config=True,
         help="The default URL to redirect to from `/`")
 
-    app_dir = Unicode('', config=True,
+    app_dir = Unicode(get_app_dir(), config=True,
         help="The app directory to launch JupyterLab from.")
 
     core_mode = Bool(False, config=True,
@@ -163,6 +164,12 @@ class LabApp(NotebookApp):
         itself is installed in development mode (`pip install -e .`).
         """)
 
+    dev_mode = Bool(False, config=True,
+        help="""Whether to start the app in dev mode. Uses the unpublished local JavaScript packages
+        in the `dev_mode` folder.  In this case JupyterLab will show a red stripe at the top of the page.  It can only be used if JupyterLab
+        is installed as `pip install -e .`.
+        """)
+
     watch = Bool(False, config=True,
         help="Whether to serve the app in watch mode")
 

jupyterlab/labextensions.py

@@ -5,9 +5,9 @@
 # Distributed under the terms of the Modified BSD License.
 from __future__ import print_function
 
-import json
 import os
 import sys
+import traceback
 
 from jupyter_core.application import JupyterApp, base_flags, base_aliases
 
@@ -16,7 +16,7 @@ from traitlets import Bool, Unicode
 from .commands import (
     install_extension, uninstall_extension, list_extensions,
     enable_extension, disable_extension,
-    link_package, unlink_package, build
+    link_package, unlink_package, build, get_app_version
 )
 
 
@@ -33,11 +33,7 @@ flags['clean'] = (
 aliases = dict(base_aliases)
 aliases['app-dir'] = 'BaseExtensionApp.app_dir'
 
-
-here = os.path.dirname(__file__)
-with open(os.path.join(here, 'package.app.json')) as fid:
-    data = json.load(fid)
-VERSION = data['jupyterlab']['version']
+VERSION = get_app_version()
 
 
 class BaseExtensionApp(JupyterApp):
@@ -54,6 +50,20 @@ class BaseExtensionApp(JupyterApp):
     should_clean = Bool(False, config=True,
         help="Whether temporary files should be cleaned up after building jupyterlab")
 
+    def start(self):
+        try:
+            self.run_task()
+        except Exception as ex:
+            ex_traceback = ex.__traceback__
+            msg = traceback.format_exception(ex.__class__, ex, ex_traceback)
+            for line in msg:
+                self.log.debug(line)
+            self.log.error(str(ex))
+            sys.exit(1)
+
+    def run_task(self):
+        pass
+
     def _log_format_default(self):
         """A default format for messages"""
         return "%(message)s"
@@ -64,17 +74,14 @@ class InstallLabExtensionApp(BaseExtensionApp):
     should_build = Bool(True, config=True,
         help="Whether to build the app after the action")
 
-    def start(self):
+    def run_task(self):
         self.extra_args = self.extra_args or [os.getcwd()]
-        [install_extension(arg, self.app_dir, logger=self.log)
-         for arg in self.extra_args]
+        [install_extension(arg, self.app_dir, logger=self.log) for
+         arg in self.extra_args]
+
         if self.should_build:
-            try:
-                build(self.app_dir, clean_staging=self.should_clean, logger=self.log)
-            except Exception as e:
-                for arg in self.extra_args:
-                    uninstall_extension(arg, self.app_dir, logger=self.log)
-                raise e
+            build(self.app_dir, clean_staging=self.should_clean,
+                 logger=self.log)
 
 
 class LinkLabExtensionApp(BaseExtensionApp):
@@ -88,17 +95,14 @@ class LinkLabExtensionApp(BaseExtensionApp):
     should_build = Bool(True, config=True,
         help="Whether to build the app after the action")
 
-    def start(self):
+    def run_task(self):
         self.extra_args = self.extra_args or [os.getcwd()]
         [link_package(arg, self.app_dir, logger=self.log)
          for arg in self.extra_args]
+
         if self.should_build:
-            try:
-                build(self.app_dir, clean_staging=self.should_clean, logger=self.log)
-            except Exception as e:
-                for arg in self.extra_args:
-                    unlink_package(arg, self.app_dir, logger=self.log)
-                raise e
+            build(self.app_dir, clean_staging=self.should_clean,
+                  logger=self.log)
 
 
 class UnlinkLabExtensionApp(BaseExtensionApp):
@@ -106,12 +110,13 @@ class UnlinkLabExtensionApp(BaseExtensionApp):
     should_build = Bool(True, config=True,
         help="Whether to build the app after the action")
 
-    def start(self):
+    def run_task(self):
         self.extra_args = self.extra_args or [os.getcwd()]
         ans = any([unlink_package(arg, self.app_dir, logger=self.log)
                    for arg in self.extra_args])
         if ans and self.should_build:
-            build(self.app_dir, clean_staging=self.should_clean, logger=self.log)
+            build(self.app_dir, clean_staging=self.should_clean,
+                  logger=self.log)
 
 
 class UninstallLabExtensionApp(BaseExtensionApp):
@@ -119,25 +124,26 @@ class UninstallLabExtensionApp(BaseExtensionApp):
     should_build = Bool(True, config=True,
         help="Whether to build the app after the action")
 
-    def start(self):
+    def run_task(self):
         self.extra_args = self.extra_args or [os.getcwd()]
         ans = any([uninstall_extension(arg, self.app_dir, logger=self.log)
                    for arg in self.extra_args])
         if ans and self.should_build:
-            build(self.app_dir, clean_staging=self.should_clean, logger=self.log)
+            build(self.app_dir, clean_staging=self.should_clean,
+                  logger=self.log)
 
 
 class ListLabExtensionsApp(BaseExtensionApp):
     description = "List the installed labextensions"
 
-    def start(self):
+    def run_task(self):
         list_extensions(self.app_dir, logger=self.log)
 
 
 class EnableLabExtensionsApp(BaseExtensionApp):
     description = "Enable labextension(s) by name"
 
-    def start(self):
+    def run_task(self):
         [enable_extension(arg, self.app_dir, logger=self.log)
          for arg in self.extra_args]
 
@@ -145,7 +151,7 @@ class EnableLabExtensionsApp(BaseExtensionApp):
 class DisableLabExtensionsApp(BaseExtensionApp):
     description = "Disable labextension(s) by name"
 
-    def start(self):
+    def run_task(self):
         [disable_extension(arg, self.app_dir, logger=self.log)
          for arg in self.extra_args]
 

jupyterlab/node-version-check.js

@@ -1,5 +0,0 @@
-#!/usr/bin/env node
-var version = parseInt(process.version.replace('v', ''));
-if (version < 5) {
-  process.exit(1);
-}

jupyterlab/process.py

@@ -81,16 +81,11 @@ class Process(object):
 
         # Kill the process.
         if proc.poll() is None:
-            try:
-                os.kill(proc.pid, signal.SIGTERM)
-            except Exception as e:
-                self.logger.error(str(e))
+            os.kill(proc.pid, signal.SIGTERM)
 
         # Wait for the process to close.
         try:
             proc.wait()
-        except Exception as e:
-            self.logger.error(e)
         finally:
             Process._procs.remove(self)
 
@@ -105,18 +100,11 @@ class Process(object):
         """
         proc = self.proc
         kill_event = self._kill_event
-        try:
-            while proc.poll() is None:
-                if kill_event.is_set():
-                    self.terminate()
-                    raise ValueError('Process Aborted')
-                time.sleep(1.)
-        except subprocess.CalledProcessError as error:
-            output = error.output.decode('utf-8')
-            self.logger.error(output)
-            self.terminate()
-            raise error
-
+        while proc.poll() is None:
+            if kill_event.is_set():
+                self.terminate()
+                raise ValueError('Process was aborted')
+            time.sleep(1.)
         return self.terminate()
 
     @gen.coroutine
@@ -125,17 +113,11 @@ class Process(object):
         """
         proc = self.proc
         kill_event = self._kill_event
-        try:
-            while proc.poll() is None:
-                if kill_event.is_set():
-                    self.terminate()
-                    raise ValueError('Process Aborted')
-                yield gen.sleep(1.)
-        except subprocess.CalledProcessError as error:
-            output = error.output.decode('utf-8')
-            self.logger.error(output)
-            self.terminate()
-            raise error
+        while proc.poll() is None:
+            if kill_event.is_set():
+                self.terminate()
+                raise ValueError('Process was aborted')
+            yield gen.sleep(1.)
 
         raise gen.Return(self.terminate())
 
@@ -149,14 +131,7 @@ class Process(object):
             kwargs['shell'] = True
 
         cmd[0] = which(cmd[0], kwargs.get('env'))
-
-        try:
-            proc = subprocess.Popen(cmd, **kwargs)
-        except subprocess.CalledProcessError as error:
-            output = error.output.decode('utf-8')
-            self.logger.error(output)
-            raise error
-
+        proc = subprocess.Popen(cmd, **kwargs)
         return proc
 
     @classmethod
@@ -229,9 +204,6 @@ class WatchHelper(Process):
         # Wait for the process to close.
         try:
             proc.wait()
-        except Exception as e:
-            print('on close')
-            self.logger.error(e)
         finally:
             Process._procs.remove(self)
 

jupyterlab/released_packages.txt

@@ -1,64 +0,0 @@
-@jupyterlab/application-top          v0.6.1  (private)
-@jupyterlab/example-app              v0.7.1  (private)
-@jupyterlab/example-console          v0.6.0  (private)
-@jupyterlab/example-filebrowser      v0.6.1  (private)
-@jupyterlab/example-notebook         v0.6.1  (private)
-jupyterlab-example-terminal          v0.6.0  (private)
-@jupyterlab/all-packages             v0.6.1  (private)
-@jupyterlab/application-extension    v0.6.0           
-@jupyterlab/application              v0.6.0           
-@jupyterlab/apputils-extension       v0.6.0           
-@jupyterlab/apputils                 v0.6.0           
-@jupyterlab/cells                    v0.6.0           
-@jupyterlab/chatbox-extension        v0.1.2           
-@jupyterlab/chatbox                  v0.1.1           
-@jupyterlab/codeeditor               v0.6.0           
-@jupyterlab/codemirror-extension     v0.6.0           
-@jupyterlab/codemirror               v0.6.0           
-@jupyterlab/completer-extension      v0.6.0           
-@jupyterlab/completer                v0.6.0           
-@jupyterlab/console-extension        v0.6.1           
-@jupyterlab/console                  v0.6.0           
-@jupyterlab/coreutils                v0.6.0           
-@jupyterlab/csvviewer-extension      v0.6.0           
-@jupyterlab/csvviewer                v0.6.0           
-@jupyterlab/docmanager-extension     v0.6.1           
-@jupyterlab/docmanager               v0.6.1           
-@jupyterlab/docregistry-extension    v0.6.0           
-@jupyterlab/docregistry              v0.6.0           
-@jupyterlab/faq-extension            v0.6.0           
-@jupyterlab/filebrowser-extension    v0.6.1           
-@jupyterlab/filebrowser              v0.6.1           
-@jupyterlab/fileeditor-extension     v0.6.0           
-@jupyterlab/fileeditor               v0.6.0           
-@jupyterlab/help-extension           v0.6.0           
-@jupyterlab/imageviewer-extension    v0.6.0           
-@jupyterlab/imageviewer              v0.6.0           
-@jupyterlab/inspector-extension      v0.6.0           
-@jupyterlab/inspector                v0.6.0           
-@jupyterlab/landing-extension        v0.7.1           
-@jupyterlab/launcher-extension       v0.6.1           
-@jupyterlab/launcher                 v0.6.0           
-@jupyterlab/markdownviewer-extension v0.6.0           
-@jupyterlab/markdownviewer           v0.6.0           
-@jupyterlab/notebook-extension       v0.6.0           
-@jupyterlab/notebook                 v0.6.0           
-@jupyterlab/outputarea               v0.6.0           
-@jupyterlab/rendermime-extension     v0.6.0           
-@jupyterlab/rendermime               v0.6.0           
-@jupyterlab/running-extension        v0.6.0           
-@jupyterlab/running                  v0.6.0           
-@jupyterlab/services-extension       v0.6.0           
-@jupyterlab/services                 v0.45.0          
-@jupyterlab/shortcuts-extension      v0.6.0           
-@jupyterlab/tabmanager-extension     v0.6.0           
-@jupyterlab/terminal-extension       v0.6.0           
-@jupyterlab/terminal                 v0.6.0           
-@jupyterlab/theme-dark-extension     v0.6.0           
-@jupyterlab/theme-light-extension    v0.6.0           
-@jupyterlab/theming                  v0.6.0           
-@jupyterlab/tooltip-extension        v0.6.0           
-@jupyterlab/tooltip                  v0.6.0           
-node-example                         v0.2.0  (private)
-browser-example                      v0.2.0  (private)
-@jupyterlab/test-all                 v0.6.1  (private)

jupyterlab/selenium_check.py

@@ -89,6 +89,10 @@ test_flags['core-mode'] = (
     {'SeleniumApp': {'core_mode': True}},
     "Start the app in core mode."
 )
+test_flags['dev-mode'] = (
+    {'SeleniumApp': {'dev_mode': True}},
+    "Start the app in dev mode."
+)
 
 
 test_aliases = dict(aliases)
@@ -103,12 +107,6 @@ class SeleniumApp(LabApp):
     flags = test_flags
     aliases = test_aliases
 
-    core_mode = Bool(False, config=True,
-        help="Whether to start the app in core mode")
-
-    app_dir = Unicode('', config=True,
-        help="The app directory to build in")
-
     def start(self):
         web_app = self.web_app
         web_app.settings.setdefault('page_config_data', dict())

jupyterlab/staging/webpack.config.js

@@ -0,0 +1,155 @@
+var webpack = require('webpack');
+var path = require('path');
+var fs = require('fs-extra');
+var Handlebars = require('handlebars');
+var Build = require('@jupyterlab/buildutils').Build;
+var package_data = require('./package.json');
+
+// Ensure a clear build directory.
+var buildDir = path.resolve('./build');
+if (fs.existsSync(buildDir)) {
+  fs.removeSync(buildDir);
+}
+fs.ensureDirSync(buildDir);
+
+// Handle the extensions.
+var jlab = package_data.jupyterlab;
+var extensions = jlab.extensions;
+var mimeExtensions = jlab.mimeExtensions;
+Build.ensureAssets({
+  packageNames: Object.keys(mimeExtensions).concat(Object.keys(extensions)),
+  output: jlab.outputDir
+});
+
+// Create the entry point file.
+var source = fs.readFileSync('index.js').toString();
+var template = Handlebars.compile(source);
+var data = {
+  jupyterlab_extensions: extensions,
+  jupyterlab_mime_extensions: mimeExtensions,
+};
+var result = template(data);
+
+fs.writeFileSync(path.join(buildDir, 'index.out.js'), result);
+fs.copySync('./package.json', path.join(buildDir, 'package.json'));
+
+
+// Set up variables for watch mode.
+var localLinked = {};
+var ignoreCache = new Map();
+Object.keys(jlab.linkedPackages).forEach(function (name) {
+  var localPath = require.resolve(path.join(name, 'package.json'));
+  localLinked[name] = path.dirname(localPath);
+});
+
+
+/**
+ * Sync a local path to a linked package path if they are files and differ.
+ */
+function maybeSync(localPath, name, rest) {
+  var stats = fs.statSync(localPath);
+  if (!stats.isFile(localPath)) {
+    return;
+  }
+  var source = fs.realpathSync(path.join(jlab.linkedPackages[name], rest));
+  if (source === fs.realpathSync(localPath)) {
+    return;
+  }
+  fs.watchFile(source, { 'interval': 500 }, function(curr) {
+    if (!curr || curr.nlink === 0) {
+      return;
+    }
+    try {
+      console.log('updating', path.join(name, rest));
+      fs.copySync(source, localPath);
+    } catch (err) {
+      console.error(err);
+    }
+  });
+}
+
+
+/**
+ * A WebPack Plugin that copies the assets to the static directory.
+ */
+function JupyterLabPlugin(options) {
+  _first = true;
+}
+
+JupyterLabPlugin.prototype.apply = function(compiler) {
+
+  compiler.plugin('after-emit', function(compilation, callback) {
+    var staticDir = jlab.staticDir;
+    if (!staticDir) {
+      callback();
+      return;
+    }
+    // Ensure a clean static directory on the first emit.
+    if (this._first && fs.existsSync(staticDir)) {
+      fs.removeSync(staticDir);
+    }
+    this._first = false;
+    fs.copySync(buildDir, staticDir);
+    callback();
+  }.bind(this));
+};
+
+
+
+module.exports = {
+  entry:  path.resolve(buildDir, 'index.out.js'),
+  output: {
+    path: path.resolve(buildDir),
+    filename: '[name].bundle.js'
+  },
+  module: {
+    rules: [
+      { test: /\.css$/, use: ['style-loader', 'css-loader'] },
+      { test: /\.json$/, use: 'json-loader' },
+      { test: /\.html$/, use: 'file-loader' },
+      { test: /\.md$/, use: 'raw-loader' },
+      { test: /\.js$/, use: ['source-map-loader'], enforce: 'pre',
+        exclude: path.join(process.cwd(), 'node_modules')
+      },
+      { test: /\.(jpg|png|gif)$/, use: 'file-loader' },
+      { test: /\.js.map$/, use: 'file-loader' },
+      { test: /\.woff2(\?v=\d+\.\d+\.\d+)?$/, use: 'url-loader?limit=10000&mimetype=application/font-woff' },
+      { test: /\.woff(\?v=\d+\.\d+\.\d+)?$/, use: 'url-loader?limit=10000&mimetype=application/font-woff' },
+      { test: /\.ttf(\?v=\d+\.\d+\.\d+)?$/, use: 'url-loader?limit=10000&mimetype=application/octet-stream' },
+      { test: /\.eot(\?v=\d+\.\d+\.\d+)?$/, use: 'file-loader' },
+      { test: /\.svg(\?v=\d+\.\d+\.\d+)?$/, use: 'url-loader?limit=10000&mimetype=image/svg+xml' }
+    ],
+  },
+  watchOptions: {
+    ignored: function(localPath) {
+      localPath = path.resolve(localPath);
+      if (ignoreCache.has(localPath)) {
+        return ignoreCache.get(localPath);
+      }
+      // Limit the watched files to those in our local linked package dirs.
+      var ignore = true;
+      Object.keys(localLinked).some(function (name) {
+        // Bail if already found.
+        var rootPath = localLinked[name];
+        var contained = localPath.indexOf(rootPath + path.sep) !== -1;
+        if (localPath !== rootPath && !contained) {
+          return false;
+        }
+        var rest = localPath.slice(rootPath.length);
+        if (rest.indexOf('node_modules') === -1) {
+          ignore = false;
+          maybeSync(localPath, name, rest);
+        }
+        return true;
+      });
+      ignoreCache.set(localPath, ignore);
+      return ignore;
+    }
+  },
+  node: {
+    fs: 'empty'
+  },
+  bail: true,
+  devtool: 'cheap-source-map',
+  plugins: [ new JupyterLabPlugin({}) ]
+}

jupyterlab/tests/test_jupyterlab.py

@@ -77,7 +77,7 @@ class TestExtension(TestCase):
         # Copy in the mock packages.
         for name in ['extension', 'incompat', 'package', 'mimeextension']:
             src = pjoin(here, 'mock_packages', name)
- 
+
             def ignore(dname, files):
                 if 'node_modules' in dname:
                     files = []
@@ -167,6 +167,17 @@ class TestExtension(TestCase):
         extensions = get_app_info(self.app_dir)['extensions']
         assert not data['name'] in extensions
 
+    def test_validation(self):
+        path = self.mock_extension
+        os.remove(pjoin(path, 'index.js'))
+        with pytest.raises(ValueError):
+            install_extension(path)
+
+        path = self.mock_mimeextension
+        os.remove(pjoin(path, 'index.js'))
+        with pytest.raises(ValueError):
+            install_extension(path)
+
     def test_uninstall_extension(self):
         install_extension(self.mock_extension)
         uninstall_extension(self.pkg_names['extension'])

jupyterlab/tsconfig.json

@@ -1,21 +0,0 @@
-{
-  "compilerOptions": {
-    "declaration": true,
-    "noImplicitAny": true,
-    "noEmitOnError": true,
-    "noUnusedLocals": true,
-    "module": "commonjs",
-    "moduleResolution": "node",
-    "target": "ES5",
-    "outDir": "./build",
-    "lib": ["ES5", "ES2015.Collection", "ES2015.Promise", "DOM"],
-    "types": ["node", "text-encoding"],
-    "baseUrl": ".",
-    "paths": {
-      "@jupyterlab/*": ["../packages/*/src"]
-    },
-    "rootDirs": [
-      "../packages"
-    ]
-  }
-}

jupyterlab/update.py

@@ -1,15 +1,40 @@
 # coding: utf-8
 # Copyright (c) Jupyter Development Team.
 # Distributed under the terms of the Modified BSD License.
+from os.path import join as pjoin
+import json
 import os
 import shutil
 
-from .commands import build, get_app_dir
+HERE = os.path.dirname(__file__)
 
-here = os.path.dirname(__file__)
-app_dir = get_app_dir()
 
-build(updating=True)
+# Get the dev mode package.json file.
+dev_path = os.path.realpath(pjoin(HERE, '..', 'dev_mode'))
+with open(pjoin(dev_path, 'package.json')) as fid:
+    data = json.load(fid)
+
+
+# Update the values that need to change and write to staging.
+data['scripts']['build'] = 'webpack'
+data['scripts']['watch'] = 'webpack --watch'
+data['scripts']['build:prod'] = (
+    "webpack --define process.env.NODE_ENV=\"'production'\"")
+data['jupyterlab']['outputDir'] = '..'
+data['jupyterlab']['staticDir'] = '../static'
+data['jupyterlab']['linkedPackages'] = dict()
+
+staging = pjoin(HERE, 'staging')
+
+with open(pjoin(staging, 'package.json'), 'w') as fid:
+    json.dump(data, fid)
+
+# Update our index file and webpack file.
+for fname in ['index.js', 'webpack.config.js']:
+    shutil.copy(pjoin(dev_path, fname), pjoin(staging, fname))
+
+
+# Get a new yarn lock file.
 
 target = os.path.join(app_dir, 'staging', 'yarn.lock')
-shutil.copy(target, os.path.join(here, 'yarn.app.lock'))
+shutil.copy(target, os.path.join(staging, 'yarn.lock'))

package.json

@@ -2,21 +2,24 @@
   "private": true,
   "scripts": {
     "add:sibling": "node buildutils/lib/add-sibling.js",
-    "build": "jlpm run integrity && jlpm run build:packages && cd jupyterlab && jlpm run build",
+    "build": "jlpm run build:dev",
+    "build:core": "cd jupyterlab/staging && jlpm run build",
+    "build:dev": "jlpm run integrity && jlpm run build:packages && cd dev_mode && jlpm run build",
     "build:examples": "lerna run build --scope \"@jupyterlab/example-*\"",
-    "build:main": "jlpm run build",
-    "build:main:prod": "jlpm run build:packages && cd jupyterlab && jlpm run build:prod",
     "build:packages": "cd packages/metapackage && jlpm run build",
     "build:src": "lerna run build --scope \"@jupyterlab/!(test-|example-)*\"",
-    "build:static": "node buildutils/lib/make-release.js && python -m jupyterlab.update",
     "build:test": "lerna run build:test",
+    "build:update": "node buildutils/lib/update-core-mode.js",
     "build:utils": "cd buildutils && jlpm run build",
-    "clean": "node buildutils/lib/clean-packages.js examples packages",
+    "clean": "jlpm run clean:dev",
+    "clean:core": "cd jupyterlab/staging && jlpm run clean",
+    "clean:dev": "cd dev_mode && jlpm run clean",
     "clean:examples": "node buildutils/lib/clean-packages.js examples",
-    "clean:main": "cd jupyterlab && jlpm run clean",
-    "clean:slate": "git clean -dfx && git checkout . && pip install -e . && jlpm install",
-    "clean:src": "node buildutils/lib/clean-packages.js packages",
-    "clean:tests": "lerna run clean --scope \"@jupyterlab/test-*\"",
+    "clean:packages": "node buildutils/lib/clean-packages.js packages",
+    "clean:slate": "python clean.py && pip install -e .",
+    "clean:src": "jlpm run clean",
+    "clean:test": "lerna run clean --scope \"@jupyterlab/test-*\"",
+    "clean:utils": "cd buildutils && jlpm run clean",
     "coverage": "lerna run coverage --stream",
     "create:package": "node buildutils/lib/create-package.js",
     "docs": "lerna run docs",
@@ -24,7 +27,7 @@
     "postinstall": "node scripts/ensure-buildutils.js",
     "integrity": "node buildutils/lib/ensure-repo.js",
     "patch:release": "node buildutils/lib/patch-release.js",
-    "publish": "npm run clean:slate && lerna publish --force-publish=* -m \"Publish\"",
+    "publish": "jlpm run clean:slate && lerna publish --force-publish=* -m \"Publish\" && jlpm run build:update",
     "remove:dependency": "node buildutils/lib/remove-dependency.js",
     "remove:package": "node buildutils/lib/remove-package.js",
     "test": "cd test && jlpm test",
@@ -33,9 +36,7 @@
     "test:ie": "lerna run test:ie --concurrency 1 --stream",
     "test:services": "cd packages/services && jlpm test && jlpm run test:integration && cd examples/node && python main.py",
     "update:dependency": "node buildutils/lib/update-dependency.js",
-    "watch": "run-p watch:*",
-    "watch:main": "cd jupyterlab && jlpm run watch",
-    "watch:packages": "cd packages/metapackage && jlpm run watch"
+    "watch": "cd packages/metapackage && jlpm run watch"
   },
   "dependencies": {},
   "devDependencies": {
@@ -43,9 +44,7 @@
     "npm-run-all": "~4.1.1"
   },
   "workspaces": [
-    "jupyterlab",
-    "jupyterlab/tests/mock_packages/extension",
-    "jupyterlab/tests/mock_packages/mimeextension",
+    "dev_mode",
     "examples/*",
     "packages/*",
     "packages/services/examples/node",

packages/services/test/manager.py

@@ -0,0 +1,27 @@
+# -*- coding: utf-8 -*-
+from jupyter_client.kernelspec import KernelSpecManager
+
+
+class TestKernelSpecManager(KernelSpecManager):
+    """ A custom KernelSpecManager for testing.
+    """
+
+    def find_kernel_specs(self):
+        """ Returns a dict mapping kernel names to resource directories.
+        """
+        kspecs = super(TestKernelSpecManager, self).find_kernel_specs()
+
+        # add conda envs kernelspecs
+        kspecs.update({name: spec.resource_dir
+                       for name, spec
+                       in self._kspecs.items()})
+        return kspecs
+
+    def get_kernel_spec(self, kernel_name):
+        """ Returns a :class:`KernelSpec` instance for the given kernel_name.
+        """
+
+        return (
+            self._kspecs.get(kernel_name) or
+            super(TestKernelSpecManager, self).get_kernel_spec(kernel_name)
+        )

pytest.ini

@@ -1,4 +1,4 @@
 [pytest]
 testpaths=jupyterlab/tests
 norecursedirs=node_modules
-addopts = --pdbcls=IPython.terminal.debugger:Pdb --pdb
+addopts = --pdbcls=IPython.terminal.debugger:Pdb

scripts/ensure-buildutils.js

@@ -9,7 +9,9 @@ var childProcess = require('child_process');
 
 
 if (!fs.existsSync(path.join('buildutils', 'lib'))) {
-  childProcess.execSync('npm run build:utils',
-    { 'stdio': [0, 1, 2] }
+  // This must be "npm" because it is run during `pip install -e .` before
+  // jlpm is installed.
+  childProcess.execSync('npm run build',
+    { 'stdio': [0, 1, 2], cwd: path.resolve('./buildutils') }
   );
 }

scripts/travis_after_success.sh

@@ -3,7 +3,7 @@
 # Copyright (c) Jupyter Development Team.
 # Distributed under the terms of the Modified BSD License.
 
-if [[ $TRAVIS_PULL_REQUEST == false && $TRAVIS_BRANCH == "master" && $GROUP == "coverage_and_docs" ]]
+if [[ $TRAVIS_PULL_REQUEST == false && $TRAVIS_BRANCH == "master" && $GROUP == "docs" ]]
 then
     echo "-- pushing docs --"
 

scripts/travis_install.sh

@@ -4,21 +4,31 @@
 # Distributed under the terms of the Modified BSD License.
 set -x
 
-if [[ $GROUP == tests || $GROUP == other ]]; then
-    wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh;
-fi
+# The miniconda directory may exist if it has been restored from cache
+if [ -d "$MINICONDA_DIR" ] && [ -e "$MINICONDA_DIR/bin/conda" ]; then
+    echo "Miniconda install already present from cache: $MINICONDA_DIR"
+else # if it does not exist, we need to install miniconda
+    rm -rf "$MINICONDA_DIR" # remove the directory in case we have an empty cached directory
+    
+    if [[ $GROUP == py2 ]]; then
+        wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh -O ~/miniconda.sh;
+    else
+        wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh;
+    fi
 
-if [[ $GROUP == coverage ]]; then
-    wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh -O ~/miniconda.sh;
+    bash ~/miniconda.sh -b -p "$MINICONDA_DIR"
+    chown -R "$USER" "$MINICONDA_DIR"
+    hash -r
 fi
 
-bash ~/miniconda.sh -b -p $HOME/miniconda
-export PATH="$HOME/miniconda/bin:$PATH"
-hash -r
+export PATH="$MINICONDA_DIR/bin:$PATH"
 conda config --set always_yes yes --set changeps1 no
 conda update -q conda
-conda info -a
-conda install -c conda-forge notebook pytest
+conda info -a # for debugging
+
+conda remove --name test --all || true
+conda create -n test -c notebook pytest
+source activate test
 
 # create jupyter base dir (needed for config retrieval)
 mkdir ~/.jupyter
@@ -26,12 +36,7 @@ mkdir ~/.jupyter
 
 # Install and enable the server extension
 pip install -v -e ".[test]"
-# Make sure the schema and theme files exist
-test -e jupyterlab/schemas/jupyter.extensions.shortcuts.json
-test -e jupyterlab/themes/jupyterlab-theme-light-extension/images/jupyterlab.svg
 jlpm versions
 jlpm config current
 jlpm cache list
-jlpm install
-jlpm run build
 jupyter serverextension enable --py jupyterlab

scripts/travis_script.sh

@@ -7,62 +7,109 @@ set -ex
 export DISPLAY=:99.0
 sh -e /etc/init.d/xvfb start || true
 
-export PATH="$HOME/miniconda/bin:$PATH"
+export PATH="$MINICONDA_DIR/bin:$PATH"
+source activate test
 
-# Run integrity first we we see the message.
-npm run integrity
 
-# Build the packages.
-npm run build:packages
+pip install jupyterlab_launcher==0.6
 
 
-if [[ $GROUP == tests ]]; then
-
-    # Run the JS and python tests
+if [[ $GROUP == py2 || $GROUP == py3 ]]; then
+    # Run the python tests
     py.test -v
-    jlpm run build:test
-    jlpm test
-    jlpm run test:services || jlpm run test:services
-
 fi
 
 
-if [[ $GROUP == coverage ]]; then
+if [[ $GROUP == js ]]; then
 
-    # Make sure the examples build
-    jlpm run build:examples
+    jlpm build:packages
+    jlpm build:test
 
-    # Run the coverage and python tests.
-    py.test -v
+    # Allow the tests to fail once due to slow CI.
+    jlpm test || jlpm test
+    jlpm run test:services || jlpm run test:services
+    jlpm run clean
+fi
+
+
+if [[ $GROUP == js_cov ]]; then
+
+    jlpm run build:packages
     jlpm run build:test
-    jlpm run coverage
+
+    # Allow the tests to fail once due to slow CI.
+    jlpm run coverage || jlpm run coverage
+
+    # Run the services node example.
+    pushd packages/services/examples/node
+    python main.py
+    popd
+
     jlpm run clean
+fi
+
+
+if [[ $GROUP == docs ]]; then
+
+    # Run the link check - allow for a link to fail once
+    py.test --check-links -k .md . || py.test --check-links -k .md --lf .
+
+    # Build the api docs
+    jlpm run docs
 
+    # Verify tutorial docs build
+    pushd docs
+    conda remove --name test_docs --all || true
+    conda env create -n test_docs -f environment.yml
+    source activate test_docs
+    make html
+    source deactivate
+    popd
 fi
 
 
-if [[ $GROUP == other ]]; then
+if [[ $GROUP == integrity ]]; then
+    # Build the packages individually.
+    jlpm run build:src
 
-    # Build the core assets.
-    jlpm run build
+    # Make sure the examples build
+    jlpm run build:examples
+
+    # Make sure we have CSS that can be converted with postcss
+    jlpm global add postcss-cli
 
-    # Make sure we can successfully load the core app.
-    python -m jupyterlab.selenium_check --core-mode
+    jlpm config set prefix ~/.yarn
+    ~/.yarn/bin/postcss packages/**/style/*.css --dir /tmp
+
+    # Make sure we can successfully load the dev app.
+    python -m jupyterlab.selenium_check --dev-mode
 
     # Make sure we can run the built app.
     jupyter labextension install ./jupyterlab/tests/mock_packages/extension
+    pip install jupyterlab_launcher==0.5.5
     python -m jupyterlab.selenium_check
     jupyter labextension list
 
     # Make sure we can non-dev install.
+    conda remove --name test_install --all || true
     conda create -n test_install notebook python=3.5
     source activate test_install
-    pip install .
-    jupyter lab build
-    pip install selenium
+    pip install ".[test]"  # this populates <sys_prefix>/share/jupyter/lab
     python -m jupyterlab.selenium_check
+
+    # Make sure we can start and kill the lab server
+    jupyter lab --no-browser &
+    TASK_PID=$!
+    # Make sure the task is running
+    ps -p $TASK_PID || exit 1
+    sleep 5
+    kill $TASK_PID
+    wait $TASK_PID
     source deactivate
+fi 
+
 
+if [[ $GROUP == cli ]]; then
     # Test the cli apps.
     jupyter lab clean
     jupyter lab build
@@ -94,44 +141,18 @@ if [[ $GROUP == other ]]; then
     jupyter labextension enable -h
     jupyter labextension disable -h
 
-    # Make sure the examples build
-    jlpm run build:examples
-
-    # Run the services node example.
-    pushd packages/services/examples/node
-    python main.py
-    popd
-
-    # Run the link check - allow for a link to fail once
-    py.test --check-links -k .md . || py.test --check-links -k .md --lf .
-
-    # Build the api docs
-    jlpm run docs
-
-    # Verify tutorial docs build
-    pushd docs
-    conda env create -n test_docs -f environment.yml
-    source activate test_docs
-    make html
-    source deactivate
-    popd
-
-    # Make sure we have CSS that can be converted with postcss
-    npm install -g postcss-cli
-    postcss packages/**/style/*.css --dir /tmp
-
     # Make sure we can add and remove a sibling package.
-    npm run add:sibling jupyterlab/tests/mock_packages/extension
-    npm run build
-    npm run remove:package extension
-    npm run build
-    npm run integrity
+    jlpm run add:sibling jupyterlab/tests/mock_packages/extension
+    jlpm run build
+    jlpm run remove:package extension
+    jlpm run build
+    jlpm run integrity
 
     # Test cli tools
-    npm run get:dependency mocha
-    npm run update:dependency mocha
-    npm run remove:dependency mocha
-    npm run get:dependency @jupyterlab/buildutils
-    npm run get:dependency typescript
-    npm run get:dependency react-native 
+    jlpm run get:dependency mocha
+    jlpm run update:dependency mocha
+    jlpm run remove:dependency mocha
+    jlpm run get:dependency @jupyterlab/buildutils
+    jlpm run get:dependency typescript
+    jlpm run get:dependency react-native 
 fi

setup.py

@@ -3,82 +3,94 @@
 
 # Copyright (c) Jupyter Development Team.
 # Distributed under the terms of the Modified BSD License.
+import glob
+from os.path import join as pjoin
+import json
+import os
+import sys
 
-from __future__ import print_function
+# Our own imports
+from setupbase import (
+    create_cmdclass, ensure_python, find_packages, get_version,
+    command_for_func, combine_commands, install_npm, HERE, run
+)
 
-#-----------------------------------------------------------------------------
-# Minimal Python version sanity check
-#-----------------------------------------------------------------------------
+from setuptools import setup
 
-import sys
 
-v = sys.version_info
-if v[:2] < (2,7) or (v[0] >= 3 and v[:2] < (3,3)):
-    error = "ERROR: %s requires Python version 2.7 or 3.3 or above." % name
-    print(error, file=sys.stderr)
-    sys.exit(1)
+NAME = 'jupyterlab'
+DESCRIPTION = 'An alpha preview of the JupyterLab notebook server extension.'
+LONG_DESCRIPTION = """
+This is an alpha preview of JupyterLab. It is not ready for general usage yet.
+Development happens on https://github.com/jupyter/jupyterlab, with chat on
+https://gitter.im/jupyter/jupyterlab.
+"""
+
+ensure_python(['2.7', '>=3.3'])
+
+data_files_spec = [
+    ('share/jupyter/lab/static', ['%s/static/*' % NAME]),
+    ('share/jupyter/lab/schemas', ['%s/schemas/**' % NAME]),
+    ('share/jupyter/lab/themes', ['%s/themes/**' % NAME])
+]
 
-PY3 = (sys.version_info[0] >= 3)
+package_data_spec = dict()
+package_data_spec[NAME] = [
+    'staging/*', 'static/**', 'tests/mock_packages/**', 'themes/**',
+    'schemas/**'
+]
 
-#-----------------------------------------------------------------------------
-# get on with it
-#-----------------------------------------------------------------------------
+staging = pjoin(HERE, NAME, 'staging')
+npm = ['node', pjoin(staging, 'yarn.js')]
+VERSION = get_version('%s/_version.py' % NAME)
 
-from distutils import log
-from hashlib import sha256
-import json
-import os
-from glob import glob
 
-try:
-    from urllib2 import urlopen
-except ImportError:
-    from urllib.request import urlopen
+def check_assets():
+    from distutils.version import LooseVersion
 
+    # Representative files that should exist after a successful build
+    targets = [
+        'static/package.json',
+        'schemas/@jupyterlab/shortcuts-extension/plugin.json',
+        'themes/@jupyterlab/theme-light-extension/images/jupyterlab.svg'
+    ]
 
-# BEFORE importing distutils, remove MANIFEST. distutils doesn't properly
-# update it when the contents of directories change.
-if os.path.exists('MANIFEST'): os.remove('MANIFEST')
+    for t in targets:
+        if not os.path.exists(pjoin(HERE, NAME, t)):
+            msg = ('Missing file: %s, `build:prod` script did not complete '
+                   'successfully' % t)
+            raise ValueError(msg)
 
-from distutils.command.build_ext import build_ext
-from distutils.command.build_py import build_py
-from setuptools.command.sdist import sdist
-from setuptools import setup
-from setuptools.command.bdist_egg import bdist_egg
-from setuptools import setup
+    if 'develop' in sys.argv:
+        run(npm, cwd=HERE)
 
+    if 'sdist' not in sys.argv and 'bdist_wheel' not in sys.argv:
+        return
 
-# Our own imports
-from setupbase import (
-    bdist_egg_disabled,
-    find_packages,
-    find_package_data,
-    find_data_files,
-    js_prerelease,
-    CheckAssets,
-    version_ns,
-    name,
-    custom_egg_info
-)
+    target = pjoin(HERE, NAME, 'static', 'package.json')
+    with open(target) as fid:
+        version = json.load(fid)['jupyterlab']['version']
 
+    if LooseVersion(version) != LooseVersion(VERSION):
+        raise ValueError('Version mismatch, please run `build:update`')
 
-here = os.path.dirname(os.path.abspath(__file__))
-pjoin = os.path.join
 
-DESCRIPTION = 'An alpha preview of the JupyterLab notebook server extension.'
-LONG_DESCRIPTION = 'This is an alpha preview of JupyterLab. It is not ready for general usage yet. Development happens on https://github.com/jupyter/jupyterlab, with chat on https://gitter.im/jupyter/jupyterlab.'
+cmdclass = create_cmdclass('jsdeps', data_files_spec=data_files_spec,
+    package_data_spec=package_data_spec)
+cmdclass['jsdeps'] = combine_commands(
+    install_npm(build_cmd='build:prod', path=staging, source_dir=staging,
+                build_dir=pjoin(HERE, NAME, 'static'), npm=npm),
+    command_for_func(check_assets)
+)
 
 
 setup_args = dict(
-    name             = name,
+    name             = NAME,
     description      = DESCRIPTION,
     long_description = LONG_DESCRIPTION,
-    version          = version_ns['__version__'],
-    scripts          = glob(pjoin('scripts', '*')),
+    version          = VERSION,
     packages         = find_packages(),
-    package_data     = find_package_data(),
-    data_files       = find_data_files(),
-    include_package_data = True,
+    cmdclass         = cmdclass,
     author           = 'Jupyter Development Team',
     author_email     = 'jupyter@googlegroups.com',
     url              = 'http://jupyter.org',
@@ -94,41 +106,22 @@ setup_args = dict(
         'Programming Language :: Python',
         'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.3',
         'Programming Language :: Python :: 3.4',
         'Programming Language :: Python :: 3.5',
+        'Programming Language :: Python :: 3.6',
     ],
 )
 
 
-cmdclass = dict(
-    build_py = build_py,
-    build_ext = build_ext,
-    sdist  = js_prerelease(sdist, strict=True),
-    bdist_egg = bdist_egg if 'bdist_egg' in sys.argv else bdist_egg_disabled,
-    jsdeps = CheckAssets,
-    egg_info = custom_egg_info
-)
-try:
-    from wheel.bdist_wheel import bdist_wheel
-    cmdclass['bdist_wheel'] = js_prerelease(bdist_wheel, strict=True)
-except ImportError:
-    pass
-
-
-setup_args['cmdclass'] = cmdclass
-
-
-setuptools_args = {}
-install_requires = setuptools_args['install_requires'] = [
+setup_args['install_requires'] = [
     'notebook>=4.3.1',
-    'jupyterlab_launcher>=0.6.0',
+    'jupyterlab_launcher>=0.5.2,<0.6.0',
     'ipython_genutils',
     "futures;python_version<'3.0'",
     "subprocess32;python_version<'3.0'"
 ]
 
-extras_require = setuptools_args['extras_require'] = {
+setup_args['extras_require'] = {
     'test:python_version == "2.7"': ['mock'],
     'test': ['pytest', 'requests', 'pytest-check-links', 'selenium'],
     'docs': [
@@ -139,21 +132,20 @@ extras_require = setuptools_args['extras_require'] = {
 }
 
 
-if 'setuptools' in sys.modules:
-    setup_args.update(setuptools_args)
+# Because of this we do not need a MANIFEST.in
+setup_args['include_package_data'] = True
 
-    # force entrypoints with setuptools (needed for Windows, unconditional because of wheels)
-    setup_args['entry_points'] = {
-        'console_scripts': [
-            'jupyter-lab = jupyterlab.labapp:main',
-            'jupyter-labextension = jupyterlab.labextensions:main',
-            'jupyter-labhub = jupyterlab.labhubapp:main',
-            'jlpm = jupyterlab.jlpmapp:main',
-        ]
-    }
-    setup_args.pop('scripts', None)
+# Force entrypoints with setuptools (needed for Windows, unconditional
+# because of wheels)
+setup_args['entry_points'] = {
+    'console_scripts': [
+        'jupyter-lab = jupyterlab.labapp:main',
+        'jupyter-labextension = jupyterlab.labextensions:main',
+        'jupyter-labhub = jupyterlab.labhubapp:main',
+        'jlpm = jupyterlab.jlpmapp:main',
+    ]
+}
 
-    setup_args.update(setuptools_args)
 
 if __name__ == '__main__':
     setup(**setup_args)

setupbase.py

@@ -1,31 +1,42 @@
-# encoding: utf-8
-"""
-This module defines the things that are used in setup.py for building JupyterLab
-This includes:
-    * Functions for finding things like packages, package data, etc.
-    * A function for checking dependencies.
-"""
+#!/usr/bin/env python
+# coding: utf-8
 
 # Copyright (c) Jupyter Development Team.
 # Distributed under the terms of the Modified BSD License.
 
+"""
+This file originates from the 'jupyter-packaging' package, and
+contains a set of useful utilities for including npm packages
+within a Python package.
+"""
+from os.path import join as pjoin
 import io
-import json
 import os
+import functools
 import pipes
+import re
+import shlex
+import subprocess
 import sys
-import shutil
-import tempfile
-import os.path as osp
-from os.path import join as pjoin
 
-from distutils import log
+
+# BEFORE importing distutils, remove MANIFEST. distutils doesn't properly
+# update it when the contents of directories change.
+if os.path.exists('MANIFEST'): os.remove('MANIFEST')
+
+
 from distutils.cmd import Command
-from distutils.version import LooseVersion
-from setuptools.command.egg_info import egg_info
+from distutils.command.build_py import build_py
+from distutils.command.sdist import sdist
+from distutils import log
+
+from setuptools.command.develop import develop
 from setuptools.command.bdist_egg import bdist_egg
-from subprocess import check_call
 
+try:
+    from wheel.bdist_wheel import bdist_wheel
+except ImportError:
+    bdist_wheel = None
 
 if sys.platform == 'win32':
     from subprocess import list2cmdline
@@ -33,197 +44,617 @@ else:
     def list2cmdline(cmd_list):
         return ' '.join(map(pipes.quote, cmd_list))
 
-# the name of the project
-name = 'jupyterlab'
 
+__version__ = '0.2.0'
 
-here = osp.dirname(osp.abspath(__file__))
-is_repo = osp.exists(pjoin(here, '.git'))
+# ---------------------------------------------------------------------------
+# Top Level Variables
+# ---------------------------------------------------------------------------
 
-version_ns = {}
-with io.open(pjoin(here, name, '_version.py'), encoding="utf8") as f:
-    exec(f.read(), {}, version_ns)
+HERE = os.path.abspath(os.path.dirname(__file__))
+is_repo = os.path.exists(pjoin(HERE, '.git'))
+node_modules = pjoin(HERE, 'node_modules')
 
+SEPARATORS = os.sep if os.altsep is None else os.sep + os.altsep
+
+npm_path = ':'.join([
+    pjoin(HERE, 'node_modules', '.bin'),
+    os.environ.get('PATH', os.defpath),
+])
+
+if "--skip-npm" in sys.argv:
+    print("Skipping npm install as requested.")
+    skip_npm = True
+    sys.argv.remove("--skip-npm")
+else:
+    skip_npm = False
 
-def run(cmd, *args, **kwargs):
-    """Echo a command before running it"""
-    log.info('> ' + list2cmdline(cmd))
-    kwargs['shell'] = (sys.platform == 'win32')
-    return check_call(cmd, *args, **kwargs)
 
+# ---------------------------------------------------------------------------
+# Public Functions
+# ---------------------------------------------------------------------------
+
+def get_version(file, name='__version__'):
+    """Get the version of the package from the given file by
+    executing it and extracting the given `name`.
+    """
+    path = os.path.realpath(file)
+    version_ns = {}
+    with io.open(path, encoding="utf8") as f:
+        exec(f.read(), {}, version_ns)
+    return version_ns[name]
+
+
+def ensure_python(specs):
+    """Given a list of range specifiers for python, ensure compatibility.
+    """
+    if not isinstance(specs, (list, tuple)):
+        specs = [specs]
+    v = sys.version_info
+    part = '%s.%s' % (v.major, v.minor)
+    for spec in specs:
+        if part == spec:
+            return
+        try:
+            if eval(part + spec):
+                return
+        except SyntaxError:
+            pass
+    raise ValueError('Python version %s unsupported' % part)
 
-#---------------------------------------------------------------------------
-# Find packages
-#---------------------------------------------------------------------------
 
-def find_packages():
+def find_packages(top=HERE):
     """
     Find all of the packages.
     """
     packages = []
-    for dir, subdirs, files in os.walk('jupyterlab'):
-        if 'node_modules' in subdirs:
-            subdirs.remove('node_modules')
-        package = dir.replace(osp.sep, '.')
-        if '__init__.py' not in files:
-            # not a package
-            continue
-        packages.append(package)
+    for d, dirs, _ in os.walk(top, followlinks=True):
+        if os.path.exists(pjoin(d, '__init__.py')):
+            packages.append(os.path.relpath(d, top).replace(os.path.sep, '.'))
+        elif d != top:
+            # Do not look for packages in subfolders if current is not a package
+            dirs[:] = []
     return packages
 
 
-#---------------------------------------------------------------------------
-# Find package data
-#---------------------------------------------------------------------------
+def update_package_data(distribution):
+    """update build_py options to get package_data changes"""
+    build_py = distribution.get_command_obj('build_py')
+    build_py.finalize_options()
+
 
-def find_package_data():
+class bdist_egg_disabled(bdist_egg):
+    """Disabled version of bdist_egg
+
+    Prevents setup.py install performing setuptools' default easy_install,
+    which it should never ever do.
     """
-    Find package_data.
+    def run(self):
+        sys.exit("Aborting implicit building of eggs. Use `pip install .` "
+                 " to install from source.")
+
+
+def create_cmdclass(prerelease_cmd=None, package_data_spec=None,
+        data_files_spec=None):
+    """Create a command class with the given optional prerelease class.
+
+    Parameters
+    ----------
+    prerelease_cmd: (name, Command) tuple, optional
+        The command to run before releasing.
+    package_data_spec: dict, optional
+        A dictionary whose keys are the dotted package names and
+        whose values are a list of glob patterns.
+    data_files_spec: list, optional
+        A list of (path, patterns) tuples where the path is the
+        `data_files` install path and the patterns are glob patterns.
+
+    Notes
+    -----
+    We use specs so that we can find the files *after* the build
+    command has run.
+
+    The package data glob patterns should be relative paths from the package
+    folder containing the __init__.py file, which is given as the package
+    name.
+    e.g. `dict(foo=['./bar/*', './baz/**'])`
+
+    The data files glob patterns should be absolute paths or relative paths
+    from the root directory of the repository.
+    e.g. `('share/foo/bar', ['pkgname/bizz/*', 'pkgname/baz/**'])`
     """
-    theme_dirs = []
-    for dir, subdirs, files in os.walk(pjoin('jupyterlab', 'themes')):
-        slice_len = len('jupyterlab' + os.sep)
-        theme_dirs.append(pjoin(dir[slice_len:], '*'))
+    wrapped = [prerelease_cmd] if prerelease_cmd else []
+    if package_data_spec or data_files_spec:
+        wrapped.append('handle_files')
+    wrapper = functools.partial(_wrap_command, wrapped)
+    handle_files = _get_file_handler(package_data_spec, data_files_spec)
 
-    schema_dirs = []
-    for dir, subdirs, files in os.walk(pjoin('jupyterlab', 'schemas')):
-        slice_len = len('jupyterlab' + os.sep)
-        schema_dirs.append(pjoin(dir[slice_len:], '*'))
+    if 'bdist_egg' in sys.argv:
+        egg = wrapper(bdist_egg, strict=True)
+    else:
+        egg = bdist_egg_disabled
 
-    return {
-        'jupyterlab': ['build/*', '*.js', 'package.app.json',
-                       'yarn.lock', 'yarn.app.lock', '.yarnrc'
-                       ] + theme_dirs + schema_dirs
-    }
+    cmdclass = dict(
+        build_py=wrapper(build_py, strict=is_repo),
+        bdist_egg=egg,
+        sdist=wrapper(sdist, strict=True),
+        handle_files=handle_files,
+    )
 
+    if bdist_wheel:
+        cmdclass['bdist_wheel'] = wrapper(bdist_wheel, strict=True)
 
-def find_data_files():
-    """
-    Find data_files.
+    cmdclass['develop'] = wrapper(develop, strict=True)
+    return cmdclass
+
+
+def command_for_func(func):
+    """Create a command that calls the given function."""
+
+    class FuncCommand(BaseCommand):
+
+        def run(self):
+            func()
+            update_package_data(self.distribution)
+
+    return FuncCommand
+
+
+def run(cmd, **kwargs):
+    """Echo a command before running it.  Defaults to repo as cwd"""
+    log.info('> ' + list2cmdline(cmd))
+    kwargs.setdefault('cwd', HERE)
+    kwargs.setdefault('shell', os.name == 'nt')
+    if not isinstance(cmd, (list, tuple)) and os.name != 'nt':
+        cmd = shlex.split(cmd)
+    cmd[0] = which(cmd[0])
+    return subprocess.check_call(cmd, **kwargs)
+
+
+def is_stale(target, source):
+    """Test whether the target file/directory is stale based on the source
+       file/directory.
     """
-    if not os.path.exists(pjoin('jupyterlab', 'build')):
+    if not os.path.exists(target):
+        return True
+    target_mtime = recursive_mtime(target) or 0
+    return compare_recursive_mtime(source, cutoff=target_mtime)
+
+
+class BaseCommand(Command):
+    """Empty command because Command needs subclasses to override too much"""
+    user_options = []
+
+    def initialize_options(self):
+        pass
+
+    def finalize_options(self):
+        pass
+
+    def get_inputs(self):
+        return []
+
+    def get_outputs(self):
         return []
 
-    files = []
 
-    static_files = os.listdir(pjoin('jupyterlab', 'build'))
-    files.append(('share/jupyter/lab/static',
-        ['jupyterlab/build/%s' % f for f in static_files]))
+def combine_commands(*commands):
+    """Return a Command that combines several commands."""
 
-    for dir, subdirs, fnames in os.walk(pjoin('jupyterlab', 'schemas')):
-        dir = dir.replace(os.sep, '/')
-        schema_files = []
-        for fname in fnames:
-            schema_files.append('%s/%s' % (dir, fname))
-        slice_len = len('jupyterlab/')
-        files.append(('share/jupyter/lab/%s' % dir[slice_len:], schema_files))
+    class CombinedCommand(Command):
+        user_options = []
 
-    for dir, subdirs, fnames in os.walk(pjoin('jupyterlab', 'themes')):
-        dir = dir.replace(os.sep, '/')
-        themes_files = []
-        for fname in fnames:
-            themes_files.append('%s/%s' % (dir, fname))
-        slice_len = len('jupyterlab/')
-        files.append(('share/jupyter/lab/%s' % dir[slice_len:], themes_files))
+        def initialize_options(self):
+            self.commands = []
+            for C in commands:
+                self.commands.append(C(self.distribution))
+            for c in self.commands:
+                c.initialize_options()
 
-    return files
+        def finalize_options(self):
+            for c in self.commands:
+                c.finalize_options()
+
+        def run(self):
+            for c in self.commands:
+                c.run()
+    return CombinedCommand
 
 
-def js_prerelease(command, strict=False):
-    """decorator for building minified js/css prior to another command"""
-    class DecoratedCommand(command):
+def compare_recursive_mtime(path, cutoff, newest=True):
+    """Compare the newest/oldest mtime for all files in a directory.
+
+    Cutoff should be another mtime to be compared against. If an mtime that is
+    newer/older than the cutoff is found it will return True.
+    E.g. if newest=True, and a file in path is newer than the cutoff, it will
+    return True.
+    """
+    if os.path.isfile(path):
+        mt = mtime(path)
+        if newest:
+            if mt > cutoff:
+                return True
+        elif mt < cutoff:
+            return True
+    for dirname, _, filenames in os.walk(path, topdown=False):
+        for filename in filenames:
+            mt = mtime(pjoin(dirname, filename))
+            if newest:  # Put outside of loop?
+                if mt > cutoff:
+                    return True
+            elif mt < cutoff:
+                return True
+    return False
+
+
+def recursive_mtime(path, newest=True):
+    """Gets the newest/oldest mtime for all files in a directory."""
+    if os.path.isfile(path):
+        return mtime(path)
+    current_extreme = None
+    for dirname, dirnames, filenames in os.walk(path, topdown=False):
+        for filename in filenames:
+            mt = mtime(pjoin(dirname, filename))
+            if newest:  # Put outside of loop?
+                if mt >= (current_extreme or mt):
+                    current_extreme = mt
+            elif mt <= (current_extreme or mt):
+                current_extreme = mt
+    return current_extreme
+
+
+def mtime(path):
+    """shorthand for mtime"""
+    return os.stat(path).st_mtime
+
+
+def install_npm(path=None, build_dir=None, source_dir=None, build_cmd='build', force=False, npm=None):
+    """Return a Command for managing an npm installation.
+
+    Note: The command is skipped if the `--skip-npm` flag is used.
+
+    Parameters
+    ----------
+    path: str, optional
+        The base path of the node package.  Defaults to the repo root.
+    build_dir: str, optional
+        The target build directory.  If this and source_dir are given,
+        the JavaScript will only be build if necessary.
+    source_dir: str, optional
+        The source code directory.
+    build_cmd: str, optional
+        The npm command to build assets to the build_dir.
+    npm: str or list, optional.
+        The npm executable name, or a tuple of ['node', executable].
+    """
+
+    class NPM(BaseCommand):
+        description = 'install package.json dependencies using npm'
 
         def run(self):
-            jsdeps = self.distribution.get_command_obj('jsdeps')
-            if not is_repo and all(osp.exists(t) for t in jsdeps.targets):
-                # sdist, nothing to do
-                command.run(self)
+            if skip_npm:
+                log.info('Skipping npm-installation')
                 return
+            node_package = path or HERE
+            node_modules = pjoin(node_package, 'node_modules')
+            is_yarn = os.path.exists(pjoin(node_package, 'yarn.lock'))
+
+            npm_cmd = npm
 
-            try:
-                self.distribution.run_command('jsdeps')
-            except Exception as e:
-                missing = [t for t in jsdeps.targets if not osp.exists(t)]
-                if strict or missing:
-                    log.warn('js check failed')
-                    if missing:
-                        log.error('missing files: %s' % missing)
-                    raise e
+            if npm is None:
+                if is_yarn:
+                    npm_cmd = ['yarn']
                 else:
-                    log.warn('js check failed (not a problem)')
-                    log.warn(str(e))
-            command.run(self)
-    return DecoratedCommand
+                    npm_cmd = ['npm']
 
+            if not which(npm_cmd[0]):
+                log.error("`{0}` unavailable.  If you're running this command "
+                          "using sudo, make sure `{0}` is availble to sudo"
+                          .format(npm_cmd[0]))
+                return
 
-def update_package_data(distribution):
-    """update build_py options to get package_data changes"""
-    build_py = distribution.get_command_obj('build_py')
-    build_py.finalize_options()
+            if force or is_stale(node_modules, pjoin(node_package, 'package.json')):
+                log.info('Installing build dependencies with npm.  This may '
+                         'take a while...')
+                run(npm_cmd + ['install'], cwd=node_package)
+            if build_dir and source_dir and not force:
+                should_build = is_stale(build_dir, source_dir)
+            else:
+                should_build = True
+            if should_build:
+                run(npm_cmd + ['run', build_cmd], cwd=node_package)
 
+    return NPM
 
-class CheckAssets(Command):
-    description = 'check for required assets'
 
-    user_options = []
+def ensure_targets(targets):
+    """Return a Command that checks that certain files exist.
 
-    # Representative files that should exist after a successful build
-    targets = [
-        pjoin(here, 'jupyterlab', 'build', 'release_data.json'),
-        pjoin(here, 'jupyterlab', 'build', 'main.bundle.js'),
-        pjoin(here, 'jupyterlab', 'schemas', '@jupyterlab',
-            'shortcuts-extension', 'plugin.json'),
-        pjoin(here, 'jupyterlab', 'themes', '@jupyterlab',
-            'theme-light-extension',
-            'images', 'jupyterlab.svg')
-    ]
+    Raises a ValueError if any of the files are missing.
 
-    def initialize_options(self):
-        pass
+    Note: The check is skipped if the `--skip-npm` flag is used.
+    """
 
-    def finalize_options(self):
-        pass
+    class TargetsCheck(BaseCommand):
+        def run(self):
+            if skip_npm:
+                log.info('Skipping target checks')
+                return
+            missing = [t for t in targets if not os.path.exists(t)]
+            if missing:
+                raise ValueError(('missing files: %s' % missing))
 
-    def run(self):
-        for t in self.targets:
-            if not osp.exists(t):
-                msg = 'Missing file: %s' % t
-                raise ValueError(msg)
+    return TargetsCheck
 
-        target = pjoin(here, 'jupyterlab', 'build', 'release_data.json')
-        with open(target) as fid:
-            data = json.load(fid)
 
-        if (LooseVersion(data['version']) !=
-                LooseVersion(version_ns['__version__'])):
-            msg = 'Release assets version mismatch, please run npm publish'
-            raise ValueError(msg)
+# `shutils.which` function copied verbatim from the Python-3.3 source.
+def which(cmd, mode=os.F_OK | os.X_OK, path=None):
+    """Given a command, mode, and a PATH string, return the path which
+    conforms to the given mode on the PATH, or None if there is no such
+    file.
+    `mode` defaults to os.F_OK | os.X_OK. `path` defaults to the result
+    of os.environ.get("PATH"), or can be overridden with a custom search
+    path.
+    """
 
-        # update package data in case this created new files
-        update_package_data(self.distribution)
+    # Check that a given file can be accessed with the correct mode.
+    # Additionally check that `file` is not a directory, as on Windows
+    # directories pass the os.access check.
+    def _access_check(fn, mode):
+        return (os.path.exists(fn) and os.access(fn, mode) and
+                not os.path.isdir(fn))
+
+    # Short circuit. If we're given a full path which matches the mode
+    # and it exists, we're done here.
+    if _access_check(cmd, mode):
+        return cmd
+
+    path = (path or os.environ.get("PATH", os.defpath)).split(os.pathsep)
+
+    if sys.platform == "win32":
+        # The current directory takes precedence on Windows.
+        if os.curdir not in path:
+            path.insert(0, os.curdir)
+
+        # PATHEXT is necessary to check on Windows.
+        pathext = os.environ.get("PATHEXT", "").split(os.pathsep)
+        # See if the given file matches any of the expected path extensions.
+        # This will allow us to short circuit when given "python.exe".
+        matches = [cmd for ext in pathext if cmd.lower().endswith(ext.lower())]
+        # If it does match, only test that one, otherwise we have to try
+        # others.
+        files = [cmd] if matches else [cmd + ext.lower() for ext in pathext]
+    else:
+        # On other platforms you don't have things like PATHEXT to tell you
+        # what file suffixes are executable, so just pass on cmd as-is.
+        files = [cmd]
+
+    seen = set()
+    for dir in path:
+        dir = os.path.normcase(dir)
+        if dir not in seen:
+            seen.add(dir)
+            for thefile in files:
+                name = os.path.join(dir, thefile)
+                if _access_check(name, mode):
+                    return name
+    return None
+
+
+# ---------------------------------------------------------------------------
+# Private Functions
+# ---------------------------------------------------------------------------
+
+
+def _wrap_command(cmds, cls, strict=True):
+    """Wrap a setup command
+
+    Parameters
+    ----------
+    cmds: list(str)
+        The names of the other commands to run prior to the command.
+    strict: boolean, optional
+        Wether to raise errors when a pre-command fails.
+    """
+    class WrappedCommand(cls):
 
+        def run(self):
+            if not getattr(self, 'uninstall', None):
+                try:
+                    [self.run_command(cmd) for cmd in cmds]
+                except Exception:
+                    if strict:
+                        raise
+                    else:
+                        pass
+            # update package data
+            update_package_data(self.distribution)
+
+            result = cls.run(self)
+            return result
+    return WrappedCommand
+
+
+def _get_file_handler(package_data_spec, data_files_spec):
+    """Get a package_data and data_files handler command.
+    """
+    class FileHandler(BaseCommand):
 
-class bdist_egg_disabled(bdist_egg):
-    """Disabled version of bdist_egg
-    Prevents setup.py install performing setuptools' default easy_install,
-    which it should never ever do.
+        def run(self):
+            package_data = self.distribution.package_data
+            package_spec = package_data_spec or dict()
+            data_spec = data_files_spec or []
+
+            for (key, patterns) in package_spec.items():
+                package_data[key] = _get_package_data(key, patterns)
+
+            data_files = self.distribution.data_files or []
+            for (path, patterns) in data_spec:
+                data_files.append((path, _get_files(patterns)))
+
+            self.distribution.data_files = data_files
+
+    return FileHandler
+
+
+def _get_files(file_patterns, top=HERE):
+    """Expand file patterns to a list of paths.
+
+    Parameters
+    -----------
+    file_patterns: list or str
+        A list of glob patterns for the data file locations.
+        The globs can be recursive if they include a `**`.
+        They should be relative paths from the top directory or
+        absolute paths.
+    top: str
+        the directory to consider for data files
+
+    Note:
+    Files in `node_modules` are ignored.
     """
-    def run(self):
-        sys.exit("Aborting implicit building of eggs. Use `pip install .` to install from source.")
+    if not isinstance(file_patterns, (list, tuple)):
+        file_patterns = [file_patterns]
+
+    for i, p in enumerate(file_patterns):
+        if os.path.isabs(p):
+            file_patterns[i] = os.path.relpath(p, top)
 
+    matchers = [_compile_pattern(p) for p in file_patterns]
 
-class custom_egg_info(egg_info):
-    """Prune JavaScript folders from egg_info to avoid locking up pip.
+    files = set()
+
+    for root, dirnames, filenames in os.walk(top):
+        # Don't recurse into node_modules
+        if 'node_modules' in dirnames:
+            dirnames.remove('node_modules')
+        for m in matchers:
+            for filename in filenames:
+                fn = os.path.relpath(pjoin(root, filename), top)
+                if m(fn):
+                    files.add(fn.replace(os.sep, '/'))
+
+    return list(files)
+
+
+def _get_package_data(root, file_patterns=None):
+    """Expand file patterns to a list of `package_data` paths.
+
+    Parameters
+    -----------
+    root: str
+        The relative path to the package root from `HERE`.
+    file_patterns: list or str, optional
+        A list of glob patterns for the data file locations.
+        The globs can be recursive if they include a `**`.
+        They should be relative paths from the root or
+        absolute paths.  If not given, all files will be used.
+
+    Note:
+    Files in `node_modules` are ignored.
     """
+    if file_patterns is None:
+        file_patterns = ['*']
+    return _get_files(file_patterns, pjoin(HERE, root))
 
-    def run(self):
-        folders = ['examples', 'packages', 'test', 'node_modules']
-        folders = [f for f in folders if os.path.exists(pjoin(here, f))]
-        tempdir = tempfile.mkdtemp()
-        for folder in folders:
-            shutil.move(pjoin(here, folder), tempdir)
-        value = egg_info.run(self)
-        for folder in folders:
-            shutil.move(pjoin(tempdir, folder), here)
-        shutil.rmtree(tempdir)
-        return value
+
+def _compile_pattern(pat, ignore_case=True):
+    """Translate and compile a glob pattern to a regular expression matcher."""
+    if isinstance(pat, bytes):
+        pat_str = pat.decode('ISO-8859-1')
+        res_str = _translate_glob(pat_str)
+        res = res_str.encode('ISO-8859-1')
+    else:
+        res = _translate_glob(pat)
+    flags = re.IGNORECASE if ignore_case else 0
+    return re.compile(res, flags=flags).match
+
+
+def _iexplode_path(path):
+    """Iterate over all the parts of a path.
+
+    Splits path recursively with os.path.split().
+    """
+    (head, tail) = os.path.split(path)
+    if not head or (not tail and head == path):
+        if head:
+            yield head
+        if tail or not head:
+            yield tail
+        return
+    for p in _iexplode_path(head):
+        yield p
+    yield tail
+
+
+def _translate_glob(pat):
+    """Translate a glob PATTERN to a regular expression."""
+    translated_parts = []
+    for part in _iexplode_path(pat):
+        translated_parts.append(_translate_glob_part(part))
+    os_sep_class = '[%s]' % re.escape(SEPARATORS)
+    res = _join_translated(translated_parts, os_sep_class)
+    return '{res}\\Z(?ms)'.format(res=res)
+
+
+def _join_translated(translated_parts, os_sep_class):
+    """Join translated glob pattern parts.
+
+    This is different from a simple join, as care need to be taken
+    to allow ** to match ZERO or more directories.
+    """
+    res = ''
+    for part in translated_parts[:-1]:
+        if part == '.*':
+            # drop separator, since it is optional
+            # (** matches ZERO or more dirs)
+            res += part
+        else:
+            res += part + os_sep_class
+
+    if translated_parts[-1] == '.*':
+        # Final part is **
+        res += '.+'
+        # Follow stdlib/git convention of matching all sub files/directories:
+        res += '({os_sep_class}?.*)?'.format(os_sep_class=os_sep_class)
+    else:
+        res += translated_parts[-1]
+    return res
+
+
+def _translate_glob_part(pat):
+    """Translate a glob PATTERN PART to a regular expression."""
+    # Code modified from Python 3 standard lib fnmatch:
+    if pat == '**':
+        return '.*'
+    i, n = 0, len(pat)
+    res = []
+    while i < n:
+        c = pat[i]
+        i = i+1
+        if c == '*':
+            # Match anything but path separators:
+            res.append('[^%s]*' % SEPARATORS)
+        elif c == '?':
+            res.append('[^%s]?' % SEPARATORS)
+        elif c == '[':
+            j = i
+            if j < n and pat[j] == '!':
+                j = j+1
+            if j < n and pat[j] == ']':
+                j = j+1
+            while j < n and pat[j] != ']':
+                j = j+1
+            if j >= n:
+                res.append('\\[')
+            else:
+                stuff = pat[i:j].replace('\\', '\\\\')
+                i = j+1
+                if stuff[0] == '!':
+                    stuff = '^' + stuff[1:]
+                elif stuff[0] == '^':
+                    stuff = '\\' + stuff
+                res.append('[%s]' % stuff)
+        else:
+            res.append(re.escape(c))
+    return ''.join(res)

test/karma-cov.conf.js

@@ -29,6 +29,8 @@ module.exports = function (config) {
       html: 'coverage/html'
     },
     browserNoActivityTimeout: 31000, // 31 seconds - upped from 10 seconds
+    browserDisconnectTimeout: 31000, // 31 seconds - upped from 2 seconds
+    browserDisconnectTolerance: 2,
     port: 9876,
     colors: true,
     singleRun: true,

test/karma.conf.js

@@ -18,6 +18,8 @@ module.exports = function (config) {
       'build/bundle.js': ['sourcemap']
     },
     browserNoActivityTimeout: 31000, // 31 seconds - upped from 10 seconds
+    browserDisconnectTimeout: 31000, // 31 seconds - upped from 2 seconds
+    browserDisconnectTolerance: 2,
     port: 9876,
     colors: true,
     singleRun: true,