Sen descrición

Leo bfce422a2f Merge branch 'yili' of http://gogsb.soaringnova.com/ylproj/jupyterlab into yili		%!s(int64=2) %!d(string=hai) anos
builder	2be6f2eb9f [ci skip] New version	%!s(int64=2) %!d(string=hai) anos
buildutils	9e7f9768d3 merge: 合并	%!s(int64=2) %!d(string=hai) anos
dag @ ea8995b3b8	defcf70435 添加DAG仓库	%!s(int64=2) %!d(string=hai) anos
design	d7a9358062 Fix a few more typos	%!s(int64=4) %!d(string=hai) anos
dev_mode	9d18741fad dagEditor插件开发	%!s(int64=2) %!d(string=hai) anos
docker	6783f3c49f update	%!s(int64=2) %!d(string=hai) anos
docker_build	195a298012 完善jupyterhub镜像构建	%!s(int64=2) %!d(string=hai) anos
docker_jupyterhub	35a1e989ce 调整JupyterHub依赖	%!s(int64=2) %!d(string=hai) anos
docs	f0a6c4f6a7 Backport PR #12635: Add more explanation for internationalization (translation python package) (#12648)	%!s(int64=2) %!d(string=hai) anos
galata	9e7f9768d3 merge: 合并	%!s(int64=2) %!d(string=hai) anos
images	e8d230f5fc 更新README	%!s(int64=2) %!d(string=hai) anos
jupyter-config	678646cec1 enable jupyterlab to work as a classic notebook extension and fix handling of binder	%!s(int64=4) %!d(string=hai) anos
jupyterlab	d325463926 集成Dag图组件	%!s(int64=2) %!d(string=hai) anos
packages	bfce422a2f Merge branch 'yili' of http://gogsb.soaringnova.com/ylproj/jupyterlab into yili	%!s(int64=2) %!d(string=hai) anos
scripts	9e7f9768d3 merge: 合并	%!s(int64=2) %!d(string=hai) anos
sparkmagic @ b52dfe740e	6b3c823509 更新submodule	%!s(int64=2) %!d(string=hai) anos
testutils	2be6f2eb9f [ci skip] New version	%!s(int64=2) %!d(string=hai) anos
typedoc-theme	c3048d1dd1 Replaced ... with ellipses unicode character in .ts files (#10208)	%!s(int64=3) %!d(string=hai) anos
.bumpversion.cfg	af732d8125 [ci skip] Publish 3.4.3	%!s(int64=2) %!d(string=hai) anos
.cleanignore	2de04f2c37 implemented fallback arg for LabIcon.resolveX methods	%!s(int64=4) %!d(string=hai) anos
.dockerignore	18f953fe43 Backport PR #10796 to 3.2.x: Add Galata in JupyterLab (#11179)	%!s(int64=3) %!d(string=hai) anos
.eslintignore	18f953fe43 Backport PR #10796 to 3.2.x: Add Galata in JupyterLab (#11179)	%!s(int64=3) %!d(string=hai) anos
.eslintrc.js	e6fbbe2e6f 添加yili的新UI	%!s(int64=2) %!d(string=hai) anos
.git-blame-ignore-revs	1d76e3c443 [3.3.x] Add git-blame-ignore-revs file (#12289)	%!s(int64=2) %!d(string=hai) anos
.gitattributes	2a3bc70c04 Upgrade to yarn 1.5.1	%!s(int64=6) %!d(string=hai) anos
.gitignore	8630063321 Backport PR #10299: Add debugger variable renderer based on mime type (#11871)	%!s(int64=2) %!d(string=hai) anos
.gitmodules	defcf70435 添加DAG仓库	%!s(int64=2) %!d(string=hai) anos
.gitpod.yml	757a1c3a38 dont open in browser	%!s(int64=3) %!d(string=hai) anos
.meeseeksdev.yml	f2afcc5aad Add meessekdev bot config file	%!s(int64=6) %!d(string=hai) anos
.pre-commit-config.yaml	6b4a77a893 去除pre-commit	%!s(int64=2) %!d(string=hai) anos
.prettierignore	18f953fe43 Backport PR #10796 to 3.2.x: Add Galata in JupyterLab (#11179)	%!s(int64=3) %!d(string=hai) anos
.prettierrc	4ee83bd2b2 Lint - updates from the new version of prettier.	%!s(int64=4) %!d(string=hai) anos
.yarnrc	b513457b01 Update lerna (#5262)	%!s(int64=6) %!d(string=hai) anos
Dockerfile	195a298012 完善jupyterhub镜像构建	%!s(int64=2) %!d(string=hai) anos
Dockerfile.combined	7148294ea6 添加dag构建命令到Dockerfile中	%!s(int64=2) %!d(string=hai) anos
LICENSE	a272515aee 添加Docker镜像构建	%!s(int64=2) %!d(string=hai) anos
MANIFEST.in	c12bea95c1 Backport PR #11388: Set default ui font to `system-ui` (#11927)	%!s(int64=2) %!d(string=hai) anos
README.md	e8d230f5fc 更新README	%!s(int64=2) %!d(string=hai) anos
Untitled.ipynb	5f42320b9f fix: 更新yili-dag依赖	%!s(int64=2) %!d(string=hai) anos
about_dev.md	bc4b9ae7ce 更新文档	%!s(int64=2) %!d(string=hai) anos
about_diff.md	bc4b9ae7ce 更新文档	%!s(int64=2) %!d(string=hai) anos
bdist_wheel.sh	9e7f9768d3 merge: 合并	%!s(int64=2) %!d(string=hai) anos
clean.py	cde0521528 [3.3.x] Run black (#12282)	%!s(int64=2) %!d(string=hai) anos
conftest.py	cde0521528 [3.3.x] Run black (#12282)	%!s(int64=2) %!d(string=hai) anos
lerna.json	b513457b01 Update lerna (#5262)	%!s(int64=6) %!d(string=hai) anos
package.json	a272515aee 添加Docker镜像构建	%!s(int64=2) %!d(string=hai) anos
postcss.config.js	c617b40ce5 Backport PR #11552: postcss 8.4.0 breaks integrity 2 CI test (#11560)	%!s(int64=3) %!d(string=hai) anos
pyproject.toml	7746b1f158 完善构建流程	%!s(int64=2) %!d(string=hai) anos
setup.cfg	eaa0de0a68 Backport PR #12250: Open terminal in cwd from launcher (#12430)	%!s(int64=2) %!d(string=hai) anos
setup.py	7746b1f158 完善构建流程	%!s(int64=2) %!d(string=hai) anos
tsconfig.eslint.json	18f953fe43 Backport PR #10796 to 3.2.x: Add Galata in JupyterLab (#11179)	%!s(int64=3) %!d(string=hai) anos
tsconfigbase.json	df0b0f7438 Turn on strict null checks (except for services)	%!s(int64=4) %!d(string=hai) anos
tsconfigbase.test.json	cdb5b99a88 Convert jest imports to typescript type references.	%!s(int64=4) %!d(string=hai) anos
tsconfigdoc.json	5f42320b9f fix: 更新yili-dag依赖	%!s(int64=2) %!d(string=hai) anos
typedoc.js	18f953fe43 Backport PR #10796 to 3.2.x: Add Galata in JupyterLab (#11179)	%!s(int64=3) %!d(string=hai) anos
untitled.dag	5be394cfed feat: 更新脚本编辑codemirror版本为4	%!s(int64=2) %!d(string=hai) anos
yarn.lock	5f42320b9f fix: 更新yili-dag依赖	%!s(int64=2) %!d(string=hai) anos

JupyterLab

JupyterLab 是 Jupyter 项目提供的强大的 Notebook 编程界面, 可以通过其插件系统进行扩展. 本项目 fork 自 https://github.com/jupyterlab/jupyterlab, 在官方代码的基础上进行二次开发, 通过 Sparkmagic 连接 Livy 提供 Spark 编程及 HDFS 访问, 并提供数据资产管理, 可视化编程, 定时任务等功能.

构建及运行方法

项目提供了构建 Docker 镜像的 Dockerfile

# 使用 recursive 克隆 sparkmagic
git clone --recursive http://gogsb.soaringnova.com/ylproj/jupyterlab.git

# 构建镜像
docker build -f Dockerfile.combined -t ylproj-jupyter .

使用如下命令运行构建好的镜像:

docker run -d \
       --name ylproj-jupyter \
       --network ylproj \
       -p 8000:8000 \
       -v <kerberos配置>:/etc/krb5.conf:ro \
       -v <hadoop配置>:/opt/cluster/conf:ro \
       -e LIVY_SERVER_URL=http://livy:8998/ \
       -e BACKEND_BASE_URL=http://backend:5000/ \
       ylproj-jupyter

运行参数说明:

用户访问端口 8000 的 web 服务 (http://localhost:8000/jupyterhub)
环境变量 LIVY_SERVER_URL 指向 Livy 服务的地址
环境变量 BACKEND_BASE_URL 指向 Backend 服务的地址
/etc/krb5.conf 是 Kerberos 的配置文件
/opt/cluster/conf 是 Hadoop 相关的配置文件以及 Kerberos 的 keytab

各组件端口说明

service	port
JupyterHub Web	8000
Backend REST	5000
Livy REST	8998
Livy RPC	31100-31103
Livy RPC (Yarn)	31100-31103
HDFS	4007, 4008
ResourceManager	5000, 5004
NodeManager	5006, 5008
KDC	88
Hive Metastore (thrift)	7004
Hive Server2	7001

README.md

JupyterLab

构建及运行方法

各组件端口说明

其他