Alnoda workspaces

Added by: bluxmit
May 24, 2023

Version

~> sparkmagic

Description

Jupyter magics and kernels for working with remote Spark clusters via Livy/Lighter

Install (execute in workspace terminal)

wrk install sparkmagic==0.21.0.pipenv

Remarks and hints

Edit `~/.sparkmagic/config.json` with your Livy endpoint

If you want local spark running directly in the workspace:
wrk install apache-spark

PAY ATTENTION TO JUPYTERLAB VERSION COMPATIBILITY!

this is sparkmagic fo Jupyterlab versionn 4.*.*.pipenv, installed in isolated environment with pipenv in /home/abc/apps/jupyterlab.

Scripts

# Istall script

cd /tmp
wget https://github.com/KSXGitHub/parallel-disk-usage/releases/download/0.9.0/pdu-x86_64-unknown-linux-musl
chmod +x pdu-x86_64-unknown-linux-musl
mv pdu-x86_64-unknown-linux-musl /home/abc/bin/pdu

wrk install nodejs==18.15.0
export PATH="/home/abc/apps/nodejs/bin:$PATH"

export DEBIAN_FRONTEND=noninteractive 
sudo -E apt-get install krb5-config libkrb5-dev -y

cd /home/abc/apps/jupyterlab && pipenv run pip install sparkmagic==0.21.0
cd /home/abc/apps/jupyterlab && pipenv run jupyter labextension install "@jupyter-widgets/jupyterlab-manager"
cd /home/abc/apps/jupyterlab && pipenv run jupyter nbextension enable --py --sys-prefix widgetsnbextension

env_dir=$(cd /home/abc/apps/jupyterlab && pipenv run pip show sparkmagic | grep Location | cut -d" " -f2)
cd /home/abc/apps/jupyterlab && pipenv run jupyter-kernelspec install --user $env_dir/sparkmagic/kernels/sparkkernel
cd /home/abc/apps/jupyterlab && pipenv run jupyter-kernelspec install --user $env_dir/sparkmagic/kernels/pysparkkernel
cd /home/abc/apps/jupyterlab && pipenv run jupyter-kernelspec install --user $env_dir/sparkmagic/kernels/sparkrkernel

cd /home/abc/apps/jupyterlab && pipenv run jupyter serverextension enable --py sparkmagic

mkdir -p /home/abc/.sparkmagic/
touch /home/abc/.sparkmagic/config.json
cat >/home/abc/.sparkmagic/config.json <<EOL
{
  "kernel_python_credentials" : {
    "username": "",
    "password": "",
    "url": "http://localhost:8998",
    "auth": "None"
  },

"kernel_scala_credentials" : {
    "username": "",
    "password": "",
    "url": "http://localhost:8998",
    "auth": "None"
  },
  "kernel_r_credentials": {
    "username": "",
    "password": "",
    "url": "http://localhost:8998"
  },

"logging_config": {
    "version": 1,
    "formatters": {
      "magicsFormatter": { 
        "format": "%(asctime)s\t%(levelname)s\t%(message)s",
        "datefmt": ""
      }
    },
    "handlers": {
      "magicsHandler": { 
        "class": "hdijupyterutils.filehandler.MagicsFileHandler",
        "formatter": "magicsFormatter",
        "home_path": "~/.sparkmagic"
      }
    },
    "loggers": {
      "magicsLogger": { 
        "handlers": ["magicsHandler"],
        "level": "DEBUG",
        "propagate": 0
      }
    }
  },
  "authenticators": {
    "Kerberos": "sparkmagic.auth.kerberos.Kerberos",
    "None": "sparkmagic.auth.customauth.Authenticator", 
    "Basic_Access": "sparkmagic.auth.basic.Basic"
  },

"wait_for_idle_timeout_seconds": 15,
  "livy_session_startup_timeout_seconds": 60,

"fatal_error_suggestion": "The code failed because of a fatal error:\n\t{}.\n\nSome things to try:\na) Make sure Spark has enough available resources for Jupyter to create a Spark context.\nb) Contact your Jupyter administrator to make sure the Spark magics library is configured correctly.\nc) Restart the kernel.",

"ignore_ssl_errors": false,

"session_configs": {
    "driverMemory": "1000M",
    "executorCores": 2
  },

"use_auto_viz": true,
  "coerce_dataframe": true,
  "max_results_sql": 2500,
  "pyspark_dataframe_encoding": "utf-8",
  
  "heartbeat_refresh_seconds": 30,
  "livy_server_heartbeat_timeout_seconds": 0,
  "heartbeat_retry_seconds": 10,

"server_extension_default_kernel_name": "pysparkkernel",
  "custom_headers": {},
  
  "retry_policy": "configurable",
  "retry_seconds_to_sleep_list": [0.2, 0.5, 1, 3, 5],
  "configurable_retry_policy_max_retries": 8
}
EOL

# Start script

compatible with all workspaces

no compatibility requirements

~> App

~> sparkmagic

Description

Tags

Links

Install (execute in workspace terminal)

Remarks and hints

Scripts

!= Workspace compatibility

!= App compatibility