Background

There appears to be lots of screen recordings that occupy gigabytes of storage space locally on an MBP, which need to be compressed. While the local tasks running on an MBP may cause the heat dissipating fan to roaring, making use of an idle desktop seems a viable solution to accelerate the workflow.

Considerations Before Starting

• Q: Why not refer to powerful cloud-top computing resources such as those on AWS/GCP/Azure?
• A: Extra costs will apply, yet the deploying, debugging and tuning operations can’t be skipped still. On the contrary, a personal desktop as the server host is likely more cost-effective and easily maintainable for personal use cases versus in production.
• Q: Why not use a U-disk to transfer the files from the MBP to the desktop device?
• A: The source files are gigabytes of videos that can’t fit into the U-disk given the capacity limit.
• Q: Why not establish a simple web service on the host over HTTP for a user-friendly interface?
• A: Those web UI features (see also Docker API documentation) are more than necessary to achieve the goal. In the meantime, exposing the port 2375 (reference) over HTTP is not secured, yet the HTTPS/TLS setup via port 2376 introduced lots of complexity even when following the Docker’s documentation.

Quick Start

Counter-intuitively, there’s actually no need to install or run Docker Desktop app on either the client or server side, as long as the Docker daemon is ready on the server host and Docker CLI is present on the client side, saving lots of confusion, time and effort on the unnecessary desktop app settings.

Either Docker daemon and Docker CLI can be installed on the common *nix OS or distros including MacOS. If on Windows, Windows Subsystem Linux (WSL) will be helpful to prepare the runtime environment without the need to reinstall the entire OS or run a virtual machine, by enabling the certain optional feature instead.

Even if Docker Desktop is installed on Windows, the runtime environment has to be held in WSL anyway.

Server-side setup in Linux/WSL distro of Ubuntu (as an example)

• Package sources

  Common practice to refresh the list of available packages and their versions:

  1	sudo apt update

  Additionally, set up Docker’s apt repository using Docker’s official GPG key and Apt sources (recommended; see also Install Docker Engine on Ubuntu).

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

  # Add Docker's official GPG key: sudo apt update sudo apt install ca-certificates curl sudo install -m 0755 -d /etc/apt/keyrings sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc sudo chmod a+r /etc/apt/keyrings/docker.asc # Add the repository to Apt sources: sudo tee /etc/apt/sources.list.d/docker.sources <<EOF Types: deb URIs: https://download.docker.com/linux/ubuntu Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") Components: stable Signed-By: /etc/apt/keyrings/docker.asc EOF sudo apt update

• Where is ifconfig?

  After updating, install the net-tools package, which contains ifconfig, using this command:

  1	sudo apt install net-tools

• OpenSSH server

  1	sudo apt install openssh-server ssh-askpass ufw

  Enable, Start and check SSH Service:

  1 2 3

  sudo systemctl enable ssh sudo systemctl start ssh sudo systemctl status ssh

  Adjust Firewall (using UFW):

  1	sudo ufw allow ssh

• Docker engine, daemon and container management (latest version)

  1	sudo apt install docker-ce docker-ce-cli containerd.io

  The Docker service starts automatically after installation. To verify that Docker is running, use:

  1	sudo systemctl status docker

  Verify that the installation is successful by running the hello-world image (expecting some human-readable self-explanatory outputs):

  1	sudo docker run hello-world

• More usages about Docker daemon (read more)

  Test Docker daemon connection in a debug mode:

  1 2 3 4 5

  dockerd --debug [--tls=true \] [--tlscert=/var/docker/server.pem \] [--tlskey=/var/docker/serverkey.pem \] [--host tcp://192.168.59.3:2376]

  By default the daemon stores data in:

  • /var/lib/docker on Linux
  • C:\ProgramData\docker on Windows

Cliend-side setup (reference)

Docker CLI needs to be installed on the client side as well, same as above.

Create a Docker context to specify a non-local Docker host:

1
2
3
4

docker context create \
    --docker host=ssh://docker-user@host1.example.com \
    --description="Remote engine" \
    my-remote-engine

The context doesn’t take effect immediately upon creation. The next step is to inform the Docker CLI to use it, and to connect to the remote engine:

1

docker context use my-remote-engine

Quickly test if the context works out:

1

docker info

Details

Additional SSH usages

• Executing a simple command:

1

ssh user@example.com "ls -l /home/user"

Windows environment setup (skip this if not using Windows on a host)

Before digging deeper, beware of the difference between the concepts of WSL and WSL distros (read more).

What’s more, the current versions of WSL are 2.x, also referred to as WSL2. And the WSL command line tool will run in 2.x version by default.

WSL Distros

• List all the distro versions ready to install

  1	wsl --list --online

• List the local distros installed

  1	wsl --list [--verbose]

• Install a specific distro (e.g. Ubuntu)

  1	wsl --install ubuntu

• Start a WSL distro instance - simply run wsl in the built-in Command Prompt, Terminal App or Powershell.

• Check for the private IP address assigned to the Distro, which is helpful for the upcoming network configs

  1	wsl hostname -I

• Stop a running WSL distro only without removing its filesystem

  As an example, it will make the changes to the .wslconfig file take effect:

  1	wsl --shutdown

• Remove a WSL distro instance with its filesystem

  1	wsl --unregister ubuntu

• Export a WSL distro to a file

  It helps to create a backup (as .tar file by default) or relocate the distro to a different directory (shutdown recommended as above):

  1	wsl --export <DistributionName> <Full path to .tar/.vhdx FileName> [--vhd]

• Import a file as a WSL distro

  This allows to create a distro filesystem in a different directory other than before:

  1	wsl --import <NewDistributionName> <InstallLocation> <Full path to .tar/.vhdx FileName>

Network Configs on a Windows host

• Allow for incoming ping requests

  Add this rule to the firewall via Powershell with administrator permissions to resolve the timeout error of ping:

  1	New-NetFirewallRule -Name "Ping" -Protocol ICMPv4 -Direction Inbound -Action Allow

• Allow for incoming SSH requests

  This is to enable the built-in firewall rule for the OpenSSH Server:

  1	Enable-NetFirewallRule -DisplayGroup "OpenSSH Server"

  Alternatively, manually create a new inbound firewall rule:

  1	New-NetFirewallRule -DisplayName "Allow SSH Inbound" -Direction Inbound -Protocol TCP -LocalPort 22 -Action Allow -Profile Any

• Set up Port Forwarding on Windows host

  This is to add a port proxy rule to forward a local port 22 on Windows host to the WSL SSH server port 22 in WSL:

  1	netsh interface portproxy add v4tov4 listenport=22 listenaddress=0.0.0.0 connectport=22 connectaddress=<wsl-ip>

• Test connecting to WSL server via SSH from another device

  1	ssh username@host_ip [-p 22]

• Alternative: SSH Tunnels from Windows to a Remote Host via WSL (SSH Tunneling)

  This option hasn’t been tested and verified yet:

  1	ssh -L <local-port>:localhost:<remote-port> <remote-host>

Miscallaneous

Known Issues of WSL

• Why does WSL end, and the established SSH connection get closed, after a while of being idle?

  The cause lies in the WSL Idle Timeout setting. By default, WSL 2 goes into a suspended state (after a short period) once all open terminal windows and running processes have exited. The best way to prevent this is to configure the vmIdleTimeout setting in a Windows configuration file.

  Navigate to the Windows user profile directory: %userprofile% (e.g., C:\Users\YourUsername).
  Create a file named .wslconfig if it doesn’t exist.
  Open the file and add the following lines:

  1 2 3 4

  [wsl2] ; Sets the time in seconds that a WSL 2 distro can be idle before it is terminated. ; To disable the timeout, set this to -1. vmIdleTimeout=-1

Known Issue of OpenSSH / Docker

• docker stderr=ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or director

  External reference

  Install the ssh-askpass package:

  1	sudo apt install [-y] ssh-askpass

• “Connection timeout” even with SSH connection tested okay

Check docker context on both host and client sides:

1
2
3

docker context ls
docker context show
docker context use [context name]

This solution is based on the built-in systemd-networkd, isc-dhcp-server and hostapd; conflicted with dnsmasq.

It is using the built-in IP forwarding feature against bridging. See also routing vs bridging

Prerequisites

• Traffic and firewall management

1
2

sudo ufw enable
sudo ufw status

• DHCP server

1
2

sudo service isc-dhcp-server start
sudo service isc-dhcp-server status

• WiFi hotspot

1
2
3

sudo systemctl unmask hostapd
sudo systemctl enable hostapd
sudo systemctl status hostapd

Prerequisites

MLNX_OFED_LINUX-5.1 no longer supports lowest-end Mellanox ConnectX3 series; using MLNX_OFED_LINUX-5.0 instead.

sudo mount [MLNX…_ubuntu20.04_x86_64.iso] /mnt/cdrom -o loop
./mlnxofedinstall

Start configuration

sudo mst start
sudo mst status

Details

ibv_devinfo

mlxconfig -d [device_identifier] query
mlxconfig -d [device_identifier] set LINK_TYPE_P1=2

sudo systemctl start opensmd

See also more.

Error: Permission denied … when running Docker (perviously seen)

Better solution than before:

1
2
3

sudo groupadd docker
sudo usermod -aG docker $USER
sudo service docker restart

[UPDATE]
Looks like it is also required to logging out and back so that the group membership change takes effect (how?):

1

su -s ${USER}

And this is to check for group membership:

1

getent group groupname | awk -F: '{print $4}'

Potential issue when Docker for Ubuntu is installed and managed by Snap

Either systemd or systemctl won’t be able to locate or control Docker, which would occur when Docker is pre-installed with Ubuntu Server. Use snap commands instead:

1
2

sudo snap services docker
sudo snap restart docker

Local Build Preparation

This is likely to be a cure-all:

1

sudo apt install -y build-essential

• OS: Ubuntu 16.04

• Building prerequisities: make build-essential python3-dev

• Parallel computing prerequisities: cuda8.0 cudnn6.0

export CUDA_HOME=/usr/local/cuda
export PATH=$PATH:/usr/include:/usr/local/cuda/include
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib:/usr/lib/x86_64-linux-gpu/:usr/local/cuda/lib64

• Python 3.5 packages: chainer tensorflow-gpu cupy

nvidia-cuda-dev is only required when building cupy locally.

If python package installation fails, try with sudo pip install --force-reinstall --ignore-installed --no-cache-dir -vvvv to debug.

https://askubuntu.com/questions/218377/install-remove-language-show-only-english-no-other-choice

http://www.linuxdiyf.com/linux/20344.html

https://askubuntu.com/questions/455682/14-04-chinese-ibus-input-no-options

sudo apt-get –reinstall install language-selector-gnome

sudo apt-get install ibus ibus-clutter ibus-gtk ibus-gtk3 ibus-qt4

ibus restart

im-config -s ibus

sudo ibus-setup

Fatal Error: ezSQL_mysql requires mySQL Lib to be compiled and or linked in to the PHP engine

yum install php56w-mysql

restart Apache

Ubuntu升级gcc,g++

• Prevent boot into Windows automatically when boot from SSD

  Rename or remove the /EFI/Microsoft in the EFI partition.

• Why the bash replied “fdisk/mkfs command not found?”

  Because /sbin is not appended to the $PATH.

• Switch Ubuntu apt-get source

  sudo cp /etc/apt/sources.list /etc/apt/sources.list_backup
  sudo nano /etc/apt/sources.list

  Be aware to choose the corresponding version.

  deb http://mirrors.163.com/ubuntu/ trusty main restricted universe multiverse
  deb http://mirrors.163.com/ubuntu/ trusty-security main restricted universe multiverse
  deb http://mirrors.163.com/ubuntu/ trusty-updates main restricted universe multiverse
  deb http://mirrors.163.com/ubuntu/ trusty-proposed main restricted universe multiverse
  deb http://mirrors.163.com/ubuntu/ trusty-backports main restricted universe multiverse
  deb-src http://mirrors.163.com/ubuntu/ trusty main restricted universe multiverse
  deb-src http://mirrors.163.com/ubuntu/ trusty-security main restricted universe multiverse
  deb-src http://mirrors.163.com/ubuntu/ trusty-updates main restricted universe multiverse
  deb-src http://mirrors.163.com/ubuntu/ trusty-proposed main restricted universe multiverse
  deb-src http://mirrors.163.com/ubuntu/ trusty-backports main restricted universe multiverse

uname -a

• mysql change a user password

SET PASSWORD FOR ‘user-name-here‘@’hostname-name-here’ = PASSWORD(‘new-password-here’);

• 开启php的curl功能

sudo apt-get install php5-curl