This solution is based on the built-in systemd-networkd, isc-dhcp-server and hostapd; conflicted with dnsmasq.

It is using the built-in IP forwarding feature against bridging. See also routing vs bridging

Prerequisites

• Traffic and firewall management

1
2

sudo ufw enable
sudo ufw status

• DHCP server

1
2

sudo service isc-dhcp-server start
sudo service isc-dhcp-server status

• WiFi hotspot

1
2
3

sudo systemctl unmask hostapd
sudo systemctl enable hostapd
sudo systemctl status hostapd

Prerequisites

MLNX_OFED_LINUX-5.1 no longer supports lowest-end Mellanox ConnectX3 series; using MLNX_OFED_LINUX-5.0 instead.

sudo mount [MLNX…_ubuntu20.04_x86_64.iso] /mnt/cdrom -o loop
./mlnxofedinstall

Start configuration

sudo mst start
sudo mst status

Details

ibv_devinfo

mlxconfig -d [device_identifier] query
mlxconfig -d [device_identifier] set LINK_TYPE_P1=2

sudo systemctl start opensmd

See also more.

TL;DR

Cache is to reduce latency in the system.

Cache prefers immutable or static data.
Cache can be stale when not updated. Ask, do we care that much?
Write Policy: Write through cache vs write back cache
Eviction policy: LRU, FIFO, LFU

How to sync between Cache and DB?

It is more complicated when syncing Cache replicas & DB replicas and maintain consistency & high throughput.

Get() follows cache-aside strategy ; update() follows write through strategy;

delete() request both operations into caches and DBs; upon DB changed, trigger some asynchronized threads to wait a certain while and notify the cache again to evict stale data that happened to be brought by concurrent reading from DB into cache (proven and incorporated by Facebook)

Hashing function basically transforms arbitrary pieces of data into fixed size values (typ. Integers)

Consistent hashing

Consistent hashing maximizes the cache hits when adding and removing cache nodes; minimizes key re-distribution; mitigate hotkey problem

TL;DR

Data Analytics at a Glance

Analysis is a detailed examination of something in order to understand its nature or determine its essential features. Data analysis is the process of compiling, processing, and analyzing data so that you can use it to make decisions.

Analytics is the systematic analysis of data. Data analytics is the specific analytical process being applied.

Data analysis solutions, which are broader than big data solutions, are made up of gathering, storing, processing, and visualizing data.

The challenges identified in many data analysis solutions can be summarized by five key challenges: volume, velocity, variety, veracity, and value.

Structured vs Semi-structured vs Unstructured data

• Structured data is organized and stored in the form of values that are grouped into rows and columns of a table. Commonly stored in relational databases.

• Semi-structured data is often stored in a series of key-value pairs that are grouped into elements within a file. Often stored in NoSQL databases or CSV, XML or JSON files.

• Unstructured data is not structured in a consistent way. Some data may have structure similar to semi-structured data but others may only contain metadata. Often takes the form of files or objects.

TL;DR

What?

A container is an executable unit of software in which application code is packaged (encapsulated), along with its libraries and dependencies, in common ways so that it can be run anywhere, whether
on a desktop, on-premises, or in the cloud.

A container include a guest OS; instead it leverages host OS.
Compared to virtual machines, containers virtualize the operating system rather than the infrastructure.
The clear benefit of containers is that you do not have to run a dedicated operating system instance for each virtual environment. Instead, one operating system is virtualized across all the containers.

To do this, containers take advantage of a form of operating system virtualization.

Benefits: isolates processes; controls resources allocated to them; small, fast portable

Use cases

• Containers are small and lightweight, which makes them a good match for microservice architectures where applications are constructed of many loosely coupled and independently deployable smaller services.
• The combination of a microservices architecture and containers is a common foundation for many teams that embrace DevOps as the way they build, ship, and run software.

TL;DR

AWS IaaS

Amazon S3

scalable, durable object storage; decoupling storage from processing; parallelization; centralized, accessible and avoid moving between systems

S3 bucket policy and IAM policy are different.
Blocking the offending website IP traffic in security group may be trivial; makes “allows” but not “denies”

AWS IaC

CloudFormation

AWS CloudFormation treats infrastructure / environment as code.

AWS DBaaS

RDS

RDS Backups are automated, daily full backup; transaction logs are backed-up every 5 min. 7-35 days of retention. Manual DB snapshots have retention as long as possible.

RDS can have up to 5 read replicas, within AZ, cross AZ or cross origin. Async replication. Reads are eventually consistent. Replicas can be promoted. It is possible to setup multi-AZ read replicas for Disaster Recovery(DR).

Error: Permission denied … when running Docker (perviously seen)

Better solution than before:

1
2
3

sudo groupadd docker
sudo usermod -aG docker $USER
sudo service docker restart

[UPDATE]
Looks like it is also required to logging out and back so that the group membership change takes effect (how?):

1

su -s ${USER}

And this is to check for group membership:

1

getent group groupname | awk -F: '{print $4}'

Potential issue when Docker for Ubuntu is installed and managed by Snap

Either systemd or systemctl won’t be able to locate or control Docker, which would occur when Docker is pre-installed with Ubuntu Server. Use snap commands instead:

1
2

sudo snap services docker
sudo snap restart docker

Local Build Preparation

This is likely to be a cure-all:

1

sudo apt install -y build-essential

Proposal 1: A NAS build is not exactly NAS-compliant

Athlon 200GE doesn’t support ECC, though literally all Ryzen CPUs do.

B450 supports ECC with some limitations, accordint to the example user manual:

• AMD Ryzen series CPUs (Pinnacle Ridge) support DDR4 3466+(OC)/3200(OC)/2933/2667/2400/2133 ECC & nonECC, un-buffered memory*
• AMD Ryzen series CPUs (Summit Ridge) support DDR4 3466+(OC)/3200(OC)/2933(OC)/2667/2400/2133 ECC & non-ECC, un-buffered memory*
• AMD Ryzen series CPUs (Raven Ridge) support DDR4 3466+(OC)/3200(OC)/2933(OC)/2667/2400/2133 non-ECC, un-buffered memory*
• For Ryzen Series CPUs (Raven Ridge), ECC is only supported with PRO CPUs.

Part of CPU Support List:

To support ECC and avoid the use of a dedicated GPU at the same time, PRO 2200G/GE is proposed as the lowest-end eligible CPU for refreshed NAS build.

SSD cache is optional for RAM less than 32GB.

Cheap refurbished NICs can be actually kinda model many years ago. 40mm quiet cooling fan replacement needed.

Proposal 2: MSI RMA request

Exchange done.

Proposal 3: Ultimate casual video editing rig, unlimited budget

Wait for annual refresh of Threadripper. TRX40 should continue to receive support.

1200W PSU for dual-GPU. ASUS ROG PSU can be installed upside-down.

Build samples:

• TensorBook
• Deep Learning Workstation

Do not mistakenly accuse the cooler for the most common overheating issue under Windows10.

• The cause is actually the unreasonable Windows power management plan.

• “Performance Mode” actually means “Balanced Mode” and vice versa.

Building environment setup can be very subtle and in-depth. Even the same python 3.6 has a different dependent static library on Linux than on Mac.

Node

brew install nvm
nvm ls
nvm install 11
nvm use 11

Java

/usr/libexec/java_home [-V] [-v 1.8]

Python

Mac built-in python wasn’t fully workable.

brew install pyenv
pyenv versions
pyenv install 3.6.10
pyenv local/global 3.6.10