Skip to content

Changes

Page history
Create tutorials/multi storage server authored by rizzle's avatar rizzle
Hide whitespace changes
Inline Side-by-side
tutorials/multi-storage-server.md 0 → 100644
View page @ 75c8828b
# OpenWRT tgt on PXE
## Building openwrt
### Prerequisite
Checkout and prepare openwrt as seen here: https://wiki.openwrt.org/doc/howto/build
Enable the following settings:
```
Target System (x86)
Subtarget (x86_64)
Target Profile (Generic)
Target Images --->
[*] ramdisk
[ ] tar.gz
[ ] ext4
[ ] squashfs
Global build settings --->
[*] Compile the kernel with asynchronous IO support
[*] Compile the kernel with direct IO support
[*] Compile the kernel with support modern file notification support
[*] Compile the kernel with SCSI generic v4 support for any block
Administration --->
<*> htop
Kernel modules --->
Block devices --->
<*> kmod-ata-core
<*> kmod-ata-ahci
<*> kmod-md-mod
<*> kmod-md-raid456
<*> kmod-scsi-generic
Input modules --->
<*> kmod-hid-generic
Native Language Support --->
<*> kmod-nls-utf8
Network Devices --->
<*> kmod-e1000
<*> kmod-e1000e
USB Support --->
<*> kmod-usb-ohci-pci
<*> kmod-usb-storage-extras
<*> kmod-usb-uhci
<*> kmod-usb2-pci
<*> kmod-usb3
Video Support --->
<*> kmod-fbcon
Network --->
File Transfer --->
<*> rsync
<*> wget
< > ppp
<*> tgt
Utilities --->
Disc --->
<*> blkid
<*> fdisk
<*> gdisk
<*> hdparm
<*> lsblk
<*> lvm2
<*> mdadm
Editors --->
<*> vim-full
Terminal --->
<*> screen
<*> mc --->
[*] Enable largefile support
[*] Enable virtual filesystem support
```
Now build the image. The kernel to boot can be found in `bin/targets/x86/64/lede-x86-64-ramfs.bzImage`.
### Openwrt configuration
To configure the image, first boot it on the target machine. Then configure the network (`/etc/config/network`), hostname (`/etc/config/system`) and after that the iscsi target (`/etc/config/tgt`). When everything works fine, the configuration files have to be copied in the openwrt build root in a subfolder called files while keeping the folder structure. this should look like this when done:
```
| source
+-+ files
+-+ etc
+-+ config
| network
| system
| tgt
```
After booting this machine for the first time, the file `/etc/dropbear/dropbear_rsa_host_key` is created. To have the same key in the image on every boot, it should be copied in the `$SOURCE/files/etc/dropbear` folder.
Afterwards rebuild openwrt (this should not take as long). Then the configuration files are included in the resulting ramfs. When booting this image the iscsi target should be exposed automatically.
### PXE configuration
Copy the resulting `bin/targets/x86/64/lede-x86-64-ramfs.bzImage` to the PXE server e.g. to `diskless/tgt1/lede-x86-64-ramfs.bzImage`. Then add the following entry to the `pxelinux.cfg`:
```
LABEL tgt1
KERNEL diskless/tgt1/lede-x86-64-ramfs.bzImage
```
# Gentoo on initiator
## Install the base system
Follow the standard gentoo installation guide and the "Boot from NFS root" guide.
### Detecting the gcc parameters
One of the following commands can be used on the target machine to detect the correct `-march` gcc parameter that has to be set in the file `portage/make.conf` `CFLAGS=“”`-section.
```
gcc -c -Q -march=native --help=target
```
or
```
gcc -march=native -E -v - </dev/null 2>&1 | grep cc1
```
### Basic program installation
The following use flags can be set for the packages that will be installed:
In `portage/package.use/screen`:
```
app-misc/screen multiuser
```
In `portage/package.use/iputils`:
```
net-misc/iputils -caps -filecaps
```
In `portage/package.use/lvm2`:
```
sys-fs/lvm2 -thin
```
Those global use flags can be used in `portage/make.con`:
```config
USE="bash-completion offensive truetype vim-syntax cjk unicode btrfs cryptsetup python"
```
Install the following additional packages. This can be done on the target system. When it is done on the target system only, `-march=native` can be used.
```bash
$ emerge -av open-iscsi app-misc/mc sshfs nfs-utils hddtemp pwgen sudo cksfv p7zip par2cmdline rar unace unrar unzip zip cdrkit cuetools app-crypt/gnupg pinentry vim app-misc/screen gentoolkit iftop nmap tcpdump vnstat bind-tools cifs-utils samba ncftp bridge-utils iputils ntp wget hdparm pciutils pv smartmontools usbutils usermode-utilities btrfs-progs cryptsetup lvm2 mdadm treesize htop iotop lsof
```
## Set up the initiator
Now the initiator can be configured. It should connect to all iSCSI targets. Afterwards the lvm is created on top of all devices. Then this device is encrypted with `cryptsetup` (the key can be encrypted via `gpg`).
This whole process should be encapsulated in a bash script, as the mounting of the encrypted volume has to be done manually with the following script:
```bash
#!/bin/bash -x
CIPHER="aes-xts-plain64"
HASH="plain"
KEYSIZE="512"
KEYID=$GPG_KEY_ID
KEYFILE="keys/key.`hostname -s`.gpg"
if [ `id -u` -ne 0 ]; thne
echo "Must be root!"
exit
fi
iscsiadm -m node -L all
lvscan
gpg --trust-model always -decrypt $KEYFILE | \
cryptsetup open --type=plain /dev/mapp/raidvg-raidlv --key-file=- --cipher=$CIPHER --hash=$HASH --key-size=$KEYSIZE raidcrypt
mount -o defaults,noatime,exec /dev/mapper/raidcrypt /media/data
```
## Create the crypted raid
First create the raid devices:
```bash
madam --create /dev/md0 -l $LEVEL -n $NUMBER_DEVICES -b internal $BLOCKDEVICES
```
Do this on every target and on the iniator.
On every target you have to export the raid device via iscsi. To assemble the raid on boot in openwrt, create a `mdadm` config in `/etc/config/mdadm`.
```
config array
option device /dev/md0
option uuid $UUID
config array
option device /dev/md1
option uuid $UUID2
```
Now you have to export the raid devices via iSCSI. Edit `/etc/config/tgt` on openwrt.
```
config target 1
option name 'iqn.2017-02.com.example.storage:md'
config lun 1_1
option device /dev/md/0
# option blocksize 4096
config lun 1_2
option device /dev/md/1
# option blocksize 4096
```
Login to all iSCSI targets on the central initiator.
```bash
$ iscsiadm -m node --targetname 'iqn.2017-02.com.example.storage:md' --portal '$IP:3260' --login
```
This should generate a block device for every lun specified in the target config. After having all those block devices on the initiator an elvm can be initialized on top of them.
```bash
$ pvcreate $DEVICES
$ vgcreate raidvg $DEVICES
$ lvcreate -l100%VG -n raidlv raidvg
```
Now we have a single big lvm volume group named `raidvg` and on it a single big logical volume named `raidlv`. We now generate our key file and encrypt it with gpg:
```bash
$ dd if=/dev/random bs=4K count=1 | gpg --encrypt -r $KEYID --trust-model always -o $KEYFILE
```
We specify the recipient key id with `$KEYID`. This should be the id of a public key known to gpg. This key can then be used to decrypt this key.
Now we can use `cryptsetup` to create the crypted block device with the generated key.
```bash
$ gpg --trust-model always --decrypt $KEYFILE | \
cryptsetup open --type=plain /dev/mapper/raidvg-raidlv --key-file=- \
--cipher=aes-xts-plain64 --hash=plain --key-size=512 raidcrypt
```
This first decrypts the key and then feeds it directly to cryptsetup. The hash has to be plain as we use a completely random key. The key-size may vary, as may the cipher. The resulting crypted block device can be found in `/dev/mapper/raidcrypt`.
Now we can create a filesystem on top the crypted block device:
```bash
$ mkfs.ext4 -m 0 -L data /dev/mapper/raidcrypt
$ mkdir /media/data
$ mount /dev/mapper/raidcrypt /media /data
$ chmod gu=rwx,o=rx /media/data
$ chown rizzle:users /media/data
$ umount /media/data
```
This creates the filesystem and sets it writable to a specific user.
To use gpg with the custom script the following line has to be added to `~/.gnupg/gpg-agent.conf`.
```
allow-loopback-pinentry
```
## Building initrd on the initiator
First create a folder that should contain the initramfs. This can be done as suggested here: https://wiki.gentoo.org/wiki/Custom_Initramfs. Assuming we use `/usr/src/initramfs` we have to created the following folders and block devices:
```bash
$ mkdir -p /usr/src/initramfs/{bin,dev,etc,lib,lib64,mnt/root,proc,root,sbin,sys}
$ cp -a /dev/{null,console,tty,tty0} /usr/src/initramfs/dev/
```
We now change to the folder:
```bash
$ cd /usr/src/initramfs
```
Now we need busybox as static binary (emerge it with the USE-flag `static`). We copy it to our initramfs:
```bash
$ cp -a /bin/busybox bin/busybox
```
As we later use the udhcpc for getting an IP and it's script does not know of busybox we need at least an sh-symlink to busybox:
```bash
$ ln -s busybox bin/sh
```
The core functionality is done in the file `init` in the root of the initramfs. It contains the following:
```bash
#!/bin/busybox sh
rescue_shell() {
echo "Something went wrong. Dropping to shell."
exec sh
}
kernel_cmdline() {
local value
value=" $(cat /proc/cmdline) "
value="${value##* $1=}"
value="${value%% *}"
[ "$value" != "" ] && echo "$value"
}
get_name_for_mac() {
local MAC=$(echo "$1" | tr '[:upper:]' '[:lower:]')
ifconfig -a | grep 'HWaddr' | while read I; do
local TEMP_MAC="${I##* }"
if [ -n "$TEMP_MAC" ] && [ "$TEMP_MAC" == "$MAC" ]; then
echo "${I%% *}"
return
fi
done
}
mount -t proc proc /proc
mount -t sysfs sysfs /sys
NIC_PRIMARY=$(kernel_cmdline 'nfsroot_primary_nic')
MAC_PRIMARY=$(kernel_cmdline 'nfsroot_primary_mac')
if [ -z "$NIC_PRIMARY" ]; then
if [ -n "$MAC_PRIMARY" ]; then
NIC_PRIMARY=$(get_name_for_mac "$MAC_PRIMARY")
fi
if [ -z "$NIC_PRIMARY" ]; then
NIC_PRIMARY="eth0"
fi
fi
echo "Using NIC $NIC_PRIMARY as primary network device."
ip link set name primary $NIC_PRIMARY
brctl addbr lan
brctl addif lan primary
ip link set dev primary up
ip link set dev lan up
NFS=$(kernel_cmdline 'nfsroot')
NFS_DEFAULT_OPTS="nolock,noacl,rw,hard,intr,vers=3,rsize=65535,wsize=65535"
if [ -z "$NFS" ]; then
echo "Please specify a nfsroot= variable to the kernel cmdline (without the options)."
echo "The default mount options are $NFS_DEFAULT_OPTS."
rescue_shell
fi
NFS_ROOT="${NFS%%,*}"
NFS_OPTS="${NFS#*,}"
if [ "$NFS_OPTS" == "$NFS" ]; then
NFS_OPTS="$NFS_DEFAULT_OPTS"
fi
udhcpc -i lan || rescue_shell
mount -t nfs -o "$NFS_OPTS" "$NFS_ROOT" /mnt/root || rescue_shell
umount /proc
umount /sys
exec switch_root /mnt/root /sbin/init || rescue_shell
```
This does the following:
1. mount `/proc` and `/sys` once the system has started
2. rename the primary NIX to the name `primary`
3. add a new bridge named `lan`
4. add the interface `primary` to said bridge
5. start the interfaces
6. run the dhcp client on the bridge interface
7. mount the nfs server to `/mnt/root`
8. umount `/proc` and `/sys`
9. switch to the system root on nfs and continue booting
The primary NIC is one of the following:
- specify a `nfsroot_primary_nic` with the name of the device
- specify a `nfsroot_primary_mac` with the MAC of the device
- specify nothing and default to `eth0`
The nfs root is specified with the `nfsroot` variable`. If no mount options are provided a reasonable default is used.
To get DHCP working we need `udhcpc` which is included in busybox. This does nothing on itself but needs a script which it executes upon state change:
```bash
$ mkdir -p usr/share/udhcpc
$ cp /usr/share/udhcpc/default.script usr/share/udhcpc
```
# Infiniband SRP on gentoo
## Infiniband Target
First install a base Gentoo system. The kernel has to be configured to enable the srpt driver for scsi over infiniband target support. I use Mellanox MT25418 ConnectX VPI PCIe 2.0 controllers. This requires the following (maybe this is too much, at least it works):
```
Device Drivers --->
<*> Generic Target Core Mod (TCM) and ConfigFS Infrastructure --->
<*> TCM/IBLOCK Subsystem Plugin for Linux/BLOCK
<*> TCM/FILEIO Subsystem Plugin for Linux/VFS
<*> TCM/pSCSI Subsystem Plugin for Linux/SCSI
<*> Linux-iSCSI.org iSCSI Target Mode Stack
<*> InfiniBand support --->
<*> InfiniBand userspace MAD support
<*> InfiniBand userspace access (verbs and CM)
[*] InfiniBand on-demand paging support
<*> Mellanox HCA support
<*> Mellanox ConnectX HCA suport
<*> InfiniBand SCSI RDMA Protocol
<*> InfiniBand SCSI RDMA Protocol target support
<*> iSCSI Extensions for RDMS (iSER)
<*> RDMA verbs transport library
File systems --->
Pseudo filesystems --->
[*] Userspace-driven configuration filesystem
Library Routines --->
<*> CRC calculation for the T10 Data Integrity Field
```
The the following use flags have to be changed. `targetcli-fb` does not work with `python3_5`:
```
sys-fabric/opensm tools
sys-fabric/ofed srp perftest qperf ofed_drivers_mlx4 ofed_drivers_mlx5 ofed_driver_mthca
sys-block/targetcli-fb -python_target_python3_5
```
`targetcli-fb` needs `dbus`:
```bash
$ sudo emerge -av opensm ofed targetcli-fb dbus
```
We have to identify the gid of our device via:
```bash
$ ibstatus
Infiniband device 'mlx4_0' port 2 status:
default gid: fe80:0000:0000:0000:0002:c903:0002:d504
...
```
This has to be used with targetcli to create the export:
```bash
$ targetcli
// first create the backstore (this are the TCM subsystems configured in the kernel above)
/> cd /backstore/iblock
/backstores/fileio> create a /root/disk 512M
// now we can prepare the endpoint of the target
/> cd /ib_srpt
/ib_srpt> create 0xfe800000000000000002c9030002d504
Created target 0xfe800000000000000002c9030002d504.
/ib_srpt> cd 0xfe800000000000000002c9030002d504/
/ib_srpt/0xfe...2c9030002d504> cd luns
// now we have to export the "disk" we prepared earlier
/ib_srpt/0xfe...0002d504/luns> create /backstores/fileio/a
Selected LUN 0.
Created LUN 0.
// now we have to allow the gid from the initiator. we can determine it via ibstatus the same way as on the target
/ib_srpt/0xfe...0002d504/luns> cd ../acls
/ib_srpt/0xfe...0002d504/acls> create 0x00000000000000000002c9030002d4d7
Created Node ACL for 0x00000000000000000002c9030002d4d7
Created mapped LUN 0.
```
It seems you have to change the GID of the initiator and replace the starting `fe8` with `000`. Just run `dmesg` on the target after trying to connect to see the gid that is trying to connect.
Now we can search for this file on the initiator via (try the different umad? if you get nothing and start the `opensm` service):
```bash
$ ibsrpdm -c -d /dev/inifiniband/umad0
id_ext=0002c9030002d502,ioc_guid=0002c9030002d502,dgid=fe800000000000000002c9030002d504,pkey=ffff,service_id=0002c9030002d502
```
After finding the connect string we can add the target via the following:
```bash
$ for i in $(ibsrpdm -c -d /dev/infiniband/umad0); \
do echo $i > /sys/class/infiniband_srp/srp-mlx4_0-2/add_target; done
```
The new block device should now exist. If not, look in `dmesg` on initiator and target.
To disconnect the iSCSI device do the following on the initiator:
```bash
$ echo 1 >/sys/class/srp_remote_ports/port-10\:1/delete
```
## Errors
Two patches are needed in case of errors on `saveconfig` in `targetcli`. The first should be saved as `/etc/portage/patches/dev-python/rtslib/fix-asList.patch`
```patch
--- a/rtslib/config_parser.py.orig 2018-04-28 15:12:46.828929573 +0200
+++ b/rtslib/config_parser.py 2018-04-28 15:13:04.748382389 +0200
@@ -225,7 +225,7 @@
value = tokin[0].asDict()
ref_path = value.get('ref_path')
if ref_path is not None:
- ref_path = " ".join(ref_path.asList())
+ ref_path = " ".join(ref_path)
tokout = {'type': 'attr',
'line': pp.lineno(idx, source),
'col': pp.col(idx, source),
```
The second one should be saved as `/etc/portage/patches/dev-python/rtslib/fix-sprt_wwn.patch`
```patch
--- rtslib/config.py.orig 2018-04-28 15:25:15.806055284 +0200
+++ rtslib/config.py 2018-04-28 15:25:50.704989268 +0200
@@ -347,6 +347,9 @@
elif val_type == 'naa':
if is_valid_wwn('naa', value):
valid_value = value
+ elif val_type == 'srpt_wwn':
+ if is_valid_wwn('srpt_wwn', value):
+ valid_value = value
elif val_type == 'backend':
if is_valid_backend(value, parent):
valid_value = value
--- rtslib/utils.py.orig 2018-04-28 15:28:12.260665182 +0200
+++ rtslib/utils.py 2018-04-28 15:29:29.458306934 +0200
@@ -560,6 +560,9 @@
and re.match(
"[0-9A-Fa-f]{8}(-[0-9A-Fa-f]{4}){3}-[0-9A-Fa-f]{12}$", wwn):
return True
+ elif wwn_type == 'srpt_wwn' \
+ and re.match("^0x[0-9a-fA-F]{32}$", wwn):
+ return True
else:
return False
```
Additionally, gentoo is missing the policy files located in `/var/target/policy`. Those can be found in the source packages, found in `/usr/portage/distfiles/rtslib-*.tar.gz`. Just extract the policy folder therein into `/var/target/`.
## Infiniband initiator
```
Device Drivers --->
<*> InfiniBand support --->
<*> InfiniBand userspace MAD support
<*> InfiniBand userspace access (verbs and CM)
[*] InfiniBand on-demand paging support
<*> Mellanox HCA support
<*> Mellanox ConnectX HCA suport
<*> InfiniBand SCSI RDMA Protocol
<*> iSCSI Extensions for RDMS (iSER)
<*> RDMA verbs transport library
```
The the following use flags have to be changed. `targetcli-fb` does not work with `python3_5`:
```
sys-fabric/opensm tools
sys-fabric/ofed srp perftest qperf ofed_drivers_mlx4 ofed_drivers_mlx5 ofed_driver_mthca
```
Take care, that the gentoo `opensm` init script allows only one running opensm demon. You need a separate demon for every link. To select the link the parameter `-p` is used. To run multiple `opensm` processes, multiple log files have to be specified by the `-f` parameter. Use this:
```bash
#!/bin/bash
# get all known ports
PORTS=`/usr/sbin/ibstat -p`
# run opensm on every port
for PORT in $PORTS; do
echo -n "Starting opensm on $PORT: "
start-stop-demon --start --quiet --make-pidfile --pidfile /var/run/opensm-$PORT --background --exec /usr/sbin/opensm -- -g $PORT -f /var/log/opensm.$PORT.log
RETVAL=$?
echo
done
```
# References
- https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-configuring_the_base_rdma_subsystem
- https://serverfault.com/questions/327947/set-up-simple-infiniband-block-storage-srp-or-iser
- http://linux-iscsi.org/wiki/Targetcli
\ No newline at end of file