I have recently upgraded Nextcloud from version 15 to version 18 on my yunohost. And even if I’m impressed by the new functionalities, it looks like my Olimex LIME2 board is reaching its limit, it takes quite some time to display Nextcloud’s homepage, and I’m starting to have MySQL’s service failing randomly. As my board at home is really about synchronizing files, pictures, calendar and contacts, I’m going to switch to some other apps that will handle each of these cases, and will deploy Nextcloud on a more performant machine when I need the collaborative features. Sogo will take care of my calendars and contacts, Seafile of my files and Piwigo of my pictures. And I’ll make the most to reinstall my Yunohost, writing everything down so that I don’t have to search everything again everytime I’m doing it, hoping it might help others too.
I’ll cover 3 parts:
- Extracting data from Nextcloud
- Setting up an encrypted home folder for Yunohost to use
- Installing and configuring the apps, and importing data from Nextcloud
Extracting data from Nextcloud
Extracting contacts
I tried to find an easy way to extract contacts from Nextcloud, but ended up retrieving them through the web interface, downloading them as a .vcf file.
Extracting calendars
For this, there is an more direct way than through the web interface. You can retrieve the calendars by just entering this URL in a web browser (or using curl):
https://<Nextcloud server>/remote.php/dav/calendars/<username>/<calendarname>/?export
You need to replace the address for your Nextcloud server, your username, and the calendar name. The default one is usually named “personal”, for the other ones, my observation is that they are all small caps, and blank spaces are replaced by hyphens: a calendar showing as “Work Calendar” should be retrieved as “work-calendar”. And there you go, one .ics file per calendar.
Extracting files
As we are not going to use Nextcloud in the future, all versionning of the current files will be lost (well, you can probably keep them, but I’m not covering this part).
All latest version of files for Yunohost’s version of Nextcloud should be in /home/yunohost.app/nextcloud/data/<username>/files. We need to retrieve these in order to put them in Seafile and/or Piwigo and/or on the Yunohost filesystem afterwards. In my case, I used rsync from the yunohost server to put them on another server. But you can also save files that are synchronized on your laptop with the Nextcloud sync app, or plug in a hard drive on your yunohost server… We just need the files :)
What about the rest?
Well, as I said, in my case Nextcloud was just a place to store and share my files, so I’m not considering retrieving data from apps like Deck. Be sure to save all you want before removing Nextcloud.
Back up Yunohost and the other apps than Nextcloud (optional)
If you want to install Yunohost from scratch, you can pass this section. If you just want to remove Nextcloud without reinstalling Yunohost, you can pass this section. If you want to keep your existing yunohost configuration and/or apps but still reinstall your server, backup your Yunohost through the web admin, or command line, and save them outside of your server.
Moving folders on the external hard drive for Yunohost to use
Principle
As mentioned above, I run Yunohost on an ARM board, so if I want to store a lot of data, using only the SD card is not enough. In my case, I have a SATA disk linked to the board, but the following will work for a USB hard drive too. The idea is to move one or several folders to the separate hard drive. /home stores all user data, so this is the most obvious candidate. Depending on the size of the SD card of your ARM board, you might also want to have /var and/or /tmp on the hard drive - your applications can store a lot of data in /var, and /tmp is sometimes used to store files for updates and I have seen some updates fail because there was not enough space on the default SD card volume for /tmp.
Encrypting the home volume: why?
Self-hosting, and self-hosting at home should make the data you host less available to the companies, associations or people that host online services. It doesn’t mean that it is safe though: anyone with access to your server might also have access to your data. When you think of potential threats, you have people with remote access to your server (did you open SSH ports to the internet? Who has access to your local network - did you give your WiFi password to friends who went to grab a coffee at your place?) and people with physical access to your server (burglars, people living at your place, your friend who is having a coffee with you). Encrypting your home volume should provide more protection to your data when your server is offline, so if your server is stolen for example.
Tools
We are going to use LVM, the logical volume manager, to create and manage the partitions of our external hard drive. And we are going to use cryptsetup to encrypt the partitions. Make sure that you have them installed on your yunohost server:
~$ sudo apt-get install lvm2 cryptsetup
Let’s go!
Prepare the external hardrive
I’m considering that you already installed Debian Stretch on your machine (when I wrote this post, Yunohost was only running on Stretch - please use the latest version required). I also assume the hard drive is plugged in, and that all the following commands are run as root. Get its partition name:
~# fdisk -l
Once you have it, create a physical volume out of it so that your hard drive is available for lvm (replace /dev/sda1 by your partition’s name):
~# pvcreate /dev/sda1
Then, create a volume group that will contain our physical volume:
~# vgcreate vgsata /dev/sda1
We’ll start now creating one logical volume per partition we want to create:
~# lvcreate -n vgsata-lvhome -L 200g vgsata
~# lvcreate -n vgsata-lvvar -L 10g vgsata
~# lvcreate -n vgsata-lvtmp -L 10g vgsata
Now for the volumes you want to encrypt, format and open them:
~# cryptsetup luksFormat /dev/mapper/vgsata-lvhome
WARNING!
========
This will overwrite data on /dev/mapper/vgsata-lvhome irrevocably.
Are you sure? (Type uppercase yes): YES
Enter passphrase:
Verify passphrase:
~# cryptsetup luksOpen /dev/mapper/vgsata-lvhome crypthome
Enter passphrase for /dev/mapper/vgsata-lvhome:
And create the filesystems for each volume:
~# mkfs.ext4 /dev/mapper/crypthome
mke2fs 1.43.4 (31-Jan-2017)
Creating filesystem with 104857088 4k blocks and 26214400 inodes
Filesystem UUID: 684ee08b-eb47-4000-8f12-3a94118e19df
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000
Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks):
done
Writing superblocks and filesystem accounting information: done
~# mkfs.ext4 /dev/mapper/vgsata-lvvar
~# mkfs.ext4 /dev/mapper/vgsata-lvtmp
Copying existing data
Now, we will first copy the existing files in the folder we want to mount on the external hard drive, and then mount them. First, create a temporary mounting point, then for each volume, mount it, copy the content of the original file and unmount the volume:
~# mkdir /mnt/tmp
~# mount /dev/mapper/crypthome /mnt/tmp
~# rsync -avz /home/ /mnt/tmp
~# umount /mnt/tmp
~# mount /dev/mapper/vgsata-lvvar /mnt/tmp
~# rsync -avz /var/ /mnt/tmp
~# umount /mnt/tmp
~# mount /dev/mapper/vgsata-lvtmp /mnt/tmp
~# rsync -avz /tmp/ /mnt/tmp
~# umount /mnt/tmp
Mounting the newly created volumes
For non-encrypted volumes, we can add them directly to the fstab file, so that they are mounted at boot time:
~# echo '/dev/mapper/vgsata-lvvar /var ext4 defaults 1 2' >> /etc/fstab
~# echo '/dev/mapper/vgsata-lvtmp /tmp ext4 defaults 1 2' >> /etc/fstab
Mounting encrypted volumes
When (re)booting, you will need to open your encrypted volumes before mounting them. Some methods using a key file directly on the server, or on a USB key allow you to have the encrypted volume mounted automatically (see here or there for example). I won’t go this way, because it makes it easier for people having physical access to your server to have the data online. You also can have the passphrase automatically asked at boot time by adding the encrypted volume to /etc/crypttab and /etc/fstab. I won’t go this way either because you won’t (and actually I don’t) always or ever have physical access to your server when you reboot it - or a screen attached to it.
The way I suggest to do it is the old fashioned way: through the command-line interface. When your server boots, the encrypted volume is not mounted. Once it has booted, you connect to your server via ssh and mount it manually. You can type the 2 commands, or save the following script outside of the home folder (for example in /root), make it executable and run it after the boot:
#!/bin/bash
cryptsetup luksOpen /dev/mapper/vgsata-lvhome crypthome
mount /dev/mapper/crypthome /home
Things to consider when you encrypt your home folder
I mentioned above to save the script outside of the home folder. Indeed, as you boot without the home folder mounted, you need access to the script to mount it! Similarly, if you later set up the ssh authentication via ssh keys, the file containing the authorized keys is, by default, saved in the home folder of each user. So if you want to be able to ssh in your server right after boot using your ssh key, you will have to either:
- change the location of the authorized_keys file in the /etc/sshd_config file (but you will potentially have to change it back every time you update yunohost, and you won’t benefit from some tools that I’m going to mention later)
- add your authorized_keys file in the relevant folders (/home/admin/.ssh/authorized_keys for example) when the encrypted home folder is not mounted
Finally, some services won’t start properly at boot time unless the encrypted home is mounted. As far as I have noticed, seafile and seahub fail to start without the encrypted home mounted, and transmission-daemon and gitea. You can add to the above script one line per service to restart, for example yunohost service restart seafile, so that the services are restarted right after the encrypted home is mounted. But in order to do so, let’s install yunohost and migrate our data!
Installing Yunohost, the apps and migrating the data we extracted
I won’t go into details about how to install either yunohost or the applications. The documentation is here.
Moving data back in Seafile
Seafile doesn’t store files directly, so there is no direct way to move the data. You can, though, use the synchronization clients. For the data you want to synchronize and that is already on a computer, simply download the synchronization client and synchronize your libraries. If the data lies on a server, or on a hard drive that you plug to your yunohost server, you can use the command-line synchronization client to migrate your data in Seafile. Install it (in my case, the install didn’t work right away and I needed to install the software-properties-common package on my yunohost server first to be able to install seafile-cli).
Now that the client is installed, let’s initialize it and create a library (all this comes from the official documentation and from the command’s help:
# choose a folder where to store the seafile client settings e.g ~/seafile-client
~# mkdir ~/seafile-client # create the settings folder
~# seaf-cli init -d ~/seafile-client # initialise seafile client with this folder
~# seaf-cli start
~# seaf-cli create -n "the library name" -t "optional: the library description" -e "optional: the library password" -s "the seafile server URL" -u "the yunohost user's email" -p "optional: the yunohost user's password" -a "optional: the 2-factor authentication code" # if you don't enter the password here, it will be prompted
In the last command, enter the server as https://{your ynh URL}/{the seafile path}, your username needs to be the full ynh email, and not only the username, and you can enter the command without the -p option so that your password doesn’t stay in clear in the bash history. The ID of the newly created library will be displayed, copy it. If you want to synchronize with an existing seafile library, you can list them via seaf-cli list-remote -s "the seafile server URL" -u "the yunohost user's email" -p "optional: the yunohost user's password" -a "optional: the 2-factor authentication code" and copy the ID from the displayed list. The ID is also in the URL of the library when you open it in a web browser.
With the ID of the library, we can synchronize the files that we retrieved from Nextcloud now:
seaf-cli sync -l "the id of the library" -s "the seafile server URL" -d "the path to the folder with Nextcloud data that we want to put in Seafile" -u "the yunohost user's email" -p "optional: the yunohost user's password" -a "optional: the 2-factor authentication code"
You can check the status of the synchronization with seaf-cli status. Once the upload is done, you can unsync the library and remove the files from the server/hard drive/USB key:
seaf-cli desync -d "the folder where the data from Nextcloud lies"
Organizing libraries and folders in Seafile
Seafile and Nextcloud are organized somewhat differently, and more importantly the synchronization clients work differently. With the Nextcloud one, you can choose with a fine granularity which folders on Nextcloud you want to synchronize locally. The Seafile synchronization client allow you to synchronize libraries only, so you should take this in consideration when organizing your data in Seafile - do not put everything in a library if you want to be able to synchronize locally with only part of it. You also can protect libraries in Seafile with a password - organizing your data in Seafile can also allow you to easily protect more sensitive data in password-protected libraries.
Importing calendars and contacts in SOGo
Remember the ics files we retrieved from Nextcloud? This is the time to use them. In SOGo (that you have already installed, right?), go to the Calendar tab, create a new calendar for each ics file you have, and once created, click on the 3 dots on the right of the calendar’s name, choose “Import” and put your ICS files there.
For contacts, the idea is the same: first create the address book(s), then click on the 3 dots, and import your vcf file.
What about photos?
This is a tricky question and I’m still not sure of my take on this. But well, if you made it til here I can talk about it. I ended up separating 3 usages I have with pictures. First, making sure I have a backup of the pictures I take with my cell phone. This is taken care of by the Seafile Android (or iOS) app. Then, having an archive of all my pictures that I can go to occasionally. I have put this archive on the home folder of my ynh user, accessing my data via ssh/sftp. To do this, yunohost provides some tools that I’m not sure are documented (at the time I write this article at least):
~# yunohost user ssh allow "username" # allows the user to connect through ssh, using his/her password
~# yunohost user ssh add-key "username" "ssh public key" # adds a public key to the user's authorized keys
As I mentioned above, Yunohost by default writes the authorized_keys in the home folder, which won’t be mounted at boot time if you have encrypted your home volume.
Finally, I use Piwigo to organize the pictures I want to share. You can add the pictures through the web interface if you don’t have a lot, but if you already have everything ordered, you can use the FTP upload. Just copy your pictures in the /home/yunohost.app/piwigo/galleries folder, and follow the instructions of the documentation.
Personal feedback
It took me a few months to write down this article, which gave me some time to use this combination of apps daily. I’m very happy with SOGo, and the web interface of Seafile, which are very fast to display and user-friendly. I had some issues at the beginning with Piwigo: I tried the FTP feature with ALL my pictures and well, my board hasn’t come more powerful so it failed, hence the organization I suggest above. Same as for the Nextcloud mobile apps, I still have some trouble having a good two-way synchronization on folders, but apart from that, I’m really happy with this new organization!