Setting Up A Local Mirror And Keeping It Up-To-Date

by Alfredo Vogel (vanitarium)

Introduction

In this previous PCLinuxOS Magazine article by Loyed from December 2007, you can learn how to set up a local mirror on an external or internal hard disk.

The article was published in 2007, and a lot of things have changed since then. So, here is my attempt to create a guide for 2020.

The need to have a local mirror may seem bizarre when everything is in the cloud or internet servers these days. However, I found that when you have several PCs running PCLinuxOS, it may come in handy to have an updated mirror ready to update your PC (laptops) without using the internet connection, but just the local area network.

This article describes the necessary commands to create the local mirror.

Then, it describes how to periodically update the mirror content.

All the necessary commands are given and described with real world examples using the main PCLinuxOS repository.

Basic Requirements

In this section, you will learn about the basic requirements to start creating a local repo mirror.

I am assuming you are working on a machine using PCLinuxOS under KDE5. If you are using XFCE or Mate, you can transpose this guide easily, I hope.

Hardware: You need a hard disk drive (HDD) or a Solid State Drive (SSD) with a capacity of 80-200GB. The drive can be an internal drive, or an external (USB) drive. Format it to use the ext4 file system using Gparted in File Tools.

Software: PCLinuxOS running and rsync installed. Check in Synaptic package manager to see if rsync is already installed. In Synaptic, search for rsync and install or verify it is there already. Check if Task Scheduler is installed in synaptic. We need this for periodically updating the mirror.

The user (YOU): You need to have some experience in using the terminal and the commands needed for this task. I also recommend that you learn a bit about FTP and directory hierarchy in the PCLinuxOS repositories. You shall learn about this in this guide.

You will also need to have patience and to be able to persevere if problems happen or something does not work!

Rsync Command And It's Secrets

In the article by Loyed mentioned previously, he writes "It took some playing around for me to finally get the right combination, but knowing how it works (or just using one of the two sources presented here), it should go faster for you." when he was trying to work out the link to the repository inside the rsync command.

Well, you will not have to guess, but you will learn how to find out what goes before and after the "::".

PLEASE NOTE THAT FROM HERE ONWARDS ALL THE COMMANDS HAVE TO BE ISSUED AS ROOT USER.

You do this by typing su and your password when prompted in the terminal console.

The rsync modules idea

When you look at the repository URL you get this:
http://ftp.nluug.nl/pub/os/Linux/distr/pclinuxos/pclinuxos/apt/ pclinuxos/64bit

If you now type

rsync
http://ftp.nluug.nl/pub/os/Linux/distr/pclinuxos/pclinuxos/apt/pclinuxos/64bit

in the terminal you get an error message like this one:

ssh: Could not resolve hostname http: Name or service not known rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.3]

The reason for this error is that rsync is running as a "daemon" on the server, and a special notation is required for the daemon to understand what you are sending to the server.

You now have to do some typing on your terminal and spend some time reading a lot of information about rsync.

I want you to type the following into your terminal:

man rsync

and scroll down to a section called CONNECTING TO AN RSYNC DAEMON.

Inside this section is a very important sentence, which I quote here:

Using rsync in this way is the same as using it with a remote shell except that:
you either use a double colon :: instead of a single colon to separate the hostname from the path

This explains why there is this magic :: in the article by Loyed, and where he struggles to find the correct notation.

So, the correct way to talk to the remote daemon is

rsync [options] [host address]::module/path... /localfolder

As an example we could type

rsync [options] ftp.nluug.nl::pub/os/Linux/distr/pclinuxos/pclinuxos/apt/pclinuxos/64bit /localfolder

for the server quoted above. Note that the http:// prefix has been omitted.

Now, to find the correct method to get to the folder, you want to download for your local mirror. The "man rsync" command gives you some more help.

To check every module of the server (the stuff after the ::) you can issue this command:

rsync ftp.nluug.nl::

with the response:

It is a very long list of all available modules on the server.

We now need to know where our PCLinuxOS files are on the server. For this purpose, open a browser and input the server address as above:

http://ftp.nluug.nl/

You will see a list of folders and files similar to the one you got in your terminal earlier. Click and look at the file highlighted in figure 1, and you will see a list of content on the server in user-friendly format, as shown in figure 2.

So, this is the folder that interests you, because it was included in the original server/repository web address shown above:

http://ftp.nluug.nl/pub/os/Linux/distr/pclinuxos/pclinuxos/apt/pclinuxos/64bit

You are now ready to prepare the full command for rsync and find the PCLinuxOS repository (and hopefully understand the method to talk to the daemon on the server).

If you now type the following into the terminal as a command, you should be able to recursively get to the folder called /64bit that contains all the PCLinuxOS files.

rsync ftp.nluug.nl::pub/os/Linux/distr/

Note the / at the end of the command. This will make the daemon show you the folder content. Try to type it without the / at the end and you get the size and info about the folder, but not its contents.

To avoid very large lists of files and errors, you should type the following command:

rsync ftp.nluug.nl::pclinuxos/pclinuxos/apt/pclinuxos/

which will return this :

drwxrwxr-x 195 2020/02/18 04:47:32 64bit

You have now reached the PCLinuxOS mirror in the folder 64bit.

If you navigate in the browser to this level on the server, you will see the contents of the mirror as shown in figure 3.

These are the files you are going to copy and synchronise into your local mirror on your hard disk.

Because we are all part of the great PCLinuxOS Community and we have access to The PCLinuxOS Magazine, the task is made easier.

There is another article in The PCLinuxOS Magazine from June 2010, by Robert Stahl (Hairyplotter).

He has created a script that is an example of good programming. He includes lots of comments, which aids understanding.

I shall quote his article here to explain further details of rsync and his script.

So, let us follow the steps as presented in Stahl's article.

Step 1:

Plan the location of your mirror:

1. on an external disk or,

2. on an internal disk with available space

You need at least 200GB to be sure to get all of the files. This depends very much on what sections of the repository you want to include, as shown in Figure 3. Normally, you would not include the Test and Unstable sections, for example.

The script you are going to use is shown in Figure 4. There are some modifications you have to do to this script to adapt it to your environment.

Figure 4: Script for cloning and updating the mirror

Step 2:

My disk is internal and is mounted on "/media/PCLOSmirror/".

You have to change that to your own mounting point. There are two places to change it near the top of the script Line 32 and 33, or wherever you see /media/PCLOSmirror/.

REPFILE="change this" and LOGFILE="change this"/rsync/log

Further down you have to change the /media/PCLOSmirror/

in the rsync command line 54.

If you have copied and pasted the script into Pluma or Kwrite you can do a simple search and replace for /media/PCLOSmirror/ and replace by your own disk directory.

Step 3:

Put the script into "/usr/local/bin/updateMIRROR.sh" and open TASK SCHEDULER.

FIGURE 5. Task Scheduler job for updating your mirror using the script updateMIRROR.sh

You have to create a new task according to Figure 5.

Step 4:

Run the script for the first time by making it executable typing

"chmod +x /usr/local/bin/updateMIRROR.sh"

into the root terminal.

This will take some time, depending on your internet connection.

Finally, open Synaptic and add your local disk to the repository list. Activate it, and next time you update or install, you will be using your local mirror.

Leave your computer running so the script can update the mirror.