Newscoop Replication: Article Synchronizing between Newscoop Servers
This document is a first approach to the Radio Package requirement regarding Trac ticket:2330 "Offline editing/Synchronizing Articles with Another Server".
Since Internet connectivity is not that good in some places, radio stations need to be able to create stories on their local intranet and then send them to the main web server (outside of the intranet) whenever connectivity is available. All media from the article must be uploaded to the web server.
The following must be synchronized:
- Database (using MySQL replication)
- images folder
- files folder (attachments)
- look folder (templates)
- And also any Airtime storage server files that have been added as attachments to Newscoop articles.
All what you need to do is run a normal Newscoop installation on both the master and slave servers. The most important thing at this point is to set the same instance name (newscoop is the default) for both installations.
Follow the installation guide from the Newscoop manual to do all of this in a right way.
Newscoop MySQL database replication
MySQL server has built-in support for replication, meaning that you can have on-the-fly copies of data stored in a main database server -known in this context as a Master server- in anothers servers -the Slave servers. Latest releases of MySQL server --5.x- allow you to create really complex replication scenarios, but this is not our case. All what you need to synchronize two newscoop servers is to use a simple MySQL replication scheme, known as Master Master replication.
When using replication, all updates to the tables that are replicated should be performed on the master server.
This way and to be clear, Local Newscoop server will be our 'Master 1' and 'Slave 2', and the Online Newscoop server will be the 'Slave 1' and 'Master 2'.
We really need to implement this scheme because of the article comments management. More details about this in the section Handling Article Comments at the end of this document.
MySQL >= 4.1.0
Make sure MySQL versions installed on master and slave are compatibles to do replication. Take a look at http://dev.mysql.com/doc/refman/4.1/en/replication-compatibility.html.
In general, what we need to do is:
- Ignore ArticleComments, all Phorum tables ( phorum*_ ) and SystemPreferences when replicating from Local to Online.
- Replicate all the rest newscoop tables from Local to Online.
- Replicate ArticleComments and all the Phorum tables ( phorum*_ ) from Online to Local.
So, here we go.
1- Create an account on the 'Master 1' server that the 'Slave 1' can use to connect.
Change 'replication_user', 'slave1server.yourdomain.com' and 'a_strong_password' to whatever you want to use and suit your installation.
2- Create an account on the 'Master 2' server that the 'Slave 2' can use to connect.
Change 'replication_user', 'slave2server.yourdomain.com' and 'a_strong_password' to whatever you want to use and suit your installation.
3- Add the following lines to the MySQL configuration file on the 'Master 1' and on the 'Master 2' servers. The MySQL configuration file path is usually '/etc/mysql/my.cnf', if you installed MySQL from the standard packages.
The value for master-host can be the valid fqdn (fully qualified domain name) or the IP address of your 'Master 2' server.
The value for master-host can be the valid fqdn (fully qualified domain name) or the IP address of your 'Master 1' server.
Note all of the above mysqld entries are replication specific. You usually have another couple of entries there, but it does not matter for replication purposes.
4- We need to make sure both 'Master 1' and 'Slave 1' will have the same Newscoop database at the same status when starting replication.
If you are doing a new installation on both 'Master 1' and 'Slave 1' and you are totally sure that you are installing the same Newscoop package and databases are exactly the same, you can skip this step and continue on 4-. If you do not, continue here.
We have got to make a dump of the 'newscoop' database from our 'Master 1' server, then we will copy it to the 'Slave 1' server. Let us use the mysqldump utility to do this.
You will be prompted to enter the corresponding password for your MySQL user (default user is the current logged in user). Remember, you need to run this command as a MySQL user with the granted privileges required to do this.
This will create a text file -/tmp/newscoop_dump.sql- containing SQL statements to create the 'newscoop' database tables with data.
Now, copy/transfer the dump file to the 'Slave 1' server, make sure MySQL is running there. If you already have a 'newscoop' database on the 'Slave 1' it is time to drop all of the tables on it -do not forget backing up, maybe you still need it-. Finally, enter something like the following on the 'Slave 1':
This will execute all of the SQL statements in the dump file, which will include the CREATE and INSERT statements. Once the dumped-up databases are loaded onto the 'Slave 1' server, go to the final step.
5- Restart both mysql 'Master 1' and 'Master 2', then execute the following SQL statement while logged in as root on both of them:
6- Check replication works
Setting up and getting work this simple kind of replication is quite easy. If you followed the above indications then replication would have to be working well now. However, you can check this by running the SHOW MASTER STATUS command on the masters and SHOW SLAVE STATUS on the slaves. Those commands will give you some information about the current status of both the Master and Slave servers.
If you are experiencing troubles and replication does not work as it should, please take a look at the corresponding section in the MySQL manual and the other links referenced at the end of this document.
Syncing Articles-related Files
This Radio Package feature allows you to be able to synchronize Newscoop Article-related files between the Local server and the Online server over the Internet. Our goal is to ensure no user intervention is required when the server is restarted (for passwords or keys).
Please note these instructions are specific to Ubuntu Linux versions 4.01 and later. However, it should be run quite well on any *NIX type OS.
Just simple like this:
Keep this in mind:
- remoteuser = the user to be use on the Online server
- remotehost = the Online server (hostname or IP address)
- localuser = the user to be use on the Local server
- localhost = the Local server (hostname or IP address)
The remote user must have write access to the Newscoop instance HTML directory on the remote host. The local user must have read access to the Newscoop instance HTML directory on the local host. The default value of this directory is '/usr/local/newscoop/www/newscoop/html'. You can achieve this by adding the remote/local user to the apache group (default is www-data on Debian/Ubuntu and apache on RedHat). On the remote machine you have to run the following command also:
1- Test whether rsync over ssh works at all.
Type the corresponding password when prompted.
2- Configure the Local server.
We need to generate a private/public pair of keys to allow a ssh connection without asking for a password.
You have got to create an empty passphrase, so just press enter when asked for it to generate the keys. Make sure to do this to avoid no other unauthorized user can read the private key file.
Now we will put the public key into the 'authorized_keys' file on remotehost (a.k.a. our Online server) specifically the one for remoteuser.
Let us use scp to transfer the file over from localhost to remotehost.
3- Configure the Online server.
SSH'ing over to remotehost. Once we are on /home/remoteuser/ we need to make sure that every directory and file that we need to authorize connections with the transfered key exist.
Now, we can make connections to the Online server from anywhere. We can edit 'authorized_keys' to increase security by adding somethink like this -use your preferred text editor.
Change the public key, that looks like this:
Note at the beginning of the line. 'local_IP' is the IP address of the Local server, and '/home/remoteuser/.newscoop-rsync/validate-rsync' is a script that you can create -on the Online server- by:
and this is its content:
If 'localhost' has a variable IP address, or shares its address (via NAT or something similar) with hosts you do not trust, omit the 'from="local_IP",' part of the line (including the comma). This way, 'rsync' will be possible from connections using this key.
Make sure that the 'validate-rsync' script is executable by 'remoteuser' on 'remotehost'.
Check you do not allow SSH root access from anywhere, this is always a good security practice. You need to have an entry like this in the SSH daemon configuration file to do that:
4- Let us test if all of the stuff above work well. Run the following command on the local server:
If it does not work, probably it is because of the file permissions, so check them for each file mentioned before.
5- What to transfer.
Files to be synchronized are those contained into 'images', 'files' and 'look' Newscoop directories. They will be transfered over from Local server to Online server. We need to create a file on the Local server with the list of directories and files to be sent.
This is the content you need to put in:
That is it, make sure each entry is on a new line. Save and close the file.
6- Cron Job setup.
Now we create the cron script 'rsync-newscoop-afiles' like this:
The cron script is easy to understand, all what you need to know about it is:
Set executable permission on it:
Finally, we create an entry for this new cron job:
line to insert:
for a by minute synchronization.
Handling Article Comments
If you are setting up a Local and Online servers environment, you need to take care of article comments. Why? Well, static and dynamic pages must be generated by working on the Local server, that is why we need to replicate data over from the Local to the Online server to keep it up to date.
On the other hand, we have got users from all around the world visiting our Website and, hopefully, most of them would like to post comments on the published articles, and they just can do that.
Said that, we will need to be able to replicate those article-comments-related tables back to the Local server.
All the stuff above means that comments management will be enabled in our Local server only whether we are online and able to reach our Online server.
This is what you need to do in order to set up comments replication:
- Create a database user on the Online Server to connect from the Local server.
Now, on the Local server:
- Login to the Newscoop admin interface and go to System Preferences.
- Check Yes for Setting up a Newscoop Replication Server?
- Fill the form fields for the Newscoop Online Server Database as follows:
- Database Host: The IP address or hostname of your Online server.
- Database User: (dbuser) This is your user for the MySQL server at your Online server.
- Database Password: (somepassword) The valid password for the database user.
- Database Port: The port to connect to the remote database server (default is 3306).
That is it, Newscoop will show whether you are able or not to manage article comments on the Local server.
Please, run tests enough to check everything works well, after that you will can go in productive =)