Data migration

Data migration

The Data Migration Tool helps automate the migration from one environment to another. 

1. Before the migration starts

Before startingthe data migration, it is necessary to ensure that the source environment is first migrated to version 12, and that the new environment is created (but not initialized). 

2. Migration Steps

2.1 Linking environments

To start the data migration, go to the creation page of the new Constellio environment and click on " Import another Constellio instance's data". A migration token will be generated. 

Return to the source environment.

1. Go to "Administration";
2. Select "Data Export ";
   
3.  Click on " Migrate data to another instance".

The migration process will begin and the next steps will unfold in order to complete the migration. 

1. Connecting to the new instance
2. Importing installation settings
3. Importing Configurations and Metadata
4. First import of files
5. Validation on the new environment
6. Final import of configurations and metadata
7. Final import of files

2.2 Connecting to the New Instance

1. Copy the URL of the new Constellio environment and paste it into the "Server URL " field of the source environment;
2. Copy the data migration key into the new Constellio environment and paste it into the "Data migration key " field;
3. Clicking on "Test", the button allows Constellio to make sure that the URL and the migration key match;
4. Click "Next".

"Test" button

When the URL and key match, you'll find this icon:

When the URL and key don't match, you'll find this icon:

2.3 Migrating Installation Settings

This step is to migrate these items to the server :

• Special properties of the constellio.properties file
• The private key kept in the key.txt
• The private key stored in the file specified in the "signatureKey.file" property (if the e-signature feature is installed)

Simply click " Next".

2.4 Migrating Configurations and Metadata

1. Migrate the configurations of your Constellio environment. Click on "Next".
2. Migrate the metadata (Solr) from your Constellio environment. Click on "Next".

Duration

It could take several hours to complete this operation. Rely on the status of the task in the background. 

2.5 Vault Migration

The migration of the file vault must be done manually. Three methods can be used to achieve this end; rsync, NFS, and SCP (as a last resort). 

2.5.1 rsync method

Prerequisite

• Have an SSHconnectionbetween the two servers directly, rather than a connection to each server via your computer.

• Install rsync on both servers. For CentOS 7, use the command: rpm -i le.fichier.rpm. Download it from internal.constellio.me/constellio/dl?id=00000962721

Procedure

1. First, make sure that the user has the necessary rights to write to the destination folder on the target server, usually under /opt/constellio/contents. It is also possible to copy the source folder to another temporary location, then change its rights before moving it to the final destination.

2. Example: constellio@serveur.de.destination $ mkdir -p /opt/constellio/contents.cible

3. Then, from the source server, run the following rsync command. Be careful, this command will generate many logs in the console:

   1. rsync -avz /opt/constellio/contents/ constellio@serveur.de.destination:/opt/constellio/contents.cible

      1. (Remove the → flag v ( rsync -az) to avoid displaying the logs).

4. Rsync allows the transfer resumption in case of interruption. To interrupt the command you can press CTRL+C.

5. After the transfer is complete, you can:

   1. Rename a folder (Rename folder to folder2): mv /chemin/vers/le/dossier /chemin/vers/le/dossier2

   2. Move files: mv /chemin/vers/le/dossier/source /chemin/vers/le/dossier/ou/est/la/destination

   3. Change the owner: chown constellio-service:constellio-service -R /chemin/vers/le/dossier

   4. Verify the owner of the files: ls -salut

2.5.2 NFS Method

Prerequisite

• SSH connection possible on both machines.

• Have portmap and nfs-utils installed on both servers. For CentOS 7, Download from internal.constellio.me/constellio/dl?id=00000962719, unzip with: tar -xvzf le.fichier.tar.gz.

Procedure

1. After installing portmap and nfs-utils on the source server, run the following commands:

   sudo systemctl enable nfs-server.service rpcbind.service sudo systemctl start nfs-server.service rpcbind.service

2. Open the file /etc/exports using the command sudo nano /etc/exports and add:

   1. /opt/constellio/contents 192.168.1.0/24(rw,sync,no_subtree_check)

      1. Where

         1. The ip address represents the server(s) that will have access to the nfs (DNS possibly accepted too).

         2. rw, write access (RO for read-only).

         3. sync to avoid corruption issues in the event of an NFS shutdown.

         4. no_subtree_check to optimize everything.

1. Run the following commands:

   sudo firewall-cmd --permanent --zone=public --add-service=nfs sudo firewall-cmd --permanent --zone=public --add-service=mountd sudo firewall-cmd --permanent --zone=public --add-service=rpc-bind sudo firewall-cmd --reload

2. On the destination server, mount the NFS disk with:

   1. mount -t nfs 10.0.0.207:/opt/constellio/contents /opt/constellio/contents.nfs

      1. Where 10.0.0.207 represents the address on the source server (DNS accepted too)

3. Finally, copy the files with:

   1. cp -an /opt/constellio/contents.nfs/* /opt/constellio/contents

      1. Where the flag n ignores existing files unless they are newer.

      2. Therefore, the command, which can be interrupted with CTRL+C, allows the resumption thanks to the flag n.

2.5.3 SCP Method (Last Resort)

Prerequisite

• Direct SSH connection between the two servers.

Procedure

1. Make sure you have the necessary rights to write to the destination folder on the target server, usually /opt/constellio/content

   1. Note that you can also copy the source folder to another folder in a different directory. Then, change the permissions of this folder and transfer its contents to the target folder, i.e /opt/constellio/contents. .

   2. Example: constellio@serveur.de.destination $ mkdir -p /opt/constellio/contents.cible

2. Then, from the source server, run the command:

   1. scp -rC /opt/constellio/contents/ constellio@serveur.de.destination:/opt/constellio/contents.cible

      1. This command can be interrupted with CTRL+C, but it will retransfer the files that have already been sent.

3. After the transfer is complete, you can:

   1. Rename a folder (Rename folder to folder2): mv /chemin/vers/le/dossier /chemin/vers/le/dossier2

   2. Move files: mv /chemin/vers/le/dossier/source /chemin/vers/le/dossier/ou/est/la/destination

   3. Change the owner: chown constellio-service:constellio-service -R /chemin/vers/le/dossier

   4. Verify the owner of the files: ls -salut

2.6 Validate the new environment

1. Go to the new Constellio
2. Enter the URL of your Constellio environment in order to arrive at the authentication page. 
3.  Carefully check the compliance of the new environment. That is to say, to check
   1. The structures of your environment (Administrative Units, Classification Plan, Retention Schedule)
   2. Folders, their contents and metadata
   3. Documents, their content and metadata
   4. Simple and advanced search
4. Click "Next".

2.7 Final metadata migration

This step is intended to update the metadata (Solr) in the new environment, in case of changes.

Click "Next".

Duration

It could take several hours to complete this operation. Rely on the status of the task in the background. 

2.8 Final Migration of the Vault

Same as step 2.5. This step can take several hours. Once complete, click "Complete".