Before upgrading OpenStack in your production environment, you may want to try an upgrade on a simple DevStack installation to flush out any issues beforehand. For our next OpenStack upgrade from Havana to Icehouse in production, we will be setting up a separate control plane (with all OpenStack controller services) with the new OpenStack version, upgrading the existing compute nodes in place, and then switching the existing compute nodes to use the new control plane. In comparison to running an in place upgrade of the control plane, switching to a new control plane has these advantages:
- Shorter downtime of the control plane from a user perspective
- Shorter rollback time if issues occur during the upgrade
- Less risk of rollback not working if needed
In our upgrade, we will also upgrade the database in place using the db sync commands. This will be necessary to sync the database tables to the new version in preparation for the new control plane to use the existing database.
Most of the content I've seen previously on the web about OpenStack upgrades addresses the case of upgrading all services in place, but there isn't much out there about switching to a separate control plane.
To test switching to the new control plane with DevStack installs, these will be the high level steps:
- Shut down services
- Take a database backup
- Migrate database
- Bring up new controller services pointed to old database
Prerequisites
Start with working DevStack installs in separate VM's. One DevStack install should be running and tested with the old version of OpenStack, and another DevStack install should be running and tested with the new version of OpenStack. To use an older version than the DevStack master branch, git clone DevStack and then switch to the older branch before running stack.sh. The two VM's will need to be able to access each other on the network. I use VMWare Fusion to run 2 VM's on my local system, but there are other software packages that can be used for virtualization.
Shut Down Services
Shut down the controller and compute services on the two DevStack installs to prevent new data from being written to the database. This can be done by stopping all services in the DevStack screen sessions.
Take a Database Backup
Even though this is just a test with DevStack, it's a good idea to get in the habit of taking backups before modifying data. On the DevStack install running the old OpenStack version, run this command, which is a variation on the backup command from the OpenStack upgrade docs:
# mysqldump -u root -p --opt --add-drop-database --all-databases > openstack-db-backup.sql
The database on the DevStack install running the old OpenStack version is analogous to your existing production database, which the new control plane will end up using by the end of the upgrade. There's no need to take a database backup on the DevStack install running the new OpenStack version, as we won't be modifying that database.
Migrate Database
It's next necessary to migrate the old database to the schema of the new OpenStack version, as the new version of the code may depend on changes to tables and columns. The easiest way to do this while also preparing for the new version of the code to use the existing database is to point the services on the DevStack install with the new OpenStack version to the database on the other DevStack install. You'll need to modify the configurations for each service you plan to test. For the nova service, modify nova.conf:
Change
sql_connection = mysql://root:[my sql password]@127.0.0.1/nova?charset=utf8
To
sql_connection = mysql://root:[my sql password]@[IP address of old DevStack]/nova?charset=utf8
Now that the new version of the code is pointing to the existing database, run the db sync:
$ nova-manage db sync
By default, the nova-manage command will use the sql_connection value in nova.conf for the location of the database to sync.
Repeat this step for each of the conf files for the other services you're interested in upgrading.
Bring up new controller services pointed to old database
In the previous step, the controller services on the new control plane were pointed to the old database. The old database has already synced to work with the new version of the code. Next, bring up the services on the new control plane.
Limitations When Testing this Method DevStack
- As DevStack uses the local file system for the Glance image repository, the Glance images need to be synced over to the VM with the new OpenStack version.
- Note that in this case, we didn't do an in place upgrade of the compute nodes. In production, you may want to do an in place upgrade of the compute nodes to reduce complexity of the entire upgrade.
After syncing the Glance images, you'll now be able to start up a VM using the new control plane that uses the migrated database!