Upgrade to Corda OS 4.6 — The Database Migration
November 24, 2020
Upgrade to Corda OS 4.6 — The Database Migration
Corda 4.6 — New Features
Last month, R3 released Corda Open Source and Enterprise 4.6, offering exciting new features such as Business Network Membership and Flow Management enhancements. You can read more about it in this post by my colleague, Ante Buterin, or read the release notes.
In this blog post, I’ll be focussing on one of the new operational improvements — Database Schema Harmonisation.
I will show you how to upgrade your Corda Node database schemas from Corda Open Source 4.4 to 4.6.1.
Getting Started
Github
For ease, I’ve create a Github repository containing a start and end code base. I encourage you to follow along from the start, so that you can better understand the steps involved for the upgrade.
I have added sample CorDapp code from the IOU example and added client endpoint to run a loop which will create IOU states in both PartyA
and PartyB
nodes.
Clone the repository using the following command:
➜ git clone [email protected]:neal-shah/corda-4.6-upgrade.git
Checking dependency versions
Import the cordapp-4.4-start
code into IntelliJ IDEA CE and allow Gradle to retrieve the required dependencies.
This is a good time to quickly review the constants.properties
file in the root of the project:
cordaReleaseGroup=net.corda
cordaCoreReleaseGroup=net.corda
cordaVersion=4.4
cordaCoreVersion=4.4
gradlePluginsVersion=5.0.12
kotlinVersion=1.2.71
junitVersion=4.12
quasarVersion=0.7.10
log4jVersion=2.11.2
platformVersion=6
slf4jVersion=1.7.25
nettyVersion=4.1.22.Final
Lines 3–4 specify Corda OS version 4.4 and line 10 specifies platform version 6. Read more about Corda platform versions in the documentation.
Custom IOUSchema overview
Take a look at the custom schema that creates the IOU_States
table in Corda 4.4:
object IOUSchema
object IOUSchemaV1 : MappedSchema(
schemaFamily = IOUSchema.javaClass,
version = 1,
mappedTypes = listOf(PersistentIOU::class.java)) {
override val migrationResource: String?
get() = "iou.changelog-master";
@Entity
@Table(name = "iou_states")
class PersistentIOU(
@Column(name = "lender")
var lenderName: String,
@Column(name = "borrower")
var borrowerName: String,
@Column(name = "value")
var value: Int,
@Column(name = "linear_id")
var linearId: UUID
) : PersistentState() {
constructor(): this("", "", 0, UUID.randomUUID())
}
}
This forms part of the app schema upgrade that you will see later on in this guide.
Deploying and running nodes
In your IntelliJ terminal, run the following command to deploy the Corda development nodes:
➜ ./gradlew deployNodes
You will see three Corda node directories under the ./build/nodes
directory:
- Notary
- PartyA
- PartyB
Open a system terminal window (for example, Terminal in MacOS or CMD in Windows), and navigate to the project ./build/nodes
directory.
Execute the following command:
➜ ./runnodes
You should see three tabs created, each with one of the nodes starting up. Here’s an example:
Listening for transport dt_socket at address: 5005
Jolokia: Agent started with URL http://127.0.0.1:7005/jolokia/
______ __
/ ____/ _________/ /___ _
/ / __ / ___/ __ / __ `/
/ /___ /_/ / / / /_/ / /_/ /
\____/ /_/ \__,_/\__,_/
--- Corda Open Source 4.4 (21e8c4f) -------------------------------------------------------------
Logs can be found in : /path/to/cordapp-4.4-start/build/nodes/Notary/logs
⚠️ ATTENTION: This node is running in development mode! ?? This is not safe for production deployment.
Advertised P2P messaging addresses : localhost:10022
RPC connection address : localhost:10023
RPC admin connection address : localhost:10043
Loaded 2 CorDapp(s) : Contract CorDapp: Template Contracts version 1 by vendor Corda Open Source with licence Apache License, Version 2.0, Workflow CorDapp: Template Flows version 1 by vendor Corda Open Source with licence Apache License, Version 2.0
Node for "Notary" started up and registered in 23.36 sec
Running P2PMessaging loop
Welcome to the Corda interactive shell.
You can see the available commands by typing 'help'.
Wed Nov 18 15:01:57 GMT 2020>>>
Seeding the nodes with data
In the source code of the clients
, you will see a very simple GET
request which executes a loop:
@GetMapping(value = ["/iou"], produces = ["text/plain"])
private fun startBatch(): String {
val otherParty = proxy.wellKnownPartyFromX500Name(CordaX500Name.parse("O=PartyB,L=New York,C=US"))
for (x in 0..50) {
val flow = proxy.startFlowDynamic(IOUFlow.Initiator::class.java, x, otherParty)
println("$x - ${flow.id} - Flow Complete.")
}
return "Complete"
}
To run the server, open the ./clients/build.gradle
and run the task runTemplateServer
. This will create a Spring Boot server for the PartyA
node and successful RPC connection will look like this:
I 15:07:20 1 ServerKt.logStarted - Started ServerKt in 4.679 seconds (JVM running for 5.297)
I 15:07:37 53 [/].log - Initializing Spring FrameworkServlet 'dispatcherServlet'
I 15:07:37 53 DispatcherServlet.initServletBean - FrameworkServlet 'dispatcherServlet': initialization started
I 15:07:37 53 DispatcherServlet.initServletBean - FrameworkServlet 'dispatcherServlet': initialization completed in 16 ms
Next, open up a terminal window and run the following command:
➜ curl http://localhost:10050/iou
The request will return a string: “Complete”
Shutdown the nodes by entering bye
in each of the CRaSH shell windows.
Wed Nov 18 15:01:57 GMT 2020>>> bye
Have a good day!
Shutting down ...
Explore the Node Database
Open your database management tool — I use DBeaver. Create a new H2 connection and add the path to the persistence.mv.db
with:
- Username set to
sa
. - Password left blank.
You cannot have more than one connection at a time to the H2 database so you must ensure that your node has exited successfully.
The custom schema in the CorDapp created a table called IOU_STATES
in the database – you can view the data and see the states previously created with the batch endpoint.
Remember to disconnect DBeaver (or your chosen database management tool) from the database to remove the lock on the persistence file.
Starting the Upgrade
Updating the constants
file
Set up your CorDapp build for Corda OS 4.6.1:
cordaReleaseGroup=net.corda
cordaCoreReleaseGroup=net.corda
cordaVersion=4.6.1
cordaCoreVersion=4.6.1
gradlePluginsVersion=5.0.12
kotlinVersion=1.2.71
junitVersion=4.12
quasarVersion=0.7.10
log4jVersion=2.11.2
platformVersion=8
slf4jVersion=1.7.25
nettyVersion=4.1.22.Final
Update lines 3–4 with 4.6.1
and line 10 with 8
, then refresh your Gradle cache. This will download the dependencies required for 4.6.
If you attempt to run the IOUFlowTests
now, you will see that the tests fail with the following error:
Could not create the DataSource: Could not find Liquibase database migration script migration/iou.changelog-master.xml. Please ensure the jar file containing it is deployed in the cordapps directory.
net.corda.nodeapi.internal.persistence.CouldNotCreateDataSourceException: Could not create the DataSource: Could not find Liquibase database migration script migration/iou.changelog-master.xml.
...
Corda 4.6 requires that you specify the Liquibase scripts required to carry out schema migrations going forward.
Add Liquibase scripts
Under contracts/src/main
add a resources/migration
directory – this directory will hold the XML scripts to carry out any custom migrations that you need.
Two database changes need to be made, which involves the addition of three Liquibase XML scripts:
- A master XML script which stores the run order of the database changeset files —
iou.changelog-master.xml
- The first changeset file containing the current
IOU_STATES
table schema —iou.changelog-v1.xml
- A second changeset file containing a modification type update to the
linear_id
column in theIOU_STATES
table —iou.changelog-v2.xml
Create a new master script under migration
called iou.changelog-master.xml
and add the follow:
<?xml version="1.1" encoding="UTF-8" standalone="no"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.5.xsd">
<include file="migration/iou.changelog-v1.xml"/>
<include file="migration/iou.changelog-v2.xml"/>
</databaseChangeLog>
Add another file under migration
called iou.changelog-v1.xml
and add the following:
<?xml version="1.1" encoding="UTF-8" standalone="no"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.5.xsd">
<changeSet author="NodeOperator" id="create_iou_state">
<preConditions onFail="MARK_RAN">
<not>
<tableExists tableName="iou_states"/>
</not>
</preConditions>
<createTable tableName="iou_states">
<column name="output_index" type="INT"/>
<column name="transaction_id" type="NVARCHAR(64)"/>
<column name="value" type="int"/>
<column name="lender" type="NVARCHAR(64)"/>
<column name="borrower" type="NVARCHAR(64)"/>
<column name="linear_id" type="NVARCHAR(64)"/>
</createTable>
</changeSet>
</databaseChangeLog>
The IOU_STATES
table also needs to be updated – the linear_id
column type needs to be modified from VARBINARY to VARCHAR.
Create another XML file called iou.changelog-v2.xml
with the following contents:
<?xml version="1.1" encoding="UTF-8" standalone="no"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.5.xsd">
<changeSet author="NodeOperator" id="Change linear_id Column Type" >
<modifyDataType tableName="iou_states" columnName="linear_id" newDataType="NVARCHAR(64)"/>
</changeSet>
</databaseChangeLog>
The master xml is required as it keeps the changelog in the correct order, whilst the iou.changelog-v1.xml
file contains the changeset to be actioned.
Add a Type
annotation to the IOUSchema
schema linear_id
column (line 36):
object IOUSchema
object IOUSchemaV1 : MappedSchema(
schemaFamily = IOUSchema.javaClass,
version = 1,
mappedTypes = listOf(PersistentIOU::class.java)) {
override val migrationResource: String?
get() = "iou.changelog-master";
@Entity
@Table(name = "iou_states")
class PersistentIOU(
@Column(name = "lender")
var lenderName: String,
@Column(name = "borrower")
var borrowerName: String,
@Column(name = "value")
var value: Int,
@Column(name = "linear_id")
@Type(type = "uuid-char")
var linearId: UUID
) : PersistentState() {
// Default constructor required by hibernate.
constructor(): this("", "", 0, UUID.randomUUID())
}
}
Run the IOUFlowTests
again, and they should now pass!
Upgrading the Real Nodes
Earlier, you deployed nodes and populated IOU states into the node databases using the server endpoint. The nodes should not be running currently. You can now replace the CorDapp contracts and Corda runtime jar files in the nodes.
Replace the contracts
CorDapp
First, build the contracts
code.
Once built, navigate to ./contracts/build/libs
and copy (and overwrite) the contracts-0.1.jar
file to both the PartyA/cordapps
and PartyB/cordapps
directories. The reason for this is that the Liquibase scripts are bundled into the contracts jar file, so they will be discovered when you run the database migrations.
Replace the corda.jar
Download Corda OS 4.6, and replace the corda.jar
file in both the PartyA
and PartyB
node directories, which can be found under the ./build/nodes
directory.
In a new terminal window, navigate to the PartyA
directory:
➜ cd /path/to/build/nodes/PartyA
The following steps apply to both
PartyA
andPartyB
nodes.
Begin by running the core schema migrations. These are the Corda core tables which may require changes to be carried out.
➜ PartyA ✗ java -jar corda.jar run-migration-scripts --core-schemas
______ __
/ ____/ _________/ /___ _
/ / __ / ___/ __ / __ `/
/ /___ /_/ / / / /_/ / /_/ /
\____/ /_/ \__,_/\__,_/
--- Corda Open Source 4.6 (85e387e) -------------------------------------------------------------
Logs can be found in : /path/to/cordapp-4.4-start/build/nodes/PartyA/logs
Running database schema migration scripts ...
Database migration scripts for core schemas complete. There are 1 outstanding app database changes.
You can see that the core schema migrations complete successfully, however there is a single app schema migration outstanding. This is the schema “upgrade” where you specified the xml file for the IOU_STATES
table – in reality, this script will not action any changes on the IOU_STATES
table. Instead, it will add an entry to the databasechangelog
table to mark that the script has been run.
Now run the app schema “upgrade”:
➜ PartyA ✗ java -jar corda.jar run-migration-scripts --app-schemas
______ __
/ ____/ _________/ /___ _
/ / __ / ___/ __ / __ `/
/ /___ /_/ / / / /_/ / /_/ /
\____/ /_/ \__,_/\__,_/
--- Corda Open Source 4.6 (85e387e) -------------------------------------------------------------
Logs can be found in : /path/to/cordapp-4.4-start/build/nodes/PartyA/logs
Running database schema migration scripts ...
Database migration scripts for app schemas complete. There are no outstanding database changes.
Carry out the same upgrade steps for
PartyB
.
Before starting the nodes, connect to the PartyA
database from your database management tool and run the following query:
SELECT ID, FILENAME, DATEEXECUTED, EXECTYPE
FROM DATABASECHANGELOG d
WHERE d.ID = 'Create IOU State' OR d.ID = 'Change linear_id Column Type'
You will see that the migration completed successfully as the result is:
Disconnect from the database.
Starting your Corda 4.6 nodes
Open a new terminal window and run the following from the ./corda-4.4-start/
directory:
➜ ./build/nodes/runnodes
Your nodes should now start with Corda OS 4.6! ?
______ __
/ ____/ _________/ /___ _
/ / __ / ___/ __ / __ `/
/ /___ /_/ / / / /_/ / /_/ /
\____/ /_/ \__,_/\__,_/
--- Corda Open Source 4.6.1 (7962aad) -------------------------------------------------------------
Logs can be found in : /path/to/cordapp-4.4-start/build/nodes/PartyA/logs
⚠️ ATTENTION: This node is running in development mode! ?? This is not safe for production deployment.
Advertised P2P messaging addresses : localhost:10005
RPC connection address : localhost:10006
RPC admin connection address : localhost:10046
Loaded 2 CorDapp(s) : Contract CorDapp: Template Contracts version 1 by vendor Corda Open Source with licence Apache License, Version 2.0, Workflow CorDapp: Template Flows version 1 by vendor Corda Open Source with licence Apache License, Version 2.0
Node for "PartyA" started up and registered in 11.41 sec
Running P2PMessaging loop
Welcome to the Corda interactive shell.
You can see the available commands by typing 'help'.
Wed Nov 18 17:03:05 GMT 2020>>>
Smoke Test with the server
Start the web server and run the cURL
command below:
➜ curl http://localhost:10050/iou
You can observe the node processing flows by running the CRaSH shell RPC command flow watch
.
A Note about Draining Flows
When upgrading your node, it’s important to consider suspended (or checkpointed) flows in progress. You will not be able to upgrade your node without allowing the node to complete all flows that are currently outstanding. You can read more about it here.
Recap
In this post, I’ve shown you how to upgrade the database core schema and app schema from Corda OS 4.4 to Corda OS 4.6.1, using the new method introduced in Corda 4.6.
Managing database upgrades explicitly via commands, rather than using node configuration allows for a granular approach to carrying out the task and removes any issues with bad configuration. With Corda Enterprise, you can use the Database Management Tool, which provides you with greater flexibility and control in schema migration.
Want to learn more about building awesome blockchain applications on Corda? Be sure to visit https://corda.net, check out our community page to learn how to connect with other Corda developers, and sign up for one of our newsletters for the latest updates.
— Neal Shah is a Solutions Engineer at R3, an enterprise blockchain software firm working with a global ecosystem of more than 350 participants across multiple industries from both the private and public sectors to develop on Corda, its open-source blockchain platform, and Corda Enterprise, a commercial version of Corda for enterprise usage.