Upgrade to Corda OS 4.6 — The Database Migration

Upgrade to Corda OS 4.6 — The Database Migration

Corda 4.6 — New Features

Last month, R3 released Corda Open Source and Enterprise 4.6, offering exciting new features such as Business Network Membership and Flow Management enhancements. You can read more about it in this post by my colleague, Ante Buterin, or read the release notes.

In this blog post, I’ll be focussing on one of the new operational improvements — Database Schema Harmonisation.

I will show you how to upgrade your Corda Node database schemas from Corda Open Source 4.4 to 4.6.1.

Getting Started

Github

For ease, I’ve create a Github repository containing a start and end code base. I encourage you to follow along from the start, so that you can better understand the steps involved for the upgrade.

I have added sample CorDapp code from the IOU example and added client endpoint to run a loop which will create IOU states in both PartyA and PartyB nodes.

Clone the repository using the following command:

➜ git clone [email protected]:neal-shah/corda-4.6-upgrade.git

Checking dependency versions

Import the cordapp-4.4-start code into IntelliJ IDEA CE and allow Gradle to retrieve the required dependencies.

This is a good time to quickly review the constants.properties file in the root of the project:

cordaReleaseGroup=net.corda
cordaCoreReleaseGroup=net.corda
cordaVersion=4.4
cordaCoreVersion=4.4
gradlePluginsVersion=5.0.12
kotlinVersion=1.2.71
junitVersion=4.12
quasarVersion=0.7.10
log4jVersion=2.11.2
platformVersion=6
slf4jVersion=1.7.25
nettyVersion=4.1.22.Final

Lines 3–4 specify Corda OS version 4.4 and line 10 specifies platform version 6. Read more about Corda platform versions in the documentation.

Custom IOUSchema overview

Take a look at the custom schema that creates the IOU_States table in Corda 4.4:

object IOUSchema

object IOUSchemaV1 : MappedSchema(
        schemaFamily = IOUSchema.javaClass,
        version = 1,
        mappedTypes = listOf(PersistentIOU::class.java)) {

    override val migrationResource: String?
        get() = "iou.changelog-master";

    @Entity
    @Table(name = "iou_states")
    class PersistentIOU(

            @Column(name = "lender")
            var lenderName: String,

            @Column(name = "borrower")
            var borrowerName: String,

            @Column(name = "value")
            var value: Int,

            @Column(name = "linear_id")
            var linearId: UUID

    ) : PersistentState() {
        constructor(): this("", "", 0, UUID.randomUUID())
    }
}

This forms part of the app schema upgrade that you will see later on in this guide.

Deploying and running nodes

In your IntelliJ terminal, run the following command to deploy the Corda development nodes:

➜ ./gradlew deployNodes

You will see three Corda node directories under the ./build/nodes directory:

• Notary
• PartyA
• PartyB

Open a system terminal window (for example, Terminal in MacOS or CMD in Windows), and navigate to the project ./build/nodes directory.

Execute the following command:

➜ ./runnodes

You should see three tabs created, each with one of the nodes starting up. Here’s an example:

Listening for transport dt_socket at address: 5005
Jolokia: Agent started with URL http://127.0.0.1:7005/jolokia/

   ______               __
  / ____/     _________/ /___ _
 / /     __  / ___/ __  / __ `/         
/ /___  /_/ / /  / /_/ / /_/ /          
\____/     /_/   \__,_/\__,_/

--- Corda Open Source 4.4 (21e8c4f) -------------------------------------------------------------

Logs can be found in                    : /path/to/cordapp-4.4-start/build/nodes/Notary/logs
⚠️   ATTENTION: This node is running in development mode! ?‍?   This is not safe for production deployment.
Advertised P2P messaging addresses      : localhost:10022
RPC connection address                  : localhost:10023
RPC admin connection address            : localhost:10043
Loaded 2 CorDapp(s)                     : Contract CorDapp: Template Contracts version 1 by vendor Corda Open Source with licence Apache License, Version 2.0, Workflow CorDapp: Template Flows version 1 by vendor Corda Open Source with licence Apache License, Version 2.0
Node for "Notary" started up and registered in 23.36 sec
Running P2PMessaging loop

Welcome to the Corda interactive shell.
You can see the available commands by typing 'help'.

Wed Nov 18 15:01:57 GMT 2020>>>

Seeding the nodes with data

In the source code of the clients, you will see a very simple GET request which executes a loop:

@GetMapping(value = ["/iou"], produces = ["text/plain"])
    private fun startBatch(): String {
        val otherParty = proxy.wellKnownPartyFromX500Name(CordaX500Name.parse("O=PartyB,L=New York,C=US"))
        for (x in 0..50) {
            val flow = proxy.startFlowDynamic(IOUFlow.Initiator::class.java, x, otherParty)
            println("$x - ${flow.id} - Flow Complete.")
        }
        return "Complete"
    }

To run the server, open the ./clients/build.gradle and run the task runTemplateServer. This will create a Spring Boot server for the PartyA node and successful RPC connection will look like this:

I 15:07:20 1 ServerKt.logStarted - Started ServerKt in 4.679 seconds (JVM running for 5.297)
I 15:07:37 53 [/].log - Initializing Spring FrameworkServlet 'dispatcherServlet'
I 15:07:37 53 DispatcherServlet.initServletBean - FrameworkServlet 'dispatcherServlet': initialization started
I 15:07:37 53 DispatcherServlet.initServletBean - FrameworkServlet 'dispatcherServlet': initialization completed in 16 ms

Next, open up a terminal window and run the following command:

➜ curl http://localhost:10050/iou

The request will return a string: “Complete”

Shutdown the nodes by entering bye in each of the CRaSH shell windows.

Wed Nov 18 15:01:57 GMT 2020>>> bye
Have a good day!

Shutting down ...

Explore the Node Database

Open your database management tool — I use DBeaver. Create a new H2 connection and add the path to the persistence.mv.db with:

• Username set to sa .
• Password left blank.

You cannot have more than one connection at a time to the H2 database so you must ensure that your node has exited successfully.

The custom schema in the CorDapp created a table called IOU_STATES in the database – you can view the data and see the states previously created with the batch endpoint.

Remember to disconnect DBeaver (or your chosen database management tool) from the database to remove the lock on the persistence file.

Starting the Upgrade

Updating the constants file

Set up your CorDapp build for Corda OS 4.6.1:

cordaReleaseGroup=net.corda
cordaCoreReleaseGroup=net.corda
cordaVersion=4.6.1
cordaCoreVersion=4.6.1
gradlePluginsVersion=5.0.12
kotlinVersion=1.2.71
junitVersion=4.12
quasarVersion=0.7.10
log4jVersion=2.11.2
platformVersion=8
slf4jVersion=1.7.25
nettyVersion=4.1.22.Final

Update lines 3–4 with 4.6.1 and line 10 with 8, then refresh your Gradle cache. This will download the dependencies required for 4.6.

If you attempt to run the IOUFlowTests now, you will see that the tests fail with the following error:

Could not create the DataSource: Could not find Liquibase database migration script migration/iou.changelog-master.xml. Please ensure the jar file containing it is deployed in the cordapps directory.
net.corda.nodeapi.internal.persistence.CouldNotCreateDataSourceException: Could not create the DataSource: Could not find Liquibase database migration script migration/iou.changelog-master.xml.
...

Corda 4.6 requires that you specify the Liquibase scripts required to carry out schema migrations going forward.

Add Liquibase scripts

Under contracts/src/main add a resources/migration directory – this directory will hold the XML scripts to carry out any custom migrations that you need.

Two database changes need to be made, which involves the addition of three Liquibase XML scripts:

1. A master XML script which stores the run order of the database changeset files — iou.changelog-master.xml
2. The first changeset file containing the current IOU_STATES table schema — iou.changelog-v1.xml
3. A second changeset file containing a modification type update to the linear_id column in the IOU_STATES table — iou.changelog-v2.xml

Create a new master script under migration called iou.changelog-master.xml and add the follow:

<?xml version="1.1" encoding="UTF-8" standalone="no"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
                   xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext"
                   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.5.xsd">

    <include file="migration/iou.changelog-v1.xml"/>
    <include file="migration/iou.changelog-v2.xml"/>
</databaseChangeLog>

Add another file under migration called iou.changelog-v1.xml and add the following:

<?xml version="1.1" encoding="UTF-8" standalone="no"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
                   xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext"
                   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.5.xsd">
    <changeSet author="NodeOperator" id="create_iou_state">
        <preConditions onFail="MARK_RAN">
            <not>
                <tableExists tableName="iou_states"/>
            </not>
        </preConditions>
        <createTable tableName="iou_states">
            <column name="output_index" type="INT"/>
            <column name="transaction_id" type="NVARCHAR(64)"/>
            <column name="value" type="int"/>
            <column name="lender" type="NVARCHAR(64)"/>
            <column name="borrower" type="NVARCHAR(64)"/>
            <column name="linear_id" type="NVARCHAR(64)"/>
        </createTable>
    </changeSet>
</databaseChangeLog>

The IOU_STATES table also needs to be updated – the linear_id column type needs to be modified from VARBINARY to VARCHAR.

Create another XML file called iou.changelog-v2.xml with the following contents:

<?xml version="1.1" encoding="UTF-8" standalone="no"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
                   xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext"
                   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.5.xsd">
    <changeSet author="NodeOperator" id="Change linear_id Column Type" >
        <modifyDataType tableName="iou_states" columnName="linear_id" newDataType="NVARCHAR(64)"/>
    </changeSet>
</databaseChangeLog>

The master xml is required as it keeps the changelog in the correct order, whilst the iou.changelog-v1.xml file contains the changeset to be actioned.

Add a Type annotation to the IOUSchema schema linear_id column (line 36):

object IOUSchema

object IOUSchemaV1 : MappedSchema(
        schemaFamily = IOUSchema.javaClass,
        version = 1,
        mappedTypes = listOf(PersistentIOU::class.java)) {

    override val migrationResource: String?
        get() = "iou.changelog-master";

    @Entity
    @Table(name = "iou_states")
    class PersistentIOU(

            @Column(name = "lender")
            var lenderName: String,

            @Column(name = "borrower")
            var borrowerName: String,

            @Column(name = "value")
            var value: Int,

            @Column(name = "linear_id")
            @Type(type = "uuid-char")
            var linearId: UUID

    ) : PersistentState() {
        // Default constructor required by hibernate.
        constructor(): this("", "", 0, UUID.randomUUID())
    }
}

Run the IOUFlowTests again, and they should now pass!

Upgrading the Real Nodes

Earlier, you deployed nodes and populated IOU states into the node databases using the server endpoint. The nodes should not be running currently. You can now replace the CorDapp contracts and Corda runtime jar files in the nodes.

Replace the contracts CorDapp

First, build the contracts code.

Once built, navigate to ./contracts/build/libs and copy (and overwrite) the contracts-0.1.jar file to both the PartyA/cordapps and PartyB/cordapps directories. The reason for this is that the Liquibase scripts are bundled into the contracts jar file, so they will be discovered when you run the database migrations.

Replace the corda.jar

Download Corda OS 4.6, and replace the corda.jar file in both the PartyA and PartyB node directories, which can be found under the ./build/nodes directory.

In a new terminal window, navigate to the PartyA directory:

➜ cd /path/to/build/nodes/PartyA

The following steps apply to both PartyA and PartyB nodes.

Begin by running the core schema migrations. These are the Corda core tables which may require changes to be carried out.

➜ PartyA ✗ java -jar corda.jar run-migration-scripts --core-schemas

   ______               __
  / ____/     _________/ /___ _
 / /     __  / ___/ __  / __ `/         
/ /___  /_/ / /  / /_/ / /_/ /          
\____/     /_/   \__,_/\__,_/

--- Corda Open Source 4.6 (85e387e) -------------------------------------------------------------

Logs can be found in                    : /path/to/cordapp-4.4-start/build/nodes/PartyA/logs
Running database schema migration scripts ...
Database migration scripts for core schemas complete. There are 1 outstanding app database changes.

You can see that the core schema migrations complete successfully, however there is a single app schema migration outstanding. This is the schema “upgrade” where you specified the xml file for the IOU_STATES table – in reality, this script will not action any changes on the IOU_STATES table. Instead, it will add an entry to the databasechangelog table to mark that the script has been run.

Now run the app schema “upgrade”:

➜  PartyA ✗ java -jar corda.jar run-migration-scripts --app-schemas

   ______               __
  / ____/     _________/ /___ _
 / /     __  / ___/ __  / __ `/         
/ /___  /_/ / /  / /_/ / /_/ /          
\____/     /_/   \__,_/\__,_/

--- Corda Open Source 4.6 (85e387e) -------------------------------------------------------------

Logs can be found in                    : /path/to/cordapp-4.4-start/build/nodes/PartyA/logs
Running database schema migration scripts ...
Database migration scripts for app schemas complete. There are no outstanding database changes.

Carry out the same upgrade steps for PartyB.

Before starting the nodes, connect to the PartyA database from your database management tool and run the following query:

SELECT ID, FILENAME, DATEEXECUTED, EXECTYPE
FROM DATABASECHANGELOG d
WHERE d.ID = 'Create IOU State' OR d.ID = 'Change linear_id Column Type'

You will see that the migration completed successfully as the result is:

Disconnect from the database.

Starting your Corda 4.6 nodes

Open a new terminal window and run the following from the ./corda-4.4-start/ directory:

➜ ./build/nodes/runnodes

Your nodes should now start with Corda OS 4.6! ?

   ______               __
  / ____/     _________/ /___ _
 / /     __  / ___/ __  / __ `/         
/ /___  /_/ / /  / /_/ / /_/ /          
\____/     /_/   \__,_/\__,_/

--- Corda Open Source 4.6.1 (7962aad) -------------------------------------------------------------

Logs can be found in                    : /path/to/cordapp-4.4-start/build/nodes/PartyA/logs
⚠️   ATTENTION: This node is running in development mode! ?‍?   This is not safe for production deployment.
Advertised P2P messaging addresses      : localhost:10005
RPC connection address                  : localhost:10006
RPC admin connection address            : localhost:10046
Loaded 2 CorDapp(s)                     : Contract CorDapp: Template Contracts version 1 by vendor Corda Open Source with licence Apache License, Version 2.0, Workflow CorDapp: Template Flows version 1 by vendor Corda Open Source with licence Apache License, Version 2.0
Node for "PartyA" started up and registered in 11.41 sec
Running P2PMessaging loop

Welcome to the Corda interactive shell.
You can see the available commands by typing 'help'.

Wed Nov 18 17:03:05 GMT 2020>>>

Smoke Test with the server

Start the web server and run the cURL command below:

➜ curl http://localhost:10050/iou

You can observe the node processing flows by running the CRaSH shell RPC command flow watch.

A Note about Draining Flows

When upgrading your node, it’s important to consider suspended (or checkpointed) flows in progress. You will not be able to upgrade your node without allowing the node to complete all flows that are currently outstanding. You can read more about it here.

Recap

In this post, I’ve shown you how to upgrade the database core schema and app schema from Corda OS 4.4 to Corda OS 4.6.1, using the new method introduced in Corda 4.6.

Managing database upgrades explicitly via commands, rather than using node configuration allows for a granular approach to carrying out the task and removes any issues with bad configuration. With Corda Enterprise, you can use the Database Management Tool, which provides you with greater flexibility and control in schema migration.

Want to learn more about building awesome blockchain applications on Corda? Be sure to visit https://corda.net, check out our community page to learn how to connect with other Corda developers, and sign up for one of our newsletters for the latest updates.

— Neal Shah is a Solutions Engineer at R3, an enterprise blockchain software firm working with a global ecosystem of more than 350 participants across multiple industries from both the private and public sectors to develop on Corda, its open-source blockchain platform, and Corda Enterprise, a commercial version of Corda for enterprise usage.

Follow Neal on LinkedIn and Instagram.