Unison: A Datastore migration library for Google Go

Unison is a lightweight library that allows you to version and manage your Datastore migrations. In this blog article we will have a look at how you can install the Unison command line tool and get started migrating entities to Datastore.

Note: At the time of writing this article the library was not compatible with Google Firestore.

Before we begin, the article assumes that you have Google Go installed on your computer. In case you do not have it installed, click here to grab an installer for your operating system.

The go-unison project comprises two packages, viz, unison and unisoner. The first is the library that you will use to migrate your scripts, while the latter is a command line tool that lets you generate new migration scripts.

Let’s start with creating a migration script that migrates some delicious fruits to Datastore. Run the following commands to set up your project.

1
2
3
4
5
6
7

$ mkdir goapp
$ cd goapp
goapp $ go mod init goapp
goapp $ go get cloud.google.com/go/datastore
goapp $ mkdir ent
goapp $ touch ent/fruit.go
goapp $ touch main.go

The ent/fruit.go file should contain the serializable Datastore entity structure as follows.

1
2
3
4
5
6

package ent

type Fruit struct {
    ID   string `datastore:"id"`
    Name string `datastore:"Name"`
}

Now let’s go ahead and install the unisoner command line tool and the unison library.

1
2

goapp $ go get github.com/utsavgupta/go-unison/unisoner
goapp $ go get github.com/utsavgupta/go-unison/unison

We are now ready to create new migration files. When you execute the unisoner command it creates the following to bootstrap unison.

• Migration package: A new package is created in the present working directory. By default the name of this package is migration. But this can be overridden by either setting the environment variable unison_migration_package or by passing --migration_package <pkg_name> while executing the command. Note: if values are received from both, environment variable and the command line param, the latter will be given precedence.
• Unison migrations type: A new type gets created within the above package. It is on this type new migrations will be defined. Finally an instance of this type needs to be passed to the unison library for it to work its magic.

Now with the details out of the way let’s create our first migration script.

1
2
3

goapp $ unisoner --migration_package gcp
Filename [.go extension is automatically appended] (default -> Apply1584396052): fruits
Description: add fruits

Let’s examine the files that were generated by unison.

1
2

goapp $ ls gcp/
fruits.go   unison.go

The file unison.go contains the migrations type that has been explained above. The other file is where the migration script goes.

Open gcp/fruits.go in a text editor and the contents should look like the following.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

package gcp

import (
    "cloud.google.com/go/datastore"
)

// Apply1584396052 add fruits
func (u *UnisonMigrations) Apply1584396052(t *datastore.Transaction, ns string) error {

    return nil
}

Each migration file we generate should ideally contain one method defined on UnisonMigrations. These method names are based on the following naming convention Apply<Timestamp>. It is based on this timestamp that the unison library sorts the order of migrations. A migration once successfully commited will not be run again. Only migrations that have a timestamp greater than the last successully executed migration will be executed.

Since I love eating apples and mangoes, I will migrate these two fruits to Datastore. Let’s make the following changes to gcp/fruits.go (you can use your favorites here 😉).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

package gcp

import (
    "goapp/ent"

    "cloud.google.com/go/datastore"
)

// Apply1584396052 add fruits
func (u *UnisonMigrations) Apply1584396052(t *datastore.Transaction, ns string) error {

    fruits := []ent.Fruit{
		ent.Fruit{ID: "apple", Name: "Apple"},
		ent.Fruit{ID: "mango", Name: "Mango"},
	}

	keys := make([]*datastore.Key, len(fruits))

	for idx, fruit := range fruits {
		keys[idx] = datastore.NameKey("Fruits", fruit.ID, nil)
	    keys[idx].Namespace = ns
	}

	_, err := t.PutMulti(keys, fruits)

	return err
}

Great! Our first migration script is ready. Now all that remains is to use the unison library to migrate this script. For that let’s make changes to main.go.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

package main

import (
    "context"

    "goapp/gcp"

    "cloud.google.com/go/datastore"
    "github.com/utsavgupta/go-unison/unison"
)

const (
	namespace = "unison-demo"
)

func main() {

	dsClient, err := datastore.NewClient(context.Background(), "*detect-project-id*")

	if err != nil {
		panic(err)
	}

	var unisonMigrations gcp.UnisonMigrations

	unison.RunMigrations(dsClient, namespace, &unisonMigrations)
}

We are a step away from migrating our first entities to Datastore. Before running the code make sure you have Google application credentials set. More on it here.

After you have set the variable you will be ready to run your shiny new migration script.

1
2
3
4

goapp $ go run .
Running Unison
Applying migration 1584477480 ... Done
We are done !!

Yay ! We are done migrating our first migration script. Head over to the Google Cloud console to verify the changes.

Note that the migration records themselves are stored as entites of kind UnisonMigrationMeta.

Now with the first migration script done, let’s create a new script to migrate a few more fruits.

1
2
3

goapp $ unisoner --migration_package gcp
Filename [.go extension is automatically appended] (default -> Apply1584126043): more_fruits
Description: add more fruits

Edit the newly created gcp/more_fruits.go file to migrate a few more fruits.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

package gcp

import (
    "goapp/ent"

    "cloud.google.com/go/datastore"
)

// Apply1584126043 add fruits
func (u *UnisonMigrations) Apply1584126043(t *datastore.Transaction, ns string) error {

    fruits := []ent.Fruit{
		ent.Fruit{ID: "banana", Name: "Banana"},
		ent.Fruit{ID: "orange", Name: "Orange"},
		ent.Fruit{ID: "watermelon", Name: "Watermelon"},
	}

	keys := make([]*datastore.Key, len(fruits))

	for idx, fruit := range fruits {
		keys[idx] = datastore.NameKey("Fruits", fruit.ID, nil)
	    keys[idx].Namespace = ns
	}

	_, err := t.PutMulti(keys, fruits)

	return err
}

Running go run . again should migrate only the new migration script.

1
2
3
4

goapp $ go run .
Running Unison
Applying migration 1584126043 ... Done
We are done !!

Please note: The project is in it’s early stages. Contributions in terms of bug reports, documentation, and testing are welcome. Do not hesitate to report issues or to raise pull requests.