Go to main contentGo to footer
Code
|
17 January 22

How to write migration scripts to change your content schema in DatoCMS

In this tutorial, we'll look at how to safely change the content schema of a DatoCMS project without affecting production, with maximum flexibility for testing before the merge.

Natalia BaezaDeveloper

Dato Environments

You've built a website or app that uses DatoCMS to manage content, and everything runs perfectly in production. But now you need to change the content structure. How do you do it safely, without affecting what's in production?

Today we'll look at how to use DatoCMS sandbox environments and migration scripts to do exactly that.

Every DatoCMS project has a primary environment, which stores all of the project's production data, including models, records, uploads, SEO settings, locale and time zone settings, as well as Dato dashboard settings, plugins, the UI theme and the content navigation bar. The primary environment is where editors manage the project's content.

A project can also have any number of sandbox environments. These always start as exact copies (or forks) of the primary environment and are meant for testing and development. Sandbox environments let developers change the project structure without interfering with what's in production. You can think of sandbox environments as something similar to working with git branches.

The primary environment can be forked directly from the DatoCMS dashboard or from the command line.

Here's how to do it from the dashboard (we'll come back to the Dato CLI further down):

Go to Settings > Environments and then click Fork.

You'll be asked to name the copy of the primary environment you're creating; the new sandbox environment will be identified by that name. Once the environment is created, you can switch between environments with a click of a button at the top right of the dashboard:

You can set the endpoint in your code to fetch data from a sandbox environment by referencing it by name. If no name is specified, data is fetched from the primary environment. The DatoCMS documentation explains several ways to do this. Here you'll find instructions for fetching data specifically in a Next.js project.

Environments are especially useful for changing the content structure without interfering with production.

Here is the workflow recommended by DatoCMS:

  1. Create a new sandbox environment by forking the primary environment (with the fork button in the dashboard or from the command line).
  2. Work exclusively inside this environment, making changes with migration scripts rather than through the UI.
  3. When you're ready, turn on maintenance mode (with the button in the dashboard or from the command line). Editors won't be able to add new content to the project.
  4. Run your migration scripts, which first create an up-to-date copy of the primary environment and then modify it according to the scripts.
  5. Deploy a new version of the website or app pointing to the new sandbox and check that everything works as expected.
  6. Promote the new sandbox environment to primary (with a button in the dashboard or from the command line).
  7. The old primary environment will now be a regular sandbox environment and can serve as a backup if needed.
  8. Turn off maintenance mode.

Installing the CLI

The first thing we need to do is install the DatoCMS command line interface. If you use npm, you can run the following command:

npm install -g datocms-client

If you use yarn, run:

yarn global add datocms-client

To run migrations, we also need to set up the full-access API token, which you'll find in the DatoCMS dashboard under Settings > Permissions > API tokens. There are several ways to set the token, either directly on the command line or by loading environment variables from a .env file; the options are described in detail here.

Now we can use the client and all its functions. If you run 

dato

on the command line, you'll see a list of all available commands and the flags each of them accepts (in the screenshot below I've cropped some of the flags, but you can see all the commands currently availablehttps://www.datocms.com/marketplace/starters/nextjs-template-bloghttps://www.datocms.com/marketplace/starters/nextjs-template-blog):

The project

In this tutorial, we'll work on a simple blog project based on this Next.js template. We've changed some of the content, but the structure is the same. The project contains blog posts, author records, categories and a home page model.

We'll focus on the Author model, which is structured like this:

Here's what a single author record looks like:

We'll create two new blocks called Contact and Avatar. The Contact block will look like this:

And this is the Avatar block:

Once both blocks are created, we'll add a new modular content field called Profile to the Author model. The Profile field will contain records created with the Avatar or Contact blocks.

Here's what the author model will look like at the end of the tutorial:

And here's what a complete author record will look like:

The documentation on writing migration scripts is detailed and thorough. For our purposes, we'll follow the steps shown in this example to create a modular content field.

Adapted to our needs, these are the steps we have to follow:

  1. Create a contact block.
  2. Create string fields.
  3. Add the modular content field to the author model and set it to accept the block we created.

After completing steps 1 to 3, we'll create a block with an image field and add it to the modular content field created in step 3.

Finally, we'll see how to add content to a record, both by migrating existing content and by passing new content directly in a script.

Creating a contact block

Let's start by creating a new migration file with the command dato new migration followed by the name we want to give our script. The first time we do this in our project, a new migrations directory will be created in the root of the project, with the new file inside it.

The file name starts with a timestamp, followed by the name we passed to the command.

dato new migration 'Create contact block'

Now we can go to the migrations directory and find our new file inside. Like all new migration files, it comes prefilled with sample code; we can go ahead and delete everything inside the module.exports function, leaving us with this:

'use strict';

module.exports = async (client) => {
}

Now we're ready to look at the details of the script we need to write to create a new block. The documentation for the DatoCMS Content Management API shows us exactly how to do it, using the same code used to create a new model but passing modularBlock: true.

So, to create our Contact block, we need to add it inside the module.exports function:

  const contactBlock = await client.itemTypes.create({
    name: 'Contact',
    apiKey: 'contact_block',
    modularBlock: true,
  });

To run the migration, we use the command dato migrate followed by the flag --destination= and the name of the sandbox environment that will be created to run our migration.

In this tutorial, we'll call our sandbox environment Example.

dato migrate --destination=example

In the console, we can see the sandbox environment being forked and the migration running.

Done! Now we can go to our DatoCMS dashboard and switch to the newly created Example environment. If we click on the Blocks Library, we'll see the Contact block we just created.

The new block is currently empty, so we need to create fields inside it.

Adding a field to the block

As always, we start by creating a new migration file.

dato new migration 'Add fields to contact block'

To create a new field in the Contact block, we need to pass the block's id and an object with the properties label, api key, field type, validators and appearance to the function client.fields.create.

In general, when creating new fields, you should go to the documentation and scroll through the list until you find the field type you want to create. There you'll find specific instructions on the values to pass for the field type, as well as the available validators and what to pass for the editor in the property appearance.

In this case, we'll create a single-line string field called "E-mail":

'use strict';

module.exports = async (client) => {
  const contactBlock = await client.itemType.find('contact_block');

  const emailField = await client.fields.create(contactBlock.id, {
    label: 'E-mail',
    apiKey: 'email',
    fieldType: 'string',
    validators: {
      required: {},
      format: {
        predefined_pattern: "email",
      },
    },
    appearance: {
      editor: 'single_line',
      parameters: {
        heading: false,
      },
      addons: [],
    },
  });
}

Now we're ready to run the migration with the command dato migrate, but instead of the --destination flag, which creates new environments, we'll pass the flags --source and --inPlace to tell Dato to run the new migration in the existing environment.

dato migrate --source=example --inPlace

When the migration is done…

We can go back to the dashboard to see the final result:

Adding a modular content field to the Author model

Now that our Contacts block is ready, we can use it to add a modular content field to the Author model, which will accept contact blocks.

As usual, we start by creating a migration file.

dato new migration 'Add profile blocks to author model'

To clear up any doubts, we can always check the documentation on how to create a new field. In the "Available field types" list, if we click on modular content we'll see the following:

In the "Validators" list, if we click on rich_text_blocks we'll see this instead:

With that in mind, we're ready to write a script that adds a Profile field to the author model. As you can see below, we'll use the function client.fields.create and pass the author model's id and an object with all the data needed to create the new modular content field.

'use strict';

module.exports = async (client) => {
  const contactBlock = await client.itemType.find('contact_block');
  const authorModel = await client.itemType.find('author');

  const profileBlockField = await client.fields.create(authorModel.id, {
    label: 'Profile',
    apiKey: 'profile_blocks',
    fieldType: 'rich_text',
    validators: {
      richTextBlocks: {
        itemTypes: [
          contactBlock.id,
        ]
      }
    },
    appearance: {
      editor: 'rich_text',
      parameters: {
        start_collapsed: true
      },
      addons: [],
    },
  });
}

Then we run the migration with:

dato migrate --source=example --inPlace

After running the script, we can check the result in the DatoCMS dashboard:

The Author model now has a new modular content field that accepts Contact blocks.

Now we'll condense a few steps into a single migration file. Our goal is to create a new block, which we'll call Avatar, with an image field. Then we'll update the profile field in the author model so it accepts Avatar blocks as well as Contact blocks.

First, let's create the migration file:

dato new migration 'Add avatar image block to profile field'

In our script, the first step will be to create the new Avatar block.

  const avatarBlock = await client.itemTypes.create({
    name: 'Avatar',
    apiKey: 'avatar_block',
    modularBlock: true,
  });

Next, we'll add a single asset field (of type image) to the block we just created:

  const imageField = await client.fields.create(avatarBlock.id, {
    label: 'Picture',
    apiKey: 'avatar_pic',
    fieldType: 'file',
    validators: {
      required: {},
      extension: {
        predefined_list: 'image',
      },
    },
    appearance: {
      editor: 'file',
      parameters: {},
      addons: [],
    },
  });

Finally, we need to add this block to the valid references for the modular content field in the Author model.

  const profileBlockField = await client.field.find('author::profile_blocks');
  const validBlockTypes = profileBlockField.validators.richTextBlocks.itemTypes;

  client.field.update(profileBlockField.id, {
    validators: {
      richTextBlocks: {
        itemTypes: [...validBlockTypes, avatarBlock.id]
      }
    }
  });

Here's what the complete migration file looks like:

'use strict';

module.exports = async (client) => {
  const avatarBlock = await client.itemTypes.create({
    name: 'Avatar',
    apiKey: 'avatar_block',
    modularBlock: true,
  });

  const imageField = await client.fields.create(avatarBlock.id, {
    label: 'Picture',
    apiKey: 'avatar_pic',
    fieldType: 'file',
    validators: {
      required: {},
      extension: {
        predefined_list: 'image',
      },
    },
    appearance: {
      editor: 'file',
      parameters: {},
      addons: [],
    },
  });

  const profileBlockField = await client.field.find('author::profile_blocks');
  const validBlockTypes = profileBlockField.validators.richTextBlocks.itemTypes;

  client.field.update(profileBlockField.id, {
    validators: {
      richTextBlocks: {
        itemTypes: [...validBlockTypes, avatarBlock.id]
      }
    }
  });
}

Now we can run the migration...

dato migrate --source=example --inPlace

And check the final result in the DatoCMS dashboard. You should find the Avatar block in the Blocks Library.

And in the Author model you should find the updated Profile field:

Adding content to the record

Although migration scripts should mainly be used to change the project structure, sometimes you also need to migrate content.

So let's look at how to add content to a record, both by migrating existing content from one field to another and by adding new content directly.

But first, as usual, we need to create a new migration file.

dato new migration 'Update author record with new profile blocks content'

As we write our script, the first thing to do is check the documentation on how to add a modular block to a record; it explains that the next step is to require the function buildModularBlock from the DatoCMS client. To do this, we need to add the following at the top of our file:

const { buildModularBlock } = require('datocms-client');

Next, we need to find the record we'll update. To do this, we'll list all author records and pick the one we want; in this case, the one whose name field is "Super Dev". We'll then update this record by adding two new modular content blocks to its Profile field; one will be of type Avatar and will contain an image with the same content as the one already in the author's image field. The other will be of type Contact and will have an email address that we pass directly in the script.

'use strict';

const { buildModularBlock } = require('datocms-client');

module.exports = async (client) => {
  // Retrieve author records
  const authorRecords = await client.items.all(
    {
       filter: {
          type: 'author'
        }
      },
    {
        allPages: true,  // otherwise we get only the first page of results
    },
  );

  // Retrieve record for Super Dev
  const superDevRecord = authorRecords.find(rec => rec.name === 'Super Dev');

  // Content in existing picture field
  const superDevPicture = superDevRecord.picture;

  // Block types that we are going to insert
  const avatarBlock = await client.itemType.find('avatar_block');
  const contactBlock = await client.itemType.find('contact_block');

  // Update record
  const createSuperDevProfile = await client.items
    .update(superDevRecord.id, {
      profileBlocks: [
        buildModularBlock({
          itemType: avatarBlock.id,
          avatarPic: superDevPicture,
        }),
        buildModularBlock({
          itemType: contactBlock.id,
          email: 'superdev@me.com',
        }),
      ]
    });
  }

Now we're ready to run the script with our now-familiar command

dato migrate --source=example --inPlace

and then check the results in the dashboard. This is the updated record for our author named "Super Dev":

Conclusion

Now that we've created all the migrations we need and we're happy with the result, we can delete the example environment and follow the workflow to bring these changes into production.

  1. Delete the example sandbox environment from the dashboard or with the command dato environment destroy example.
  2. Turn on maintenance mode from the dashboard or with the command dato maintenance on. If someone is currently entering content in the project, the command will fail. We can pass the flag --force , and they'll be forced out of the project until maintenance mode is turned off.
  3. Run the migrations again from the command line with dato migrate --destination=environment_name.
  4. This is where you should set the endpoint in your project to read data from the new DatoCMS environment and deploy your website or app to make sure everything works correctly.
  5. Promote the newly created environment to primary from the dashboard or with the command dato environment promote environment_name.
  6. Turn off maintenance mode from the dashboard or with the command dato maintenance off.

That's it! If you need to roll back the changes, the old primary environment will still be among the sandbox environments, so you can simply restore it by promoting it.

As we've seen, every step can be run from the command line, which makes it easy to automate the process in a CI/CD pipeline.

Changing content schemas with migration scripts is simple and versatile, and lets you modify your content schema with complete peace of mind, knowing you can experiment in sandboxes without interfering with your production environment.

We recently taught a lesson on this topic for the HeadlessCreator course on DatoCMS; if you're interested, you can watch the start of our lesson on YouTube at the top of this article, or sign up on the HeadlessCreator website to watch the full lesson.

Tags
Code
Development

More articles

Discover more
footer