One of the common concerns people raise about checkpoint sync is the risk that someone might hack Infura and return malicious initial states causing nodes to sync and be stuck on the wrong chain. Given users usually don’t verify the initial state and Infura is currently the only publicly available service supplying initial states, there is certainly some risk there but how concerned should we really be?

The initial state you use for checkpoint sync is important because it tells the beacon node which chain it should sync and that it should reject all others. So if the initial state is from the wrong chain, your node will sync that chain and any information you get from your node is likely to be wrong. That could result in your attesting to an incorrect chain and getting inactivity penalties on the real one or post-merge being fooled into buying or selling at a bad price because your node gave you an incorrect view of the market.

Definitely sounds scary but let’s work it through.

Firstly, all the big players - staking services, exchanges etc, wouldn’t use Infura, or any other public service, to get their initial states because they can get the initial state from one of their other nodes. So immediately nearly all the high value targets are safe.

Secondly, the attack only works on nodes syncing from scratch and the attacker can’t force people to resync their nodes¹. There’s also a fairly limited amount of time before someone notices their node got an invalid initial state and blows the whistle and the attack would be stopped.

So this attack doesn’t look like a very good way to make money. What about just causing chaos? There’s a relatively small number of nodes syncing the chain at any time and only one of them needs to notice the problem to raise the alarm. So the potential for causing chaos is also quite limited.

Ultimately, the idea that an attacker who is able to compromise Infura would use that to mess with checkpoint sync seems pretty unlikely. They could just mess with the data Infura returned to DApps and directly misrepresent the world state - a much more direct and likely more profitable way of achieving the same result. Or most likely they could just snoop on the incoming stream of transactions and keep all the best MEV opportunities all to themselves.

Does that mean you shouldn’t verify your initial state? Absolutely not. While there’s little reason for an attacker to hack Infura for this purpose, that doesn’t mean it won’t ever happen. And more likely Infura might have a bug which causes it to follow the wrong chain by accident. There’s a lot of room between panicking and claiming checkpoint sync is unsafe (it’s not) and saying that it’s fine to not verify anything (it’s not).

Which is to say…

Apart from being awesomely fast, checkpoint sync also exists to ensure that you can safely sync despite the limitations of weak subjectivity. The initial state you use is considered trusted - you are telling your beacon node that this state is the canonical chain and it should ignore all others. So it’s important to ensure you get the right state.

Get It From Somewhere You Trust

The simplest and best way to ensure the state is right is to get it from somewhere trusted. There are a few options.

The best source is one of your own nodes. For setups that run multiple nodes that’s easy as they can get an initial state for a new node from any of their existing nodes. Even people running single nodes may be able to use a state from their own node in some cases. For example if you need to re-sync your node or are switching clients you can store the current finalized state from your node before stopping it, then use that as the initial state for your new sync. In both these cases the solution is completely trustless - your using data from your own node so no need to trust anyone else and no need for further verification.

If you can’t get the state from your own node you’ll have to get it from someone else. A friend or family member you trust that runs their own node would be an excellent source. This isn’t trustless, but will usually still have a very high level of trust even without any further verification.

Otherwise you’ll have to get the state from a public provider. Currently that’s just Infura, but ideally more options will be available in the future. We’re beyond personal trust circles but Infura is certainly still a very reputable provider so most people would still have a reasonable level of trust in them.

The final option is to get the state from some random person on the internet. Seems crazy and is definitely not something to be trusted, but it is still an option if you verify the state against more trusted sources. Early on, before Infura supported the API to download the state this was actually the most common way people used checkpoint sync. I would just periodically put a state up in a GitHub repo so they could access it.

Verify The State

If you get the state from a source you don’t fully trust, you’ll need to verify it. You can do this by calculating the hash tree root of the state, then checking that against one or more block explorers. In essence you’re aggregating your trust from multiple services until you (hopefully) reach a level you’re comfortable with.

The main problem with this is that you’ll need a tool to calculate the hash tree root of the state itself. It’s simpler to just use the state to sync your node then confirm that the block roots your node reports match block explorers. If you wind up at the right chain head, you must have started from a canonical state. You will likely want to disable any validators you run until you’ve verified the blocks - otherwise you may attest to something you don’t actually trust.

Why Can’t The Beacon Node Do It For Me?

There have been some proposals for beacon nodes to automatically verify the state using heuristics like whether the state matches what the majority of your peers have. I’m personally very skeptical of such ideas because if you could trust the information from the network, you wouldn’t need checkpoint sync in the first place. The fundamental challenge introduced by weak subjectivity is that your node simply can’t determine what the canonical chain is (if it’s been offline for too long) and so has to be told. Your node’s peers are essentially the adversary we’re trying to defend against with checkpoint sync so we want to avoid using any information from them to second guess the initial state.

The beacon node could automate the process of checking against multiple block explorers but there are two problems with that.

Firstly, there isn’t an agreed API to perform that check so we’d need to either design that and get block explorers to support it or write custom code in clients to support each block explorer.

Secondly, and more problematically, client developers would have to decide which block explorers are trust worthy and embed the list into their clients. There’s already a lot of responsibility in being a client dev and a lot of trust the community puts in us - we really don’t want to expand that by also being responsible for deciding which services users should trust. There could just be a config option so users could specify their own list of explorers to verify against but that’s a pretty clunky UX and it’s very unlikely users would go to the effort of finding suitable URLs and specifying them. Besides, it’s probably more work for them than just verifying the block roots manually.

Let’s just admit it, weak subjectivity checkpoints have failed…

The beacon chain brings to Ethereum the concept of weak subjectivity, so prior to the launch of the beacon chain MainNet it was seen as important to have a solution agreed that would allow users starting a new node to verify the chain they sync’d was in fact the canonical chain.

This was before anyone client had implemented checkpoint sync so the proposal was that the epoch and block root from a point close enough to the chain head that it was within the weak subjectivity period but far enough back that it wouldn’t need to change often (e.g update only every 256 epochs). Since it’s small and doesn’t change often people could easily publish it via all kinds of methods - block explorers could show it, twitter bots could post it, it could even be printed in newspapers. Thus it would be easy for users to find it and they’d have lots of different sources to verify against.

Clients would then sync from genesis and when they got to the epoch in the weak subjectivity checkpoint, they’d check that the block root matches the block they have, thus confirming they sync’d the right chain. If it didn’t match the client would crash and the user would have to either correct the weak subjectivity checkpoint they were providing or delete the current sync and try again, hoping the client wound up on the right chain this time.

Of course, that’s the first major problem with weak subjectivity checkpoints - you can waste days syncing before realising you’re on the wrong chain. Then once you discover that you have to start again but don’t really have anything you can do to avoid the problem happening again.

Somewhat surprisingly though, that’s not the biggest problem with weak subjectivity checkpoints. The biggest problem is that they’re actually extremely hard to find. For a while beaconscan provided an endpoint that would give back the current weak subjectivity checkpoint - no explanation of what it was or how to use it, but at least you could get the value. It appears they silently removed it at some point though. I don’t recall seeing weak subjectivity checkpoints published anywhere else.

In fact, I’m actually having trouble turning up a clear reference of how to calculate the weak subjectivity checkpoint. There is this doc in the specs which says how to calculate the weak subjectivity period, which amusingly basically has a TODO how to distribute the checkpoints. There’s also this doc from Aditya which gives a lot more detail on weak subjectivity periods. I had thought there was some spec for ensuring everyone selected the same epoch to use for the weak subjectivity checkpoint and I thought it was a simple “every X epochs” but I can’t actually find that documented anywhere.

We could fix this - a clearer and/or simpler spec on how to get the weak subjectivity checkpoint could be created (and there is a suggestion of adding it to the standard REST API), we could put pressure on providers to publish those checkpoints and setup twitter bots etc so it was available. Even then, we’d still have a truly lousy user experience with slow sync times that get longer as more blocks are piled on to the beacon chain and that can only retrospectively tell you that you’ve been wasting your time.

So let’s just admit it, weak subjectivity checkpoints have failed. The future is in using checkpoint sync to start from the current finalized state. That minimises sync time and users can realistically take advantage of it today. It’s not perfect, we need more places to provide that state and better ways of validating that the state is in fact the right one, but it’s already more accessible than weak subjectivity checkpoints and has infinitely more users actually using it (ie more than zero). If we can move on from the idea of weak subjectivity checkpoints, we can focus on getting the most out of checkpoint sync in terms of usability and security.

Recording of my talk on Checkpoint Sync and Weak Subjectivity at EthStaker’s gathering as part of DevConnect 2022.

I was invited to talk at EthStaker’s gathering as part of DevConnect 2022 on Checkpoint Sync and Weak Subjectivity. Stashing the link to the recording here for posterity.

In Teku, we’ve come up with a new way to both serialize/deserialize objects to JSON and generate OpenAPI documentation from the same declarative source. This significantly reduces the amount of boiler-plate code we have to write and also avoids a lot of bugs where the generated JSON diverged from the OpenAPI.

Previously, Teku’s JSON serialization and deserialization was controlled by a set of custom classes with a bunch of annotations added to define the JSON serialization. The OpenAPI speicfication utilised those, plus a bunch more annotations to generate the OpenAPI.

As an example of how this works, let’s look at the get block root API. It returns data like:

{
    "data": {
        "root": "0xcf8e0d4e9587369b2301d0790347320302cc0943d5a1884560367e8208d920f2"
    }
}

To generate that with the old system we’d need to create two classes:

public class GetBlockRootResponse {

  @JsonProperty("data")
  public final Root data;

  @JsonCreator
  public GetBlockRootResponse(
      @JsonProperty("data") final Root data) {
    this.data = data;
  }
}

public class Root {
  @Schema(
      type = "string", 
      format = "byte", 
      description = "Bytes32 hexadecimal",
      example = "0xcf8e0d4e9587369b2301d0790347320302cc0943d5a1884560367e8208d920f2")
  public final Bytes32 root;

  @JsonCreator
  public Root(@JsonProperty("root") final Bytes32 root) {
    this.root = root;
  }
}

And then to generate the OpenAPI we’d need additional annotations on the API handler method:

@OpenApi(
    path = ROUTE,
    method = HttpMethod.GET,
    summary = "Get block root",
    tags = {TAG_BEACON},
    description = "Retrieves hashTreeRoot of BeaconBlock/BeaconBlockHeader",
    responses = {
    @OpenApiResponse(
        status = RES_OK,
        content = @OpenApiContent(from = GetBlockRootResponse.class)),
    @OpenApiResponse(status = RES_BAD_REQUEST),
    @OpenApiResponse(status = RES_NOT_FOUND),
    @OpenApiResponse(status = RES_INTERNAL_ERROR)
    })
@Override
public void handle(@NotNull final Context ctx) throws Exception { ... }

There are a number of drawbacks to this approach. Firstly, that’s quite a simple format for JSON output but we’ve had to define two separate classes to model it in Java. Secondly, the @Schema attestation on the Bytes32 root needs to include quite a few options to correctly define the byte32 object in the OpenAPI which has to be repeated for every Bytes32 field in the API. We can extract the actual string values as constants, but the annotation has to be added correctly each time. And that Bytes32 is actually an interface from a third-party library so we still need a custom serializer for it.

The biggest issue though is that the OpenAPI and the actual JSON serialization is only fairly losely tied together. There’s nothing to stop the handler from returning a response code that wasn’t listed in the OpenAPI, or even returning data that isn’t of type GetBlockRootResponse.

Fundamentally, trying to use Java classes to define a JSON data structure is just a really poor match. Layering on more annotations and more reflection to also generate the OpenAPI documentation from those classes just makes the problem worse. So we’ve come up with an alternative approach based on the idea of type definitions.

With this approach, we define a SerializableTypeDefinition which specifies how to convert a class - our reguar internal data structure - into JSON, using a declarative approach. Here’s what it looks like for the block root response:

public static final StringValueTypeDefinition<Bytes32> BYTES32_TYPE =
    DeserializableTypeDefinition.string(Bytes32.class)
        .formatter(Bytes32::toHexString)
        .parser(Bytes32::fromHexString)
        .example("0xcf8e0d4e9587369b2301d0790347320302cc0943d5a1884560367e8208d920f2")
        .description("Bytes32 hexadecimal")
        .format("byte")
        .build();

private static final SerializableTypeDefinition<Bytes32> GET_ROOT_RESPONSE =
      SerializableTypeDefinition.object(Bytes32.class)
          .withField(
              "data",
              SerializableTypeDefinition.object(Bytes32.class)
                  .withField("root", BYTES32_TYPE, Function.identity())
                  .build(),
              Function.identity())
          .build();

The first part - BYTES32_TYPE is a reusable definition for serializing a Bytes32. It’s represented as a String in the JSON, so we provide a formatter and a parser to convert a Bytes32 to a String and back. The example, description and format provide the additional documentation required for the OpenAPI output.

The second part - GET_ROOT_RESPONSE defines the two objects involved in the JSON, the outer wrapper which is an object with a single field called data. The data field in turn is a single field object with a root property. The root field is a bytes32 field so uses that reusable definition. We could extract the data object as a reusable definition too but since it’s only used here it’s simpler to define it inline.

The withField call in that definition takes three things - the name of the field, the type of the field and a getter Function which maps from the source value to the value for the field. In this case those getter functions are both just Function.identity() because the only data we need is the Bytes32 root value, so we just pass it through the layers of JSON objects unchanged until it gets used as the value for the root field and actually serialized by the formatter specified in BYTES32_TYPE. In other situations, where we have more complex data structures, the getter function could perform calculations or just be a reference to an actually get*() method in the Java object.

Those getters provide one of the key benefits of this approach because they decouple the internal Java object holding the data from the external JSON representation. That allows us to refactor the internal representation we use to best suit our code, while maintaining a stable external API. Whereas with the annotation based approach, changing the structure of the object would automatically change the structure of the resulting JSON.

That same type definition can be used to generate an OpenAPI specification for the JSON. Most of the information required for the OpenAPI is inferred from the way we defined the serialization. We know GET_ROOT_RESPONSE is an object and it has a data field, we know the type of that field etc. We can add an additional description to any of those types to provide more documentation, just like we did for BYTES32_TYPE.

One final detail here is that the above object type is a SerializableTypeDefinition so it can convert from an internal type to JSON, but it can’t parse JSON back to the internal type. We can also create a DeserializableTypeDefintion for objects which then require providing both a getter and a setter for each field (a builder can also be used, allowing for immutable types). Most of the time we only need to serialize though so there’s no need to define how to parse the type.

The final piece is to provide the definition of the actual REST endpoint - equivalent to the @OpenApi annotation in the original version:

EndpointMetadata.get(ROUTE)
    .summary("Get block root")
    .tags(TAG_BEACON)
    .description("Retrieves hashTreeRoot of BeaconBlock/BeaconBlockHeader")
    .response(SC_OK, "Request successful", GET_ROOT_RESPONSE)
    .withBadRequestResponse()
    .withNotFoundResponse()
    .withInternalErrorResponse()
    .build();

While these simple examples are fairly similar, the type definitions approach results in far less boilerplate code. That’s not the main advantage though.

The type definitions provide an explicit mapping from our internal classes to the serialization we need on the API. There’s no need to create custom classes just to define the API, or to spread Jackson annotations all through our internal code. Most importantly there’s no risk that renaming a class or field will unexpectedly change the external API.

The EndpointMetadata and type definitions also build a simple to use model of the types and the expected behaviour of the endpoint. We’ve added a very thin wrapper around Javalin (which we were already using) that checks the responses being sent actually match what is declared in the metadata (and thus in the OpenAPI generated from it). Unfortunately those checks are still occurring at runtime, but it means that unit tests will fail if the OpenAPI doesn’t match where previously it was very difficult to add tests that check the OpenAPI and responses actually match.

For Teku, there’s another big advantage - we can create type definitions straight off the SSZ schemas we already have for the various types defined in the beacon chain spec. Given the majority of responses from the standard REST API are JSON versions of the SSZ objects, we save a ton of code by not having to redefine the type for JSON. For example, the endpoint to get a validator from a state defines it’s response type as:

SerializableTypeDefinition<StateValidatorData> DATA_TYPE =
    SerializableTypeDefinition.object(StateValidatorData.class)
        .withField("index", UINT64_TYPE, StateValidatorData::getIndex)
        .withField("balance", UINT64_TYPE, StateValidatorData::getBalance)
        .withField("status", STATUS_TYPE, StateValidatorData::getStatus)
        .withField(
            "validator",
            Validator.SSZ_SCHEMA.getJsonTypeDefinition(),
            StateValidatorData::getValidator)
        .build();

The index, balance and status are additional fields that aren’t part of the SSZ definitions so have been added, but the validator field is just a JSON version of the Validator type from the spec. So rather than redefine it, we can just get the SSZ schema and ask it for it’s JSON type definition which is automatically created. And the getter (StateValidatorData::getValidator) is just returning the exact Validator instance we got from the BeaconState.

We initially adopted this new approach for the whole validator key manager API which confirmed it worked well. We’re now in the process of converting our existing beacon node API over to the new style. We’re also using parts of it to create the JSON we send to the execution layer calls and plan to use more. Eventually we’ll replace all our Jackson annotation based JSON handling over to use type definitions.

Bookmarking some resources for remote pairing I’ve recently stumbled across but don’t have time to look into right now.

I’ve recently been pointed to a few resources for remote pairing and remote development teams in general that look like they’re worth trying. I don’t have time to actually try them out now and I need to get my browser tabs back under control, so dropping some links here so I don’t lose them.

• mob.sh - simple script that works with git to make it easy to transfer work between a group of people mobbing (would also work for pairing).
• Effective Home Office has some general tips for setting up a home office, including some recommended Zoom settings.
• Tuple is a screen sharing system designed for pairing.
• Pop is another one.

Both the screen sharing options are interesting because they appear to be full screen sharing, meaning you can use your favourite IDE instead of being stuck with a specific text editor because it has multi-user capabilities.

After far too many years of not getting around to it, I’ve finally migrated this blog off of WordPress. It now uses Hugo to statically generate the site.

While I’m still scratching the surface of what Hugo can do, and not entirely happy with this particular theme, the switch seems to have been pretty easy. I used WordPress to Hugo Exporter to migrate the content over which was pretty painless. Given this blog has had quite a few different approaches to authoring content over the years there will no doubt be some formatting that hasn’t come across all that well. Then again, there’s still a bunch of old format that doesn’t format correctly because of the transition from MoveableType to WordPress way back in the day. Oh well.

There’s a relatively good chance I’ve either broken RSS feeds and/or caused content to be double published. Sorry about that… The worst instance of that is that I took down our family travel blog and the domain wound up with a default resolve to my personal blog, which then caused a whole bunch of technical blog posts to be sent out to everyone who subscribed to our travel blog. Also sorry about that. You shouldn’t get this post as an email, but if you do, er, sorry again…

With the merge fast approaching, all beacon chain nodes will need to run their own local execution layer client (e.g. Besu or Geth). For validators who have just been using Infura for eth1 data so far, that means some extra setup and potentially more hardware (mostly disk). To avoid this, some people are considering going the other direction and not running their own beacon node either, just running a validator client against a public beacon node API like Infura. While this is possible, it’s extremely likely to result in reduced rewards for a couple of reasons.

Note: This is about using a third party beacon node. A third party execution layer client doesn’t work at all.

The first reason is simple – a remote beacon node is likely to perform slightly worse because of the latency in making requests from the validator client to the beacon node. The difference isn’t huge but it does add up.

The bigger reason though is that you’re very likely to miss out on the transaction inclusion fees that will be paid to validator’s execution layer accounts when they proposed blocks post-merge. These are the “miner tips” that are currently paid to miners and while block proposals don’t come along often, they do pay particularly well because of these fees.

So why wouldn’t you get these fees? To explain that, we need to understand how the block creation process works post-merge.

There are multiple requests involved in creating a block. First, basically at startup and then every couple of epochs, the validator client sends a call to /eth/v1/validator/prepare_beacon_proposer which tells the beacon node which validators it should expect to produce blocks for and what fee recipient to use for them.

The beacon node then watches for when one of those validators is scheduled to prepare a block and a little before the block’s slot starts, sends a engine_forkChoiceUpdated call to the execution layer client containing the details required for the execution client to begin preparing a block – including the fee recipient to use. The execution client returns a payloadId which the beacon node keeps track of.

Then when the block is actually due to be produced, the validator client sends a request to eth/v2/validator/blocks/:slot asking the beacon node to actually produce the block. The beacon node uses the payload ID it already has from the execution client to request the execution payload, bundles it all up and sends it back to the validator client to sign.

Finally the validator client sends the signed block back to the beacon node to import and publish out onto the libp2p gossip network.

So if I wanted to steal inclusion fees from anyone using a public beacon node, I’d just send regular calls to /eth/v1/validator/prepare_beacon_proposer with every validator ID and specifying my account as the fee recipient. There’s no requirement to prove I have the validator keys because the system is designed around validators running their own beacon node. That makes sense because the job of a validator is to run their own node and independently track and verify the state of the chain. That’s literally what they’re paid to do. It would be reasonably easy to add a requirement to the prepare_beacon_proposer API that the request is signed by the validator key to stop this interference but that would add extra overhead for the majority of validators doing the right thing and running their own node to benefit validators doing the wrong thing and shirking their responsibilities.

If that sounds harsh, remember that Ethereum is not here to make validators rich – quite the opposite, validators are here to provide a service to Ethereum (securing the chain) and are paid appropriately for that service.

There are future plans to further strengthen the requirement for validators to run their own node by introducing proof of execution so validators could potentially be slashed if they don’t run their own execution client to properly validate payloads before attesting to them. Otherwise there’s a potential tragedy of the commons where every validator assumes the other validators are executing payloads and so doesn’t do it themselves. Taken to the extreme payloads may not get validated at all and that would be a major problem for Ethereum security.

So bite the bullet today and get an execution client up and running. While you’re at it, do your bit for client diversity and run something other than Geth – Besu with its new bonsai trees storage is fantastic.

Yesterday I spun up two new nodes on Prater, and sent deposits for 5000 validators to run on each. Due to the restrictions on how many new validators can activate per epoch (currently 4 for Prater), the number of active validators are currently gradually increasing giving a really nice insight into the cost of adding new validators.

This is most clearly seen by graphing the number of active validators and CPU usage on a single chart.  Here the green is CPU usage and the yellow the number of active validators.

Once validators start activating we see CPU usage increase which is what we’d intuitively expect and for the first 60-70 validators there’s a clear correlation between adding validators and increasing CPU usage. However beyond about 70 validators the CPU usage flattens out even while the active validator count keeps increasing – why?

The answer lies in the way attestations are handled. To allow the beacon chain to scale to hundreds of thousands of validators, validators are assigned to separate committees. This sets not only which slot of the epoch they attest in (thus each slot only has 1/32 of the total validators attesting) but also which committee within that slot they are grouped into. There are a maximum of 64 committees per slot and each committee publishes their unaggregated attestations into a separate gossip topic. Then a subset of the validators in the committee are assigned to collect up those unaggregated attestations and publish an aggregated attestation which combines them.

Nodes with no active validators have no reason to subscribe to the gossip topics for unaggregated attestations so only process the aggregates. As validators activate though, each one is required to randomly select an attestation gossip topic to subscribe to long term and if they’re a validator need to subscribe to the topic they are aggregating for a period so they can collect and aggregate the individual attestations.

So for the first 64 validators the node winds up subscribing to an additional attestation gossip topic and processing 1/32 * total_validators individual attestations every slot. For prater that’s nearly 9,000 additional attestations per slot, for each active validator. No surprise then that CPU usage increases as we have to validate all of those attestations.

We see roughly the same correlation in the rate of gossip messages we receive. Here we’ve added gossip messages per second to the graph shown in blue.

There’s a somewhat unexpected dip in gossip messages as the number of validators are ramping up – I think that’s actually a point where I restarted the node to apply some updates, somewhat ruining my pretty pictures…

Even so, we can see the very clear growth in gossip messages as the first validators activate, along with the increased CPU. Once we get to 64 validators activated both the CPU usage and gossip message rate levels off even as the validators continue to grow. At that point, we’re already subscribed to every gossip topic and processing every individual attestation. Adding more validators after that doesn’t take very much – they need to calculate their duties and produce one attestation per epoch, plus sometimes creating a block but that’s very cheap compared to validating all that attestation gossip.

So the first 64 validators you add to a node are quite expensive in terms of CPU usage and network traffic, but if you can run 64 validators well, then you can likely run thousands of validators well because you’re already processing the bulk of the attestations.

Of course, as the total number of validators continues to increase, nodes that are subscribed to all attestation gossip topics will have to process more and more attestations – they don’t get the benefit of the aggregation which is such a key part of the beacon chain scaling. For now it’s still quite manageable and there’s a fair bit of headroom left, but eventually large staking providers may need to start being smarter about how they manage attestation gossip requirements by spreading the duties across their fleet of nodes rather than having each node work independently. There’s a few ways that could work but that’s a story for another day…

For many years I’ve been using the free tier of Google Workspace (previously Google Apps for Your Domain) to host my email. While I almost never use the actual web interface itself having google handle the IMAP hosting and spam filtering was an awful lot easier than hosting it myself. Of course Google are now ending their free tier and requiring all users to pay – fair enough, but the cost per user is just too much for me to justify it.

It turns out though that as part of my Office 365 subscription to get access to office apps, we also get Outlook.com premium which provides quite suitable email hosting for what I need. While it can, at least theoretically, use a custom domain directly, that domain has to be hosted with GoDaddy and the transfer process seemed hard to do without downtime and migration issues. The family would not be forgiving of that. So instead, since my domains are already hosted with CloudFlare we’ll just use their email forwarding so emails for our domain get forwarded to Outlook.com and Outlook.com happily lets us send with the custom domain as the From address so it all works out. Switching over will just mean setting up the right forwards in CloudFlare and pressing the button to switch the MX records from Google to CloudFlare.

Except there’s a whole heap of email in Google’s IMAP store that needs to be moved over to Outlook’s IMAP store. That could be done by simply adding both accounts to an email application and dragging the emails over. For small numbers of emails that works pretty well, but it’s pretty easy for things to get interrupted part way through and some emails are copied over and others aren’t and then it’s hard to restart the copy without getting duplicates etc. Pretty quickly it becomes unworkable for a large number of emails.

Instead, I’ve setup isync (actual command name mbsync) to do the transfer. It appears to be very good at resuming after an interruption and just picking up where it left off copying folders and emails. It even gets things like the read/unread status right. The configuration isn’t entirely straight forward so for anyone else doing this migration, my config is below.

It’s setup with two users but more can be added by following the same pattern. You can either just run mbsync -a to migrate all email for all users or mbsync user1 to migrate just email for user1 and similarly for user2. Migrating them all in one command does them both sequentially so generally it will be faster to migrate each user separately.

Often, mbsync will exit for a variety of reasons (gmail suddenly closing the connection, outlook.com enforcing request limits etc) so I run it with a simple bash for loop:

while true; do mbsync adrian; echo "Disconnected, reconnecting after delay..."; sleep 30; done

Note that it will loop forever so you’ll want to check it from time to time and when it’s reporting that nothing needed to be copied just ctrl-C it to exit.

IMAP is not a particularly efficient protocol and rate limiting winds up being applied so this is definitely not a fast process, but you can just set it off and leave it run in the background until it’s done. 

IMAPAccount gmail-user1
Host imap.gmail.com
User 
# Simplest approach is to enable 2FA and generate an app specific password to use here.
# Revoke the app specific password again once the transfer is complete
Pass 
Timeout 120
AuthMechs LOGIN
SSLType IMAPS


IMAPStore user1-source
Account gmail-user1

IMAPAccount outlook-user1
Host imap-mail.outlook.com
User 
# Simplest approach is to enable 2FA and generate an app specific password to use here.
# Revoke the app specific password again once the transfer is complete
Pass 
SSLType IMAPS
Timeout 120

IMAPStore user1-target
Account outlook-user1

IMAPAccount gmail-user2
Host imap.gmail.com
User 
Pass 
Timeout 120
AuthMechs LOGIN
SSLType IMAPS

IMAPStore user2-source
Account gmail-user2

IMAPAccount outlook-user2
Host imap-mail.outlook.com
User 
Pass 
SSLType IMAPS

IMAPStore user2-target
Account outlook-user2

Channel user1-main
Far :user1-source:
Near :user1-target:
Patterns * ![Gmail]* !Archive
Create Near
Expunge Near
CopyArrivalDate yes
Sync Pull
SyncState *
SyncState ~/.mail/imap-transfer/user1/

Channel user1-sent
Far :user1-source:"[Gmail]/Sent Mail"
Near :user1-target:"Sent"
Create Near
Expunge Near
Sync Pull
CopyArrivalDate yes
SyncState *
SyncState ~/.mail/imap-transfer/user1/

Channel user1-trash
Far :user1-source:"[Gmail]/Trash"
Near :user1-target:"Deleted"
Create Near
Expunge Near
Sync Pull
CopyArrivalDate yes
SyncState *
SyncState ~/.mail/imap-transfer/user1/

Group user1
Channel user1-main
Channel user1-sent
Channel user1-trash



Channel user2-main
Far :user2-source:
Near :user2-target:
Patterns * ![Gmail]* !Archive
Create Near
Expunge Near
CopyArrivalDate yes
Sync Pull
SyncState *
SyncState ~/.mail/imap-transfer/user2/

Channel user2-sent
Far :user2-source:"[Gmail]/Sent Mail"
Near :user2-target:"Sent"
Create Near
Expunge Near
Sync Pull
CopyArrivalDate yes
SyncState *
SyncState ~/.mail/imap-transfer/user2/

Channel user2-trash
Far :user2-source:"[Gmail]/Trash"
Near :user2-target:"Deleted"
Create Near
Expunge Near
Sync Pull
CopyArrivalDate yes
SyncState *
SyncState ~/.mail/imap-transfer/user2/

Group user2
Channel user2-main
Channel user2-sent
Channel user2-trash