Stop 13 · Cloud Migration & Multi-CloudTopics · 4/8

Data Migration Techniques

In short

Moving the app is the easy part. Moving 20 TB of customer data without losing a single order, while people keep placing new orders, is where migrations get exciting.

Moving large volumes of data into the cloud — online transfer for smaller datasets, offline/physical transfer devices for very large ones, and continuous replication for minimal-downtime cutovers.

The blob character loads books into a big metal trunk while a bicycle with a few books waits.

Think of it like this

Moving the college library

A few shelves of books can go by bicycle, trip after trip. The whole library goes in sealed trunks on a truck. But students keep returning books during the move, so a librarian notes every new return and sends them over at the end.

Watch it happen

Full load, then change data capture, then cutover

Old database

Replication

Cloud database

App

SELECT * (full load)

1/6The replication service copies every existing row. For big tables this can take hours, and the app stays live meanwhile.

The big ideas

What you need to know

Online transfer

For gigabytes or a few terabytes, copy over the internet or a private link with tools like aws s3 sync or rclone.

Offline devices

For huge datasets, providers ship rugged storage devices, such as AWS Snowball. You fill it and ship it back.

Continuous replication

Database migration services copy everything once, then stream changes, so the final cutover is short.

Validate, don't hope

Compare row counts and checksums on both sides before cutover. A silent missing table is the worst bug.

See it in code

Walk through the code

bash

1# Rough transfer time: 10 TB over a 100 Mbps link2# 10e12 bytes x 8 bits / 100e6 bits per sec = 800,000 s, about 9 days3# That's why big datasets travel on devices.4 5# Copy files to S3, re-runnable: only changed files move again6aws s3 sync /data/uploads s3://campus-migration-uploads/ \7  --storage-class STANDARD_IA --only-show-errors8 9# Check both sides match before cutover10find /data/uploads -type f | wc -l11aws s3 ls s3://campus-migration-uploads/ --recursive | wc -l

1/3Back-of-envelope maths. Bytes times 8 gives bits, divided by link speed gives seconds.

Do the maths before choosing online or offline. Real links rarely run at full speed, so the actual time is longer.

Save yourself the pain

Common mistakes

Oops

Copying the database once on Friday and switching on Monday.

Do this instead

Data written over the weekend is lost. Use change replication, or freeze writes during the copy.

Oops

Assuming the internet link will run at its full rated speed for days.

Do this instead

Measure real throughput first and add a buffer. Plan an offline device if the maths gets ugly.

Your mission

Sync and verify a folder

~30 min

Quick check

Why do database migrations use change data capture after the first full copy?

Go deeper

1Learn when a physical data transfer device becomes faster than transferring over the internet.
2Understand the trade-off between a one-time bulk transfer and continuous replication before cutover.
3Practice estimating transfer time for a hypothetical dataset size given available bandwidth.