1.x to 2.x Migration Guide

Here you will find the required steps to migrate an existing single-tenant (1.x) Stellar Disbursement Platform application (SDP) to a multi-tenant (2.x+) version.

Why Migrate?

Starting with version 2.x, the SDP provides a multi-tenant architecture, where multiple organizations can manage disbursements under a unified infrastructure while maintaining separate datasets and funds management.

New features and fixes will only be published to the multi-tenant version.

The multi-tenant version also suits single-tenant scenarios, through a simplified configuration that will be described later in this document.

DISCLAIMER: this guide was prepared and tested with 1.1.6 -> 2.0.0-rc1 code bases. If you're in a later version, it's possible that the new database migrations cause an error when following these steps.

You have the option of using the 2.0.0-rc1 version to perform this migration, and then upgrade to the latest version after the migration is completed.

Preparing for the Migrations

Halt the Single-Tenant Version 🚧

Before proceeding with the migration, ensure that the single-tenant version of the SDP is not running. This is crucial to prevent any data inconsistencies or conflicts during the migration process.

Double-Spending Prevention 🚨

To avoid double-spending, ensure that no payment is in the PENDING state, otherwise this could result in having the same payment submitted independently by both single-tenant and multi-tenant instances. You can do that by checking the payments table for any payment in the PENDING state:

SELECT * FROM public.payments WHERE status = ANY(array['PENDING']::payment_status[]);

Similarly, check the submitter_transactions table for any transaction in the PENDING state:

SELECT * FROM public.submitter_transactions WHERE status = ANY(array['PROCESSING']::transaction_status[]);

If there are any payments in the PENDING state or transactions in the PROCESSING state, you should wait for them to be processed and transitioned to either SUCCESS or FAILED before proceeding with the migration.

Remove Channel Accounts 🧹

In version 2.x, the channel accounts are encrypted using the CHANNEL_ACCOUNT_ENCRYPTION_PASSPHRASE, which is a different env from the one used in the single-tenant version (DISTRIBUTION_SEED). To avoid any issues, let's remove all existing channel accounts in the single-tenant version priot to the migration:

bash

./stellar-disbursement-platform channel-accounts delete --delete-all-accounts

Database Backup & Setup 💾

Backup your single-tenant database before proceeding with the migration. At this point, the single-tenant instance should be halted, and there shouldn't be any PENDING payments nor PROCESSING submitter_transactions.

It's also recommended that you create a new database for the multi-tenant version to avoid any data loss. Use the dump from the single-tenant database to populate the initial state of this multi-tenant one. From this point onwards, anytime we refer to the database, we'll be referring to the new multi-tenant database that was created from a dump of the single-tenant database.

Here's how you can do the backup and restore:

bash

pg_dump --dbname=$singleTenantDB > sdp-singleTenant.sql
createdb $multiTenantDB
psql --dbname=$multiTenantDB < sdp-singleTenant.sql

Changes Explained

Among the changes introduced in the multi-tenant version, the most significant ones are:

Infrastructure: the multi-tenant version includes an event-broker (Kafka) as an additional infrastructure component that is highly-recommended for multi-tenant setups, especially when high throughput is required.
Environment Variables: the multi-tenant version introduces new environment variables to configure the event broker, as well as additional variables relevant for multi-tenancy.
Seggregation of Funds: the multi-tenant version introduces the concept of a distribution account for each tenant, which is used to submit transactions on behalf of the tenant. There's also the HOST distribution account, that's used to fund the tenant distribution accounts and the TSS channel accounts.
CLI Commands: some CLI commands have been revised to be tenant-aware, while others have been included to support new multi-tenant features.
Database Structure: the database structure has been revised to accommodate multi-tenancy and the tables are now distributed across multiple schemas, providing isolation between tenants.

Infrastructure

For the infrastructure setup, SDP Multi-tenant offers flexible operational modes.

Event Broker vs Scheduled Jobs

Administrators can choose between using an event broker for event-driven operations, or scheduled jobs for periodic operations. Event brokers are recommended for multi-tenant setups, as they provide a scalable and reliable way to handle events, while scheduled jobs are recommended for local setups or single-tenant SDPs. Also, event-brokers provide faster communication between services.

Anchor Platform Version

While the single-tenant used stellar/anchor-platform:2.1.3, the multi-tenant version requires stellar/anchor-platform:2.6.0 or later.

Environment Variables

Below are the environment variables that have been added or modified in the multi-tenant version.

General environment variables:

ADMIN_ACCOUNT: The username of the admin account used to authenticate HTTP requests to the Admin server. The Admin-targeted requests should add the "Authorization" header, formatted as Base64-encoded "ADMIN_ACCOUNT:ADMIN_API_KEY".
ADMIN_API_KEY: The api key of the admin accountused to authenticate HTTP requests to the Admin server. The Admin-targeted requests should add the "Authorization" header, formatted as Base64-encoded "ADMIN_ACCOUNT:ADMIN_API_KEY".
ADMIN_PORT: the port of the Admin server used to create and manage tenants. Default is 8003.
INSTANCE_NAME: the name of the SDP instance to be displayed in the stellar.toml file. Example: "SDP Testnet".
SINGLE_TENANT_MODE: When set to "true", it enables the single-tenant mode, which is useful for local development or single-tenant setups. In addition to set it to true, you'll need to configure the default tenant by calling the POST /tenants/default-tenant request.
TENANT_XLM_BOOTSTRAP_AMOUNT: The amount of XLM that the HOST Stellar account will deposit deposited to the tenant distribution account for tenant bootstrap.

Stellar accounts configuration environment variables:

CHANNEL_ACCOUNT_ENCRYPTION_PASSPHRASE: A Stellar-compliant ed25519 private key used to encrypt/decrypt the channel accounts' private keys. When not set, it will default to the value of the 'distribution-seed' option, which was used in the single-tenant version. Attention, when migrating from the single-tenant, setting this config to something different from the old DISTRIBUTION_SEED will prevent the code from being able to decrypt the channel accounts.
DISTRIBUTION_SIGNER_TYPE: The type of signer used to sign Stellar transactions for the tenants' distribution accounts. Options: DISTRIBUTION_ACCOUNT_DB (recommended), DISTRIBUTION_ACCOUNT_ENV (same as the single-tenant version).
DISTRIBUTION_ACCOUNT_ENCRYPTION_PASSPHRASE: A Stellar-compliant ed25519 private key used to encrypt/decrypt the in-memory distribution accounts' private keys. It's mandatory when the distribution-signer-type is set to DISTRIBUTION_ACCOUNT_DB.

Event broker configuration environment variables:

BROKER_URLS: A comma-separated list of the message broker URLs.
CONSUMER_GROUP_ID: Specifies a group ID for the broker consumers.
EVENT_BROKER_TYPE: Specifies the type of event broker to be used. Options: "KAFKA", "NONE". Defaults to "Kafka".
KAFKA_SECURITY_PROTOCOL: Defines the security protocol for Kafka. Options: PLAINTEXT, SASL_PLAINTEXT, SASL_SSL, SSL.
KAFKA_SASL_USERNAME: Specifies the Kafka SASL Username, required when the kafka security protocol is set to either SASL_PLAINTEXT or SASL_SSL.
KAFKA_SASL_PASSWORD: Specifies the Kafka SASL Password, required when the kafka security protocol is set to either SASL_PLAINTEXT or SASL_SSL.
KAFKA_SSL_ACCESS_KEY: The Kafka Access Key (keystore) in PEM format, required when the kafka security protocol is set to SSL.
KAFKA_SSL_ACCESS_CERTIFICATE: The Kafka SSL Access Certificate in PEM format that matches with the Kafka Access Key, required when the kafka security protocol is set to SSL.

Scheduler environment variables:

ENABLE_SCHEDULER: Default "false". This enables scheduled jobs that replace the operations of a broker for synchronizing payments between SDP and TSS as well as submitting receiver invitations. It should be set to "true" when the event broker is disabled.
SCHEDULER_PAYMENT_JOB_SECONDS: Interval in seconds for the job that synchronizes payments between SDP and TSS. Minimum is 5s.
SCHEDULER_RECEIVER_INVITATION_JOB_SECONDS: Interval in seconds for the job that submits receiver invitations. Minimum is 5s.

On the Anchor Platform side, we must set the following envs:

SEP10_HOME_DOMAINS="localhost:8000, *.stellar.local:8000": a comma-separated list of home domains used for SEP-10. This should contain the domains of the SDP tenant instances.
SEP10_HOME_DOMAIN="": this should be an empty string for the Multi-tenant version.
SEP10_WEB_AUTH_DOMAIN=<localhost:8080>: the home domain of the anchor platform instance.

Seggregation of Funds

In the multi-tenant version, we support the segregation of funds between tenants by configuring the DISTRIBUTION_SIGNER_TYPE:

When set to DISTRIBUTION_ACCOUNT_ENV, the distribution account private key is read from the environment variable DISTRIBUTION_SEED and the funds are not seggregated.
When set to DISTRIBUTION_ACCOUNT_DB, the distribution account private key is stored in the database and the funds are seggregated between tenants. The secret is encrypted using the DISTRIBUTION_ACCOUNT_ENCRYPTION_PASSPHRASE environment variable.

The channel accounts on the other hand are shared between tenants, and they are created by the HOST distribution account, set in the DISTRIBUTION_SEED environment variable. The channel accounts are encrypted using the CHANNEL_ACCOUNT_ENCRYPTION_PASSPHRASE environment variable, or the DISTRIBUTION_SEED if the former is not set. In the single-tenant version, the channel accounts were encrypted by the DISTRIBUTION_SEED.

CLI Commands

The following CLI commands were updated to become tenant-aware:

Database Migrations & Population

The single-tenant commands for DB migration and pre-population used to be:

bash

./stellar-disbursement-platform db migrate up           # ⚠️ DECOMISSIONED in 2.x!
./stellar-disbursement-platform db auth migrate up      # ⚠️ 2.x REQUIRES A (NEW) TENANT-AWARE FLAG!
./stellar-disbursement-platform db setup-for-network    # ⚠️ 2.x REQUIRES A (NEW) TENANT-AWARE FLAG!

The new multi-tenant commands are:

bash

./stellar-disbursement-platform db admin migrate up
./stellar-disbursement-platform db tss migrate up
./stellar-disbursement-platform db auth migrate up --all
./stellar-disbursement-platform db sdp migrate up --all
./stellar-disbursement-platform db setup-for-network --all

Please notice that some commands require a tenant-aware flag, that can be either:

--all: to execute these migrations in all tenants.
--tenant-id: to specify the tenant ID to execute the migrations.

Channel Accounts

The ensure command was updated from using the --num-channel-accounts-ensure flag to using a positional argument:

bash

./stellar-disbursement-platform channel-accounts ensure --num-channel-accounts-ensure 1 # OLD WAY
./stellar-disbursement-platform channel-accounts ensure 1                               # NEW WAY

Database Structure

In the updated version, the database structure has been revised to accommodate multi-tenancy. As a result, the tables are now distributed across multiple schemas, and new tables have been introduced to support the multi-tenant architecture. The following schemas are used in the multi-tenant version:

admin: it houses tables associated with tenant administration. It serves as the central point for managing tenant-related information and to resolve tenant schema names based on tenant IDs. None of these tables existed in the single-tenant version.
sdp_<tenant_name>: are per-tenant schemas that are prefixed with sdp_. For example, for tenants BlueCorp and RedCorp, the provisioned schemas would be named sdp_bluecorp and sdp_redcorp, respectively. These schemas contain all necessary tables for the SDP operation tailored to each tenant, including per-tenant user authentication.
tss: is a schema dedicated to the Transaction Submitter Service (TSS). The TSS tables do not belong to any tenant, although each TSS transaction contains a column that signals which tenant it belongs to.

These changes allow for the isolation of tenant data per schema, ensuring that each tenant's data is kept separate from other tenants.

Step-by-Step Migration Guide

EDIT: the code contains a helper script called ./dev/migrate_1.1.6_to_2.0.0.sh that does most of the below steps automatically for you. Keep in mind that you'll need to edit the script with values such as your database DSN, and your tenant name. If you see any errors, you may still need to resort to the steps below to incrementally execute the migration manually.

Using the new database created from the single-tenant database dump as a starting point (as described in the Database Backup & Setup section), we can now proceed with the migration steps below.

Deploy the New Version

To transaction to a multi-tenant setup, deploy the latest version of the SDP 2+ version. If you're using helm charts, you can get the helm chart from the SDP Helm Chart repository.

Executing Initial Database Migrations

Following the deployment of the multi-tenant SDP, the next step is to perform the initial database migrations. It does not include the tenant-specific tables yet, they will be created later.

Migration Commands:

bash

./stellar-disbursement-platform db admin migrate up
./stellar-disbursement-platform db tss migrate up

These commands will create the tenant admin tables on the admin schema and the Transaction Submitter Service tables on the tss schema, respectively.

New Tenant Provisioning Process

After successfully applying the database migrations, the next step is to provision a new tenant. This is achieved by making an HTTP request to the Admin API.

Be aware that it will provision the tenants with all the per-tenant migrations included.

Starting the SDP API server

To facilitate tenant provisioning, initiate the SDP Server using the command:

bash

./stellar-disbursement-platform serve

Posting to the Admin API

API port: The Admin API is accessible on port 8003 by default. This port setting can be adjusted by altering the ADMIN_PORT environment variable.
Authentication: Admin API employs Basic Authentication for securing access. To authenticate, populate the request "Authorization" header with "Authorization: Basic ${Base64(ADMIN_ACCOUNT:ADMIN_API_KEY)}".

Here's a shell script that can be used to create a tenant through the Admin API:

bash

ADMIN_ACCOUNT="SDP-admin"
ADMIN_API_KEY="api_key_1234567890"
basicAuthCredentials=$(echo -n "$ADMIN_ACCOUNT:$ADMIN_API_KEY" | base64)
AuthHeader="Authorization: Basic $basicAuthCredentials"

tenant="bluecorp"
baseURL="http://$tenant.stellar.local:8000"
sdpUIBaseURL="http://$tenant.stellar.local:3000"
ownerEmail="owner@$tenant.org"

curl -X POST http://localhost:8003/tenants \
     -H "Content-Type: application/json" \
     -H "$AuthHeader" \
     -d '{
            "name": "'"$tenant"'",
            "organization_name": "Blue Corp",
            "base_url": "'"$baseURL"'",
            "sdp_ui_base_url": "'"$sdpUIBaseURL"'",
            "owner_email": "'"$ownerEmail"'",
            "owner_first_name": "john",
            "owner_last_name": "doe"
     }'

TODO: check if base_url and sdp_ui_base_url should be removed after https://stellarorg.atlassian.net/browse/SDP-1148 is done.

In the SDP Multi-tenant, certain configurations previously managed through environment variables in the single-tenant setup are now stored within the tenants table in the admin schema. This change allows each tenant to have its own custom configuration.

The field name is important because it's the tenant identifier. The other fields can be set to the same values used on the environment variables used on the non-tenant version.

Importing your data

Now that we've provisioned a new tenant, we can import our data into it. We need to copy our old tables' data to the new tenant schema.

Please notice that the following tables don't need to be copied:

gorp_migrations
auth_migrations
countries
organizations

To import your data from the single-tenant structure, we'll start by removing pre-existing multi-tenant entries that were automatically added when provisioning the tenant:

BEGIN TRANSACTION;

DELETE FROM sdp_<tenant name>.auth_users;
DELETE FROM sdp_<tenant name>.wallets_assets;
DELETE FROM sdp_<tenant name>.assets;
DELETE FROM sdp_<tenant name>.wallets;

COMMIT;

Now we can import the data for the remaining tenant tables. Please notice that some tables cannot be imported directly due to a custom-type conflict. For instance, the status column in the public.payments table depends on public.payment_status enum type, while the one in the sdp_<tenant name>.payments table depends on the sdp_<tenant name>.payment_status enum type. To solve this, we update the type in the source table to match the destination table type:

-- INSERT new data:
BEGIN TRANSACTION;

-- These tables can be copied without changing any types in the source table columns:
INSERT INTO sdp_<tenant name>.wallets SELECT * FROM public.wallets;
INSERT INTO sdp_<tenant name>.assets SELECT * FROM public.assets;
INSERT INTO sdp_<tenant name>.wallets_assets SELECT * FROM public.wallets_assets;
INSERT INTO sdp_<tenant name>.auth_users SELECT * FROM public.auth_users;
INSERT INTO sdp_<tenant name>.receivers SELECT * FROM public.receivers;
INSERT INTO sdp_<tenant name>.auth_user_mfa_codes SELECT * FROM public.auth_user_mfa_codes;
INSERT INTO sdp_<tenant name>.auth_user_password_reset SELECT * FROM public.auth_user_password_reset;

-- These tables need to have the type of some columns changed, we do that with the `ALTER TABLE` directives:
-- NOTE: we're not reverting the types back to the original ones, as the source tables will be dropped after the migration.
ALTER TABLE public.receiver_wallets
	ALTER COLUMN status DROP DEFAULT,
	ALTER COLUMN status TYPE sdp_<tenant name>.receiver_wallet_status
		USING status::text::sdp_<tenant name>.receiver_wallet_status;
INSERT INTO sdp_<tenant name>.receiver_wallets SELECT * FROM public.receiver_wallets;

ALTER TABLE public.receiver_verifications
	ALTER COLUMN verification_field TYPE sdp_<tenant name>.verification_type
		USING verification_field::text::sdp_<tenant name>.verification_type;
INSERT INTO sdp_<tenant name>.receiver_verifications SELECT * FROM public.receiver_verifications;

ALTER TABLE public.disbursements
	ALTER COLUMN status DROP DEFAULT,
	ALTER COLUMN status TYPE sdp_<tenant name>.disbursement_status
		USING status::text::sdp_<tenant name>.disbursement_status;
ALTER TABLE public.disbursements
	ALTER COLUMN verification_field DROP DEFAULT,
	ALTER COLUMN verification_field TYPE sdp_<tenant name>.verification_type
		USING verification_field::text::sdp_<tenant name>.verification_type;
INSERT INTO sdp_<tenant name>.disbursements SELECT * FROM public.disbursements;

ALTER TABLE public.payments
	ALTER COLUMN status DROP DEFAULT,
	ALTER COLUMN status TYPE sdp_<tenant name>.payment_status
		USING status::text::sdp_<tenant name>.payment_status;
INSERT INTO sdp_<tenant name>.payments SELECT * FROM public.payments;

ALTER TABLE public.messages
	ALTER COLUMN status DROP DEFAULT,
	ALTER COLUMN status TYPE sdp_<tenant name>.message_status
		USING status::text::sdp_<tenant name>.message_status;
ALTER TABLE public.messages
	ALTER COLUMN type TYPE sdp_<tenant name>.message_type
		USING type::text::sdp_<tenant name>.message_type;
INSERT INTO sdp_<tenant name>.messages SELECT * FROM public.messages;

COMMIT;

This concludes the SDP tenant data import, so the next step will be to import the Transaction Submitter Service (TSS) data. As a pre-requisite for that, make sure you have exactly one result in from this query and that no fields are empty:

SELECT id, name, distribution_account_address FROM admin.tenants;

Now we can use these fields to import the TSS data:

BEGIN TRANSACTION;

---- 1. add new columns to the transaction_submitter table and populate them
ALTER TABLE public.submitter_transactions
    ADD COLUMN tenant_id VARCHAR(36),
    ADD COLUMN distribution_account_address VARCHAR(56);
WITH SelectedTenant AS (
    SELECT id AS tenant_id, distribution_account_address
    FROM admin.tenants
    LIMIT 1
)
UPDATE public.submitter_transactions SET tenant_id = (SELECT tenant_id FROM SelectedTenant), distribution_account_address = (SELECT distribution_account_address FROM SelectedTenant);

---- 2. copy values to the new table
INSERT INTO tss.submitter_transactions 
SELECT
    id, external_id,
    status::text::tss.transaction_status AS status,
    status_history, status_message, asset_code, asset_issuer, amount, destination, created_at, updated_at, locked_at, started_at, sent_at, completed_at, synced_at, locked_until_ledger_number, stellar_transaction_hash, attempts_count, xdr_sent, xdr_received, tenant_id, distribution_account_address
FROM public.submitter_transactions;

COMMIT;

This concludes the migration of the SDP data to the multi-tenant version. The next step is to drop the old tables that are no longer needed.

Drop Old Tables

Now, the only missing step is to drop the single-tenant tables that should not be in the multi-tenant database:

BEGIN TRANSACTION;

DROP TABLE public.messages CASCADE;
DROP TABLE public.payments CASCADE;
DROP TABLE public.disbursements CASCADE;
DROP TABLE public.receiver_verifications CASCADE;
DROP TABLE public.receiver_wallets CASCADE;
DROP TABLE public.auth_user_password_reset CASCADE;
DROP TABLE public.auth_user_mfa_codes CASCADE;
DROP TABLE public.receivers CASCADE;
DROP TABLE public.auth_users CASCADE;
DROP TABLE public.wallets_assets CASCADE;
DROP TABLE public.assets CASCADE;
DROP TABLE public.wallets CASCADE;
DROP TABLE public.organizations CASCADE;
DROP TABLE public.gorp_migrations CASCADE;
DROP TABLE public.auth_migrations CASCADE;
DROP TABLE public.countries CASCADE;
DROP TABLE public.submitter_transactions CASCADE;
DROP TABLE public.channel_accounts CASCADE;

COMMIT;

Conclusion 🎉

This should conclude the data migration from the single-tenant version to the multi-tenant version of the SDP. Please, make sure to run an e2e test to ensure that everything is working as expected.

Why Migrate?​

Preparing for the Migrations​

Halt the Single-Tenant Version 🚧​

Double-Spending Prevention 🚨​

Remove Channel Accounts 🧹​

Database Backup & Setup 💾​

Changes Explained​

Infrastructure​

Event Broker vs Scheduled Jobs​

Anchor Platform Version​

Environment Variables​

Seggregation of Funds​

CLI Commands​

Database Migrations & Population​

Channel Accounts​

Database Structure​

Step-by-Step Migration Guide​

Deploy the New Version​

Executing Initial Database Migrations​

New Tenant Provisioning Process​

Starting the SDP API server​

Posting to the Admin API​

Importing your data​

Drop Old Tables​

Conclusion 🎉​