Background Migrations

Langfuse uses background migrations to perform long-running changes within the storage components when upgrading the application. These may include the addition and backfilling of new columns or the migration of data between storages. Background migrations are executed on startup of the worker container and run in the background until completion or failure.

Next to background migrations, fast migrations are applied directly to the database on startup of the web container.

Monitoring

You can monitor the progress of background migrations within the Langfuse UI. Click on the Langfuse version tag and select “Background Migrations”. You see all migrations that ever ran and their status. You can also monitor the progress of background migrations via the worker container logs.

If migrations are running or have failed, we show a status indicator within the UI to guide users towards the background migrations overview.

Deployment stops

Langfuse does not require deployment stops between minor releases as of now. However, we recommend that you monitor the progress of background migrations after each update to ensure that all migrations have completed successfully before attempting another update. We will highlight within the changelog if a deployment stop becomes required.

Configuration

Background migrations are enabled by default and can be disabled by setting LANGFUSE_ENABLE_BACKGROUND_MIGRATIONS=false. This is not recommended as it may leave the application in an inconsistent state where the UI and API does not reflect the current state of the data correctly.

Troubleshooting

Failed to convert rust String into napi string

If you are seeing the message above for the traces, observations, or scores background migration, it is usually due to large blob data within the rows. The error happens, because the postgres database client tries to concatenate all the data into a single string, before parsing it in Node.js. If the string exceeds the maximum Node.js string size, the error is thrown.

We can circumvent the issue by loading fewer rows at the same time. To do so, adjust the batchSize of your migration by editing the background_migrations table. Add { "batchSize": 2500 } to the args column of the migration you want to adjust. Afterward, restart the migration via the UI.