Laravel Sharding

Laravel Sharding is a toolkit for distributing data across multiple databases while keeping a familiar Eloquent workflow. The package powers production applications and provides pluggable strategies so each table can select the most appropriate sharding approach. Shards can run on MySQL, PostgreSQL, SQL Server, or SQLite as long as the connections are available to Laravel.

Requirements

• PHP ^8.2
• Laravel 12.x or any framework using Illuminate 12 components
• MySQL, PostgreSQL, SQL Server, or SQLite for shard connections
• Redis (optional) when using the Redis-backed strategy

Installation

Require the package via Composer:

composer require allnetru/laravel-sharding

The service provider is auto-discovered. Publish the configuration and optional migrations with:

php artisan vendor:publish --tag=laravel-sharding-config
php artisan vendor:publish --tag=laravel-sharding-migrations

Run the migrations to create metadata tables used by the built-in strategies:

php artisan migrate

Configuration

1. Define shard connections through the DB_SHARDS environment variable. Each entry follows the format name:host:port:database and multiple shards are separated by semicolons.
2. When preparing to migrate or remove shards, list them in DB_SHARD_MIGRATIONS. New writes are skipped for shards in this list until you finish rebalancing.
3. Review config/sharding.php to map tables to strategies, configure shard groups, and choose ID generators. Every shard-aware model should use the provided Shardable trait.

A minimal example stitches these pieces together:

# .env
DB_SHARDS="shard-1:10.0.0.10:3306:app_shard_1;shard-2:10.0.0.11:3306:app_shard_2;shard-archive:10.0.0.12:3306:app_archive"
DB_SHARD_MIGRATIONS="shard-legacy;shard-archive"

// config/sharding.php
return [
    'default' => 'hash',

    'strategies' => [
        'hash' => Allnetru\Sharding\Strategies\HashStrategy::class,
        'redis' => Allnetru\Sharding\Strategies\RedisStrategy::class,
        'range' => Allnetru\Sharding\Strategies\RangeStrategy::class,
        'db_range' => Allnetru\Sharding\Strategies\DbRangeStrategy::class,
        'db_hash_range' => Allnetru\Sharding\Strategies\DbHashRangeStrategy::class,
    ],

    'id_generator' => [
        'default' => 'snowflake',
        'strategies' => [
            'snowflake' => Allnetru\Sharding\IdGenerators\SnowflakeStrategy::class,
            'sequence' => Allnetru\Sharding\IdGenerators\TableSequenceStrategy::class,
        ],
        'sequence_table' => 'shard_sequences',
        // 'meta_connection' => 'pgsql', // use any connection name configured in database.php
    ],

    'connections' => [
        'shard-1' => ['weight' => 2],
        'shard-2' => ['weight' => 1],
        // 'shard-archive' => ['weight' => 1, 'replica' => true],
    ],

    'replica_count' => 1,

    'tables' => [
        // 'users' => [
        //     'strategy' => 'redis',
        //     'redis_connection' => 'shards',
        //     'redis_prefix' => 'user_shard:',
        //     'group' => 'user_data',
        // ],

        'users' => [
            'strategy' => 'db_hash_range',
            'slot_size' => 250_000,
            'connections' => [
                'shard-1' => ['weight' => 2],
                'shard-2' => ['weight' => 1],
            ],
            'meta_connection' => 'mysql',
            'group' => 'user_data',
        ],

        'profiles' => [
            // inherits the shard selected for the `users` table
            'group' => 'user_data',
            // 'id_generator' => 'sequence',
        ],

        'orders' => [
            'strategy' => 'db_range',
            'connections' => [
                'shard-1' => ['weight' => 2],
                'shard-2' => ['weight' => 1],
            ],
            'range_size' => 50_000,
            'meta_connection' => 'mysql',
            // 'range_table' => 'order_ranges',
        ],

        // 'payments' => [
        //     'strategy' => 'range',
        //     'ranges' => [
        //         ['start' => 1, 'end' => 1_000_000, 'connection' => 'shard-1'],
        //         ['start' => 1_000_001, 'end' => null, 'connection' => 'shard-2'],
        //     ],
        // ],
    ],

    'groups' => [
        'user_data' => ['users', 'profiles', 'orders'],
        // 'billing' => ['payments', 'refunds'],
    ],
];

Update config/database.php to merge the generated shard connections with your base definitions. The examples below use MySQL, but you can swap in any of Laravel's supported drivers:

// config/database.php (excerpt)

use Allnetru\Sharding\Support\Config\Shards;

return [
    'default' => env('DB_CONNECTION', 'mysql'),

    'connections' => array_merge([
        'mysql' => [
            'driver' => 'mysql',
            'url' => env('DB_URL'),
            'host' => env('DB_HOST', '127.0.0.1'),
            'port' => env('DB_PORT', '3306'),
            'database' => env('DB_DATABASE', 'forge'),
            'username' => env('DB_USERNAME', 'forge'),
            'password' => env('DB_PASSWORD', ''),
            // ... keep the rest of your base connection definition
        ],

        // other non-sharded connections...
    ], Shards::databaseConnections(env('DB_SHARDS', ''))),

    // ...
];

Note Passing the DB_SHARDS string ensures shard definitions are available while configuration files are still being evaluated. In other contexts you may call Shards::databaseConnections() without arguments and it will read the DB_SHARDS environment variable directly.

A full walkthrough is available in docs/en/sharding.md.

Usage

Creating sharded tables

Create tables with an unsigned big integer primary key and the is_replica flag to track replicated rows:

Schema::create('items', function (Blueprint $table) {
    $table->unsignedBigInteger('id')->primary();
    $table->boolean('is_replica')->default(false);
    $table->timestamps();
});

Then register the table inside config/sharding.php, select a strategy (hash, redis, range, db_range, or db_hash_range), and list the shard connections the table can use.

ID generation

The default snowflake generator creates sortable 64-bit identifiers laid out as 41 bits timestamp | 10 bits worker | 12 bits sequence. You can switch the global default or override per table to use a database-backed sequence generator or any other configured strategy.

Give every process that mints ids its own worker_id. The sequence counter is per process, so two processes sharing a worker id share a counter space and can produce the same identifier. On one shard that is a primary key violation; across shards it is a silent duplicate that later breaks find() and rebalancing.

# distinct per container, per app server, per queue worker host
SHARDING_WORKER_ID=1

Valid range is 0 to 1023. When the option is unset the worker id is derived from the hostname and the process id, which is adequate for a single node but is a fallback, not a guarantee.

The epoch defaults to 2020-01-01 and 41 bits cover about 69 years from it. Changing SHARDING_EPOCH_MS on a populated database reorders identifiers, so treat it as fixed once data exists.

Grouping related tables

Group tables so records that belong together end up on the same shard:

'groups' => [
    'user_data' => ['users', 'organizations', 'billing', 'transactions'],
],

When models belong to a group they reuse the shard selected for the group's primary table (for example, users).

Colocation needs three things together, and missing the third is the common mistake because nothing fails loudly:

1. the entry in groups, with the owning table listed first;
2. 'group' => 'user_data' in the table's own configuration;
3. protected string $shardKey = 'user_id'; on the child model.

Without the third the child is hashed by its own primary key and lands on an arbitrary shard, so the parent and its children end up apart while everything appears to work.

A child whose shard key is not its primary key must not use an auto-incrementing primary key either: two shards would hand out the same sequence values. Declare public $incrementing = false; and the package generates the key for you.

Working with data

Models using the Shardable trait behave like standard Eloquent models:

$user = User::find(15);

$partners = Organization::where('status', OrganizationStatus::partner)
    ->paginate(50);

Insertions also resolve the target shard automatically. If you omit the primary key the configured ID generator assigns one before the record is saved.

Running under Swoole

When the PHP process is executed inside a Swoole coroutine context (for example, Laravel Octane with the Swoole engine), shard fan-out queries are dispatched concurrently. The package detects the coroutine runtime automatically and uses channels to aggregate results without blocking on each individual shard. When a request is not already inside a coroutine, the dispatcher boots a Swoole\Coroutine::run() scheduler so the queries still run in parallel. No additional configuration is required.

Custom coroutine drivers

The default behaviour can be overridden from config/sharding.php. The coroutines section accepts any class or closure that returns an implementation of Allnetru\Sharding\Support\Coroutine\CoroutineDriver, allowing you to disable coroutines entirely or integrate with an alternative runtime:

'coroutines' => [
    'default' => env('SHARDING_COROUTINE_DRIVER', 'swoole'),
    'drivers' => [
        'swoole' => Allnetru\Sharding\Support\Coroutine\Drivers\SwooleCoroutineDriver::class,
        'sync' => Allnetru\Sharding\Support\Coroutine\Drivers\SyncCoroutineDriver::class,
        'amphp' => App\Sharding\AmpCoroutineDriver::class,
    ],
],

Point the default driver to sync (or set SHARDING_COROUTINE_DRIVER=sync) to keep fan-out queries synchronous. Custom drivers may be resolved through the Laravel container, so you can bind them as singletons or expose factory closures for more advanced scenarios.

Console tooling

Use the bundled Artisan commands to inspect and maintain shards:

• php artisan shards:distribute {model} [{model} ...] [--dry-run] – put existing rows on the shard their own key names, in chunks. One model per table: the tables of a colocation group share a key but not the column it lives in (users.id and user_roles.user_id are the same key under two names), and only the model knows its own. --dry-run counts what is misplaced without moving anything, which is what to run before an upgrade.
• php artisan shards:rebalance {model} [{model} ...] – move the rows of a colocation group between shards with optional --from, --to, --start, and --end filters. Model classes rather than table names, one per table of the group, for the same two reasons shards:distribute takes them: a table name cannot find the model outside App\Models, and only the model knows the column its own shard key lives in. Every populated table of the group has to be named — the routing being handed over belongs to the group, so moving one table would strand the others behind it — and an empty sibling may be left out. --start and --end bound the shard key; a range strategy needs both of them with --to, and db_hash_range refuses either of them with --to, because its slots are hashes of the key rather than ranges of it and a partly moved slot cannot be redirected.
• php artisan shards:migrate – run shard-specific migrations across every configured connection.

Testing

Clone the repository and install dependencies before running the test suite:

composer install
composer test

Contributing

Please review the CONTRIBUTING.md guide for details about our workflow, coding standards, and security policy.

Security

If you discover a security vulnerability, please follow the disclosure process described in CONTRIBUTING.md.

License

Laravel Sharding is open-sourced software licensed under the MIT license.