TurboStream Export Engine

A high-performance Laravel package for exporting 300M+ records with professional PDF support (9000+ pages without breaking). Supports CSV, XLSX, PDF, DOCX, and SQL formats.

Overview

TurboStream Export Engine is designed for Laravel applications that need to export massive datasets (300M+ records) efficiently without memory issues. It uses chunked queries to process data in batches and leverages Laravel Queues with Redis for background processing.

Key Features

• 300M+ Records Support: Optimized chunk sizes for massive datasets (10K-20K per chunk)
• 9000+ Page PDFs: Professional PDF generation without breaking
• Memory Efficient: Uses cursor() instead of chunk() to stream records without loading all into memory
• 5 Export Formats: CSV, XLSX, PDF, DOCX, SQL
• PDF Full Features: Subtotals, grand totals, colspan, rowspan, custom headers
• Async Processing: Background jobs via Laravel Queues with Redis
• Real-time Progress: Track export progress via Redis cache
• Filter Names in Filename: Downloaded files include applied filters in filename
• Multiple Queue Drivers: Redis (recommended), Database, or Sync
• Auto Chunk Sizing: Automatically adjusts chunk size based on data volume
• Laravel Native: Integrates seamlessly with Laravel 9, 10, 11, 12, and 13

Requirements

• PHP 8.1+
• Laravel 9.0, 10.0, 11.0, 12.0, or 13.0
• Redis (recommended) or Database queue driver
• ext-json, ext-mbstring

Dependencies (All Included)

Installation

1. Install via Composer

composer require turbostream/export-engine

2. Publish Configuration

php artisan vendor:publish --tag=turbo-export

3. Configure Environment

Add Redis configuration to your .env file:

CACHE_DRIVER=redis
QUEUE_CONNECTION=redis
REDIS_HOST=127.0.0.1
REDIS_PORT=6379

4. Run Migrations (optional - for database queue)

php artisan migrate

Quick Start

Basic Usage

use TurboStreamExport\Facades\ExportFacade;
use App\Models\User;

$exportId = ExportFacade::createExport([
    'model' => User::class,
    'columns' => ['id', 'name', 'email', 'created_at'],
    'format' => 'csv',
    'filename' => 'users_export',
]);

Using Filters

$exportId = ExportFacade::createExport([
    'model' => \App\Models\User::class,
    'columns' => ['id', 'name', 'email', 'status'],
    'format' => 'csv',
    'filters' => [
        ['status', '=', 'active'],
        ['created_at', '>=', '2026-01-01'],
    ],
]);

Filename with filters: When filters are applied, the downloaded file includes filter details:

users_export_filtered_status=_active_created_at=_>=_2026-01-01.csv

Check Progress

$progress = ExportFacade::getProgress($exportId);

echo $progress['progress'];      // 75
echo $progress['status'];        // 'processing' or 'completed'
echo $progress['filters'];      // Array of applied filters
echo $progress['filter_summary']; // 'status=_active'

Download File

$downloadUrl = ExportFacade::getDownloadUrl($exportId);
// Returns signed URL valid for 1 hour

Export Formats

This package supports 5 export formats for different use cases:

┌─────────────────────────────────────────────────────────────────────────────┐
│                        5 EXPORT FORMATS SUPPORTED                            │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐      │
│  │    CSV   │  │   XLSX   │  │   PDF    │  │   DOCX   │  │   SQL    │      │
│  ├──────────┤  ├──────────┤  ├──────────┤  ├──────────┤  ├──────────┤      │
│  │    📄    │  │    📊    │  │    📑    │  │    📝    │  │  🗄️     │      │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘      │
│       │             │             │             │             │          │
│       ▼             ▼             ▼             ▼             ▼          │
│  Streaming      Memory-       TCPDF          Memory-       Streaming     │
│  (300M+ OK)     based         (Full Features)based         (300M+ OK)     │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

CSV (Recommended for Large Data)

┌────────────────────────────────────────────────────────┐
│  Format:  comma-separated-values.csv                   │
│  Speed:   ⚡⚡⚡⚡⚡ (FASTEST)                            │
│  Memory:  ~1GB (constant - streaming)                  │
│  Best for: 300M+ records, data backup, API export     │
│  Install:  ✅ Included (League CSV)                   │
└────────────────────────────────────────────────────────┘

XLSX (Excel)

┌────────────────────────────────────────────────────────┐
│  Format:  spreadsheet.xlsx                             │
│  Speed:   ⚡⚡⚡ (moderate)                             │
│  Memory:  ~2GB (grows with records)                   │
│  Best for: Reports, sharing, printing                 │
│  Install:  ✅ Included (PhpSpreadsheet)               │
└────────────────────────────────────────────────────────┘

PDF (Professional Documents)

┌────────────────────────────────────────────────────────┐
│  Format:  document.pdf                                 │
│  Speed:   ⚡⚡ (slower)                                 │
│  Memory:  ~2-4GB (grows with records)                 │
│  Best for: Invoices, reports, 9000+ page documents   │
│                                                        │
│  Features (all included):                             │
│  ✅ Subtotals per group   ✅ Grand Total              │
│  ✅ Colspan & Rowspan     ✅ Custom Headers/Footers   │
│  ✅ 9000+ pages           ✅ Professional layouts    │
│  Install:  ✅ TCPDF included (no extra package)        │
└────────────────────────────────────────────────────────┘

DOCX (Word)

┌────────────────────────────────────────────────────────┐
│  Format:  document.docx                                │
│  Speed:   ⚡⚡ (slower)                                 │
│  Memory:  ~2GB (grows with records)                   │
│  Best for: Documentation, letters, contracts          │
│  Install:  ✅ Included (PhpWord)                       │
└────────────────────────────────────────────────────────┘

SQL (Database)

┌────────────────────────────────────────────────────────┐
│  Format:  dump.sql                                     │
│  Speed:   ⚡⚡⚡⚡⚡ (fast)                              │
│  Memory:  ~1GB (constant - streaming)                  │
│  Best for: DB backup, migration, re-import            │
│  Install:  ✅ Built-in (no extra package)             │
└────────────────────────────────────────────────────────┘

Quick Decision Guide

┌────────────────────────────────────────────────────────────────────┐
│                    WHICH FORMAT TO CHOOSE?                         │
├────────────────────────────────────────────────────────────────────┤
│                                                                    │
│  How many records?                                                 │
│                                                                    │
│  ┌─────────────┐                                                  │
│  │  100M+     │ ──────▶  CSV (streaming, memory efficient)       │
│  └─────────────┘                                                  │
│                                                                    │
│  ┌─────────────┐                                                  │
│  │  1M - 100M  │ ──────▶  CSV (recommended)                      │
│  └─────────────┘         or SQL                                    │
│                                                                    │
│  ┌─────────────┐                                                  │
│  │  < 1M       │ ──────▶  Any format (CSV/XLSX/PDF/DOCX/SQL)     │
│  └─────────────┘                                                  │
│                                                                    │
│  ─────────────────────────────────────────────────────────         │
│                                                                    │
│  What is the purpose?                                              │
│                                                                    │
│  • Data backup/migration ──▶ CSV or SQL                           │
│  • Excel report ──────────▶ XLSX                                   │
│  • Print document ────────▶ PDF                                    │
│  • Word document ────────▶ DOCX                                   │
│  • Share with non-tech ──▶ XLSX                                    │
│                                                                    │
└────────────────────────────────────────────────────────────────────┘

API Reference

Create Export

ExportFacade::createExport(array $config);

Configuration Options:

Example:

$exportId = ExportFacade::createExport([
    'model' => \App\Models\Transaction::class,
    'columns' => ['id', 'amount', 'status', 'created_at'],
    'format' => 'xlsx',
    'filename' => 'transactions_report',
    'filters' => [
        ['status', '=', 'completed'],
        ['amount', '>', 100],
        ['created_at', '>=', '2026-01-01'],
        ['category_id', 'IN', [1, 2, 3]],
    ],
]);

Get Progress

ExportFacade::getProgress(string $exportId): array;

Returns:

{
    "export_id": "550e8400-e29b-41d4-a716-446655440000",
    "progress": 75,
    "total": 100000000,
    "status": "processing",
    "file_path": null,
    "filters": [
        ["status", "=", "completed"],
        ["created_at", ">=", "2026-01-01"]
    ],
    "filter_summary": "status=_completed_created_at=_>=_2026-01-01",
    "updated_at": "2026-03-19T10:30:00Z"
}

When completed:

{
    "export_id": "550e8400-e29b-41d4-a716-446655440000",
    "progress": 100,
    "total": 100000000,
    "status": "completed",
    "file_path": "exports/transactions_report_filtered_status=_completed.csv",
    "filters": [["status", "=", "completed"]],
    "filter_summary": "status=_completed",
    "updated_at": "2026-03-19T12:30:00Z"
}

Get Download URL

ExportFacade::getDownloadUrl(string $exportId, int $minutes = 60): string;

Returns a signed URL valid for the specified duration.

List Exports

ExportFacade::listExports(int $limit = 10): array;

Delete Export

ExportFacade::deleteExport(string $exportId): bool;

PDF Export (300M+ Records, 9000+ Pages)

This package uses TCPDF with advanced features for professional PDF generation:

┌─────────────────────────────────────────────────────────────────────────┐
│                    PDF EXPORT CAPABILITIES                              │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│  ✅ 300M+ Records Support    │  ✅ Colspan & Rowspan                    │
│  ✅ 9000+ Pages Without Break│  ✅ Auto Subtotals                       │
│  ✅ Grand Total              │  ✅ Custom Headers/Footers              │
│  ✅ Memory Optimized        │  ✅ Professional Layouts               │
│                                                                         │
│  No additional packages needed - TCPDF included by default!           │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘

When to Use PDF

PDF Features

use TurboStreamExport\Contracts\Drivers\PdfExportDriver;

$driver = new PdfExportDriver();

// Set report information
$driver->setReportInfo('Annual Financial Report 2021-2026', [
    'company' => 'ABC Corporation',
    'start_date' => '2021-01-01',
    'end_date' => '2026-03-31',
]);

// Enable automatic subtotals by department
$driver->setGroupBy('department');

// Define columns with numeric formatting
$columns = ['id', 'employee', 'department', 'year', 'basic', 'allowances', 'gross', 'tax', 'net'];
$driver->setNumericColumns(['basic', 'allowances', 'gross', 'tax', 'net']);

// Write header
$driver->writeHeader($columns);

// Add custom row with colspan for year header
$driver->addCustomRow([
    0 => [
        'TEXT' => '═══ YEAR 2026 ═══',
        'COLSPAN' => 9,
        'STYLE' => 'subtotal',
        'FONT_WEIGHT' => 'B',
        'TEXT_ALIGN' => 'C',
        'BACKGROUND_COLOR' => [52, 152, 219],
    ]
]);

// Write data rows
foreach ($employees as $employee) {
    $driver->writeRow([...]);
}

// Grand total added automatically at end
$driver->finalize($filePath);

PDF Methods Reference

Real-World PDF Examples

1. Employee Salary Report with Subtotals by Department

$driver = new PdfExportDriver();
$driver->setReportInfo('Employee Salary Report', [
    'department' => 'All',
    'month' => 'March 2026',
]);

$driver->setGroupBy('department');
$driver->setNumericColumns(['basic_salary', 'house_rent', 'medical', 'gross', 'net']);

$columns = ['id', 'name', 'department', 'basic_salary', 'house_rent', 'medical', 'gross', 'net'];
$driver->writeHeader($columns);

foreach ($employees as $emp) {
    $driver->writeRow([
        $emp->id,
        $emp->name,
        $emp->department,
        $emp->basic_salary,
        $emp->house_rent,
        $emp->medical,
        $emp->gross_salary,
        $emp->net_salary,
    ]);
}

$driver->finalize(storage_path('app/exports/salary_report.pdf'));
// Output: Department subtotals + Grand total at end

2. Sales Report with Monthly Section Headers (Colspan)

$driver = new PdfExportDriver();
$driver->setReportInfo('Quarterly Sales Report Q1 2026', [
    'quarter' => 'Q1',
    'year' => 2026,
]);

$driver->setGroupBy('month');
$columns = ['product', 'jan', 'feb', 'mar', 'total'];
$driver->setNumericColumns(['jan', 'feb', 'mar', 'total']);
$driver->writeHeader($columns);

foreach ($months as $month) {
    // Add month header with colspan
    $driver->addCustomRow([
        0 => [
            'TEXT' => "━━━ $month ━━━",
            'COLSPAN' => 5,
            'STYLE' => 'header',
            'TEXT_ALIGN' => 'C',
            'BACKGROUND_COLOR' => [41, 128, 185],
        ]
    ]);
    
    foreach ($products as $product) {
        $driver->writeRow([...]);
    }
}

$driver->finalize($filePath);

3. Inventory Report with Rowspan for Categories

$driver = new PdfExportDriver();
$driver->setReportInfo('Inventory Status Report');

$columns = ['category', 'product', 'quantity', 'value'];
$driver->writeHeader($columns);

$currentCategory = '';
foreach ($inventory as $item) {
    if ($item->category !== $currentCategory) {
        // First item of category - use rowspan
        $driver->addCustomRow([
            0 => [
                'TEXT' => $item->category,
                'ROWSPAN' => $categoryCount[$item->category],
                'STYLE' => 'header',
                'BACKGROUND_COLOR' => [149, 165, 166],
            ],
            1 => ['TEXT' => $item->name],
            2 => ['TEXT' => $item->quantity, 'TEXT_ALIGN' => 'R'],
            3 => ['TEXT' => number_format($item->value), 'TEXT_ALIGN' => 'R'],
        ]);
        $currentCategory = $item->category;
    } else {
        $driver->writeRow(['', $item->name, $item->quantity, $item->value]);
    }
}

$driver->finalize($filePath);

Memory Optimization for Large PDFs

For 9000+ page PDFs with 300M+ records:

// In config/turbo-export.php
return [
    'memory_limit_pdf' => '4G',  // Increase for large PDFs
    
    // For very large exports, process in chunks
    'large_data_chunk_size' => 20000,
    'log_progress_interval' => 500000,
];

Cell Configuration Options

// Write data rows foreach ($employees as $employee) { $driver->writeRow([ $employee->id, $employee->name, $employee->department, $employee->basic_salary, $employee->house_rent, $employee->medical, $employee->gross_salary, $employee->deductions, $employee->net_salary, ]); }

// Add custom section header with colspan $driver->addCustomRow([ 0 => [ 'TEXT' => '★ Report Summary - Last 5 Years Financial Data ★', 'COLSPAN' => 9, 'STYLE' => 'subtotal', 'FONT_WEIGHT' => 'B', 'TEXT_ALIGN' => 'C', 'BACKGROUND_COLOR' => [68, 114, 196], ] ]);

// Grand total added automatically at end

// Finalize and save $filePath = storage_path('app/exports/salary_report.pdf'); $driver->finalize($filePath);

### Methods Reference

#### setReportInfo(string $name, array $filters = [])

Set the report title and filters to be displayed.

```php
$driver->setReportInfo('Monthly Sales Report', [
    'start_date' => '2026-01-01',
    'end_date' => '2026-03-31',
    'region' => 'Dhaka'
]);

setGroupBy(string $column)

Enable automatic subtotals when a column value changes.

// Subtotals will be added automatically when department changes
$driver->setGroupBy('department');

// Can also group by year, category, region, etc.
$driver->setGroupBy('year');

setNumericColumns(array $columns)

Define which columns contain numeric values (for formatting).

$driver->setNumericColumns([
    'basic_salary', 
    'house_rent', 
    'gross_salary', 
    'net_salary'
]);

addCustomRow(array $cellData)

Add a custom row with full control over each cell.

// Section header spanning all columns
$driver->addCustomRow([
    0 => [
        'TEXT' => 'Quarterly Summary - Q1 2026',
        'COLSPAN' => 8,
        'STYLE' => 'subtotal',
        'FONT_WEIGHT' => 'B',
        'TEXT_ALIGN' => 'C',
    ]
]);

// Custom row with specific cell values
$driver->addCustomRow([
    0 => ['TEXT' => 'Section A', 'STYLE' => 'header'],
    1 => ['TEXT' => '', 'STYLE' => 'header'],
    2 => ['TEXT' => 'Total Amount', 'STYLE' => 'header', 'TEXT_ALIGN' => 'R'],
    3 => ['TEXT' => '1,234,567', 'STYLE' => 'subtotal', 'TEXT_ALIGN' => 'R'],
    4 => ['TEXT' => '', 'STYLE' => 'header'],
]);

addColspanRow(array $data, int $colspan, string $text, string $style = 'subtotal')

Add a row with a cell spanning multiple columns.

// Create a row where first 3 columns are merged
$driver->addColspanRow(
    ['', '', '', 'Value 1', 'Value 2'],  // Cell data
    3,                                     // Span 3 columns
    'Merged Header Text',                   // Text for merged cell
    'subtotal'                             // Style
);

addEmptyRow()

Add a blank row for visual separation.

$driver->addEmptyRow();

Cell Configuration Options

Each cell in addCustomRow() supports these options:

Example: Financial Report with All Features

use TurboStreamExport\Contracts\Drivers\PdfExportDriver;

$driver = new PdfExportDriver();
$driver->setReportInfo('Annual Financial Report 2021-2026', [
    'start_date' => '2021-01-01',
    'end_date' => '2026-03-31',
    'company' => 'ABC Corporation'
]);

// Group by department for subtotals
$driver->setGroupBy('department');

// Define columns with numeric ones
$columns = ['id', 'employee', 'department', 'year', 'basic', 'allowances', 'gross', 'tax', 'net'];
$driver->setNumericColumns(['basic', 'allowances', 'gross', 'tax', 'net']);

// Write header
$driver->writeHeader($columns);

// Data by year and department
$years = [2021, 2022, 2023, 2024, 2025, 2026];
$departments = ['HR', 'IT', 'Finance', 'Operations', 'Marketing'];

foreach ($years as $year) {
    // Year header with colspan
    $driver->addCustomRow([
        0 => [
            'TEXT' => "═══ YEAR $year ═══",
            'COLSPAN' => 9,
            'STYLE' => 'subtotal',
            'FONT_WEIGHT' => 'B',
            'TEXT_ALIGN' => 'C',
            'BACKGROUND_COLOR' => [52, 152, 219],
        ]
    ]);
    
    foreach ($departments as $dept) {
        // Write employee rows (subtotals added automatically when department changes)
        foreach ($employees as $emp) {
            if ($emp->department === $dept && $emp->year === $year) {
                $driver->writeRow([
                    $emp->id,
                    $emp->name,
                    $emp->department,
                    $emp->year,
                    $emp->basic,
                    $emp->allowances,
                    $emp->gross,
                    $emp->tax,
                    $emp->net,
                ]);
            }
        }
    }
}

// Grand total row added automatically at the end
$filePath = storage_path('app/exports/financial_report.pdf');
$driver->finalize($filePath);

Configuration

Publish the configuration file:

php artisan vendor:publish --tag=turbo-export --force

config/turbo-export.php

return [
    'disk' => env('EXPORT_DISK', 'local'),
    
    'chunk_size' => env('EXPORT_CHUNK_SIZE', 5000),
    
    'large_data_chunk_size' => env('EXPORT_LARGE_DATA_CHUNK_SIZE', 10000),
    
    'queue' => env('EXPORT_QUEUE', 'exports'),
    
    'retention_hours' => env('EXPORT_RETENTION_HOURS', 24),
    
    'max_records' => env('EXPORT_MAX_RECORDS', 100000000),
    
    'formats' => [
        'csv',
        'xlsx',
        'pdf',
        'docx',
        'sql',
    ],
    
    'default_format' => env('EXPORT_DEFAULT_FORMAT', 'csv'),
    
    // Default memory limit for CSV/SQL (streaming exports)
    'memory_limit' => env('EXPORT_MEMORY_LIMIT', '1G'),
    
    // Format-specific memory limits for memory-based exports
    'memory_limit_xlsx' => env('EXPORT_MEMORY_LIMIT_XLSX', '2G'),
    'memory_limit_pdf' => env('EXPORT_MEMORY_LIMIT_PDF', '2G'),
    'memory_limit_docx' => env('EXPORT_MEMORY_LIMIT_DOCX', '2G'),
    
    'batch_commit_size' => env('EXPORT_BATCH_COMMIT_SIZE', 50000),
    
    'include_filter_in_filename' => env('EXPORT_INCLUDE_FILTER_IN_FILENAME', true),
    
    'download_expiry_minutes' => env('EXPORT_DOWNLOAD_EXPIRY_MINUTES', 60),
    
    'drivers' => [
        'csv' => \TurboStreamExport\Contracts\Drivers\CsvExportDriver::class,
        'xlsx' => \TurboStreamExport\Contracts\Drivers\XlsxExportDriver::class,
        'pdf' => \TurboStreamExport\Contracts\Drivers\PdfExportDriver::class,
        'docx' => \TurboStreamExport\Contracts\Drivers\DocxExportDriver::class,
        'sql' => \TurboStreamExport\Contracts\Drivers\SqlExportDriver::class,
    ],
    
    'large_data_threshold' => env('EXPORT_LARGE_DATA_THRESHOLD', 1000000),
    
    'enable_progress_logging' => env('EXPORT_PROGRESS_LOGGING', true),
    
    'log_progress_interval' => env('EXPORT_LOG_PROGRESS_INTERVAL', 100000),
];

Memory Management

The package uses cursor() instead of chunk() to stream database records without loading all into memory:

Testing Package

For a complete Laravel application demonstrating the export engine in action, see the testing package:

Testing App: https://github.com/moshiur1412/turbo-export-app

This testing package includes:

• Full Laravel application setup
• Example models and controllers
• Export API routes and views
• Queue worker configuration
• Sample data generators

Testing

# Run all tests
composer test

# Run with coverage
composer test:coverage

# Run unit tests only
composer test:unit

# Run feature tests only
composer test:feature

# Run large data tests (memory/performance)
composer test:large

# Or using Pest directly
./vendor/bin/pest
./vendor/bin/pest --coverage
./vendor/bin/pest tests/Unit
./vendor/bin/pest tests/LargeData

Test Coverage

Writing Tests

use TurboStreamExport\Contracts\Drivers\CsvExportDriver;
use TurboStreamExport\Services\ExportService;

class ExportServiceTest extends TestCase
{
    public function test_csv_driver_returns_correct_format(): void
    {
        $driver = new CsvExportDriver();
        
        $this->assertEquals('csv', $driver->getFormat());
        $this->assertEquals('text/csv', $driver->getContentType());
        $this->assertEquals('csv', $driver->getFileExtension());
    }
    
    public function test_filter_summary_generation(): void
    {
        $service = new ExportService('local', [new CsvExportDriver()]);
        
        $filters = [
            ['status', '=', 'active'],
            ['category_id', '=', '5'],
        ];
        
        // Access private method via reflection
        $reflection = new \ReflectionClass($service);
        $method = $reflection->getMethod('buildFilterSummary');
        $method->setAccessible(true);
        
        $summary = $method->invoke($service, $filters);
        
        $this->assertStringContainsString('status', $summary);
        $this->assertStringContainsString('active', $summary);
    }
}

Large Data Testing

public function test_csv_driver_handles_large_batch(): void
{
    $driver = new CsvExportDriver();
    $filePath = $this->tempDir . '/large_export.csv';
    $handle = fopen($filePath, 'w');
    
    $columns = ['id', 'name', 'email', 'status', 'created_at'];
    $driver->writeHeader($columns, $handle);
    
    $recordCount = 100000;
    $startTime = microtime(true);
    
    for ($i = 1; $i <= $recordCount; $i++) {
        $driver->writeRow([
            $i,
            "User Name $i",
            "user$i@example.com",
            $i % 2 === 0 ? 'active' : 'inactive',
            '2026-01-15 10:30:00',
        ], $handle);
    }
    
    $driver->finalize($handle, $filePath);
    
    $duration = microtime(true) - $startTime;
    
    $this->assertFileExists($filePath);
    $this->assertLessThan(30, $duration, 'Large batch export took too long');
}

Architecture

Export Process Flow

┌─────────────────────────────────────────────────────────────────────┐
│                        USER REQUEST                                 │
│  ExportFacade::createExport([                                       │
│    'model' => User::class,                                         │
│    'format' => 'csv',                                               │
│    'filters' => [['status', '=', 'active']]                        │
│  ])                                                                  │
└────────────────────────────────┬────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                         VALIDATION                                   │
│  • Validate model exists                                           │
│  • Check columns exist                                             │
│  • Validate format (csv/xlsx/pdf/docx/sql)                        │
└────────────────────────────────┬────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                      DISPATCH JOB                                   │
│  ProcessExportJob dispatched to 'exports' queue                   │
│  Job ID = exportId (UUID)                                          │
└────────────────────────────────┬────────────────────────────────────┘
                                 │  async
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    QUEUE WORKER (Redis)                             │
│                                                                     │
│  ┌──────────────┐   ┌──────────────┐   ┌──────────────┐          │
│  │ Worker 1     │   │ Worker 2     │   │ Worker N     │          │
│  │ ProcessExport│   │ ProcessExport│   │ ProcessExport│          │
│  └──────────────┘   └──────────────┘   └──────────────┘          │
└────────────────────────────────┬────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                      CHUNK PROCESSING                                │
│                                                                     │
│   Auto Chunk Sizing:                                                │
│   ┌─────────────┬─────────────┬─────────────┬─────────────┐      │
│   │   < 1M      │  1M - 10M   │ 10M - 100M  │   100M+     │      │
│   │  5,000 rec  │  10,000 rec │ 15,000 rec  │ 20,000 rec  │      │
│   └─────────────┴─────────────┴─────────────┴─────────────┘      │
│                                                                     │
│   ┌─────────────┐   ┌─────────────┐   ┌─────────────┐            │
│   │   cursor()  │──▶│   Write     │──▶│   Progress  │            │
│   │  (stream)   │   │   Batch     │   │   Update   │            │
│   └─────────────┘   └─────────────┘   └─────────────┘            │
│        │                 │                 │                      │
│        ▼                 ▼                 ▼                      │
│   ┌─────────────┐   ┌─────────────┐   ┌─────────────┐            │
│   │  Memory     │   │   File      │   │    Redis    │            │
│   │  Efficient  │   │   Output    │   │    Cache    │            │
│   └─────────────┘   └─────────────┘   └─────────────┘            │
└────────────────────────────────┬────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                      FILE GENERATED                                 │
│                                                                     │
│   exports/users_export_filtered_status=_active_2026-01-01.csv     │
│                                                                     │
└────────────────────────────────┬────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                      DOWNLOAD READY                                  │
│                                                                     │
│   ExportFacade::getDownloadUrl($exportId)                          │
│   → Signed URL (valid 1 hour)                                      │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

5 Export Formats - When to Use Which

┌────────────────────────────────────────────────────────────────────────────┐
│                        CHOOSE YOUR FORMAT                                  │
└────────────────────────────────┬───────────────────────────────────────────┘
                                 │
        ┌────────────────────────┼────────────────────────────────┐
        │                        │                                │
        ▼                        ▼                                ▼
┌───────────────┐      ┌─────────────────┐           ┌─────────────────┐
│   CSV         │      │   XLSX          │           │   SQL           │
│   (FASTEST)   │      │   (Excel)       │           │   (Database)    │
├───────────────┤      ├─────────────────┤           ├─────────────────┤
│ Best for:     │      │ Best for:       │           │ Best for:       │
│ • 100M+ rec  │      │ • Reports       │           │ • DB Migration  │
│ • Data backup │      │ • Sharing       │           │ • Backup        │
│ • API export  │      │ • Printing      │           │ • Re-import     │
├───────────────┤      ├─────────────────┤           ├─────────────────┤
│ Memory: 1GB   │      │ Memory: 2GB     │           │ Memory: 1GB     │
│ Stream: YES   │      │ Stream: NO      │           │ Stream: YES     │
└───────────────┘      └─────────────────┘           └─────────────────┘
        │                        │                                │
        └────────────────────────┼────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────┐      ┌─────────────────┐
│   PDF           │      │   DOCX          │
│   (Professional) │      │   (Word)        │
├─────────────────┤      ├─────────────────┤
│ Best for:       │      │ Best for:       │
│ • 9000+ pages   │      │ • Documentation │
│ • Invoices      │      │ • Letters       │
│ • Reports       │      │ • Contracts     │
├─────────────────┤      ├─────────────────┤
│ TCPDF: Full     │      │ Memory: 2GB      │
│ Memory: 2-4GB   │      │ Stream: NO      │
│ Stream: NO      │      └─────────────────┘
└─────────────────┘

Memory Usage by Format

┌─────────────────────────────────────────────────────────────────────────┐
│                        MEMORY EFFICIENCY                                │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│  CSV / SQL              ████████████████░░░░░░░░░░░  ~1GB (Streaming)  │
│                                                                         │
│  XLSX                   █████████████████████████░░  ~2GB               │
│                                                                         │
│  PDF (TCPDF)           █████████████████████████░░  ~2-4GB             │
│                                                                         │
│  DOCX                  █████████████████████████░░  ~2GB                │
│                                                                         │
├─────────────────────────────────────────────────────────────────────────┤
│  KEY INSIGHT:                                                           │
│  • CSV/SQL use STREAMING - constant memory (best for 100M+)            │
│  • XLSX/PDF/DOCX load ALL data - memory grows with records             │
│  • For 100M+ records, ALWAYS use CSV!                                   │
└─────────────────────────────────────────────────────────────────────────┘

Chunk Size Strategy

Directory Structure

turbostream/export-engine/
├── src/
│   ├── Contracts/
│   │   ├── ExportDriverInterface.php
│   │   ├── ExportableInterface.php
│   │   └── Drivers/
│   │       ├── CsvExportDriver.php
│   │       ├── XlsxExportDriver.php
│   │       ├── PdfExportDriver.php       # Unified: simple, subtotals, colspan, 100M+ records
│   │       ├── DocxExportDriver.php
│   │       └── SqlExportDriver.php
│   ├── Facades/
│   │   └── ExportFacade.php
│   ├── Http/Controllers/
│   │   └── ExportController.php
│   ├── Jobs/
│   │   └── ProcessExportJob.php
│   ├── Providers/
│   │   └── TurboStreamExportServiceProvider.php
│   └── Services/
│       └── ExportService.php
├── config/
│   └── turbo-export.php
├── tests/
│   ├── Unit/
│   │   ├── ExportDriverTest.php
│   │   ├── ExportServiceTest.php
│   │   └── ProcessExportJobTest.php
│   ├── Feature/
│   │   └── ExportServiceFeatureTest.php
│   └── LargeData/
│       └── LargeDataExportTest.php
├── routes/
│   └── api.php
├── composer.json
├── phpunit.xml
└── README.md

Queue Workers

Start Queue Worker

php artisan queue:work redis --queue=exports

High Priority Queue

For urgent exports:

php artisan queue:work redis --queue=exports-high,exports

Production Setup (Supervisor)

[program:export-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /path/to/artisan queue:work redis --queue=exports --sleep=3 --tries=3
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=4
redirect_stderr=true
stdout_logfile=/var/log/export-worker.log

Extending the Package

Custom Export Driver

Create your own driver by implementing ExportDriverInterface:

<?php

namespace App\Export\Drivers;

use TurboStreamExport\Contracts\ExportDriverInterface;

class CustomExportDriver implements ExportDriverInterface
{
    public function getFormat(): string
    {
        return 'custom';
    }

    public function getContentType(): string
    {
        return 'application/custom';
    }

    public function getFileExtension(): string
    {
        return 'ext';
    }

    public function writeHeader(array $columns, $handle): void
    {
        // Write header
    }

    public function writeRow(array $data, $handle): void
    {
        // Write row
    }

    public function writeBatch($records, array $columns, $handle): void
    {
        // Write batch
    }

    public function finalize($handle, string $filePath): string
    {
        // Finalize
        return $filePath;
    }
}

Register Custom Driver

// In your ServiceProvider
use TurboStreamExport\Facades\ExportFacade;

ExportFacade::extendDriver('custom', CustomExportDriver::class);

Troubleshooting

"Class Redis not found"

Install the Redis PHP extension or use predis:

composer require predis/predis

Queue Issues

Export stuck in "processing"

1. Check if queue worker is running:

php artisan queue:work redis --queue=exports

2. Check Laravel logs for errors:

tail -f storage/logs/laravel.log

3. Verify Redis connection:

php artisan tinker
Redis::ping();

Clear stuck queue jobs

If jobs are stuck in the queue or failed:

# Clear all pending jobs from exports queue
php artisan queue:clear --queue=exports

# Retry failed jobs
php artisan queue:retry

# Delete all failed jobs
php artisan queue:flush

# View failed jobs
php artisan queue:failed

Queue worker not processing

If the worker is running but jobs aren't being processed:

1. Check Redis connection in .env:

REDIS_HOST=127.0.0.1
REDIS_PORT=6379
QUEUE_CONNECTION=redis

2. Restart the worker:

# Stop existing worker (Ctrl+C)
# Start fresh worker
php artisan queue:work redis --queue=exports

3. For high-volume exports, run worker with verbose output:

php artisan queue:work redis --queue=exports -vvv

Memory issues with large exports

For XLSX/PDF/DOCX with large datasets, increase memory limits:

# Default for CSV/SQL (streaming)
EXPORT_MEMORY_LIMIT=1G

# Higher for XLSX/PDF/DOCX (memory-based)
EXPORT_MEMORY_LIMIT_XLSX=2G
EXPORT_MEMORY_LIMIT_PDF=2G
EXPORT_MEMORY_LIMIT_DOCX=2G

The XLSX driver now uses running totals instead of storing all records in memory, reducing memory usage significantly.

Performance issues

For 100M+ records:

EXPORT_LARGE_DATA_CHUNK_SIZE=20000
EXPORT_LOG_PROGRESS_INTERVAL=500000

Security

If you discover security vulnerabilities, please email moshiur.bro@gmail.com instead of using the public issue tracker.

License

The TurboStream Export Engine is open-sourced software licensed under the MIT license.

Changelog

See CHANGELOG.md for version history.

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.