Other connection information for "Video Upload S3"
Other .env connection information
# Server Configuration
SERVER_PORT=8081
SESSION_SECRET=change-me-in-production
# Video Processing
TEMP_DIR=./temp
UPLOAD_DIR=./uploads
# Upload/Transcode Settings
MAX_CONCURRENT_UPLOADS=2
UPLOAD_QUEUE_SIZE=50
MAX_CONCURRENT_TRANSCODES=2
TRANSCODE_QUEUE_SIZE=50
# S3 Upload Settings
S3_UPLOADER_CONCURRENCY=2
S3_UPLOAD_MAX_RETRIES=3
S3_UPLOAD_BASE_DELAY_MS=500
S3_UPLOAD_MAX_DELAY_MS=8000
# Upload/Download Retry Settings
UPLOAD_DOWNLOAD_MAX_RETRIES=3
UPLOAD_DOWNLOAD_BASE_DELAY_MS=800
UPLOAD_DOWNLOAD_MAX_DELAY_MS=12000
Note clearly:
Configuration source
- API: Read in config/config.go when calling LoadConfig() (file .env or system variables).
- Worker: Read env directly in cmd/worker/main.go for S3 and retry upload groups — not using config.AppConfig.
1. Upload and transcode queue (mainly API)
| Variable | Meaning |
|---|---|
| MAX_CONCURRENT_UPLOADS | Number of workers handling ProcessUpload in parallel. Value ≤ 0 is clamped to minimum 2. |
| UPLOAD_QUEUE_SIZE | Queue buffer depth. Out of space → upload failed (Upload queue is full…. |
| MAX_CONCURRENT_TRANSCODES | Number of HLS transcode workers (ProcessVideoToHLS) running in parallel. ≤ 0 → minimum 2. |
| TRANSCODE_QUEUE_SIZE | Buffer for transcode jobs. |
Suggestion: Increase MAX_* when resources are sufficient; increase *_QUEUE_SIZE to handle bursts (may experience longer wait times).
2. S3 (clearly effective on Worker)
| Variable | Worker | API (web) |
|---|---|---|
| S3_UPLOADER_CONCURRENCY | Exists: Uploader.Concurrency and limits parallel HLS file uploads. | AppConfig is loaded but UploadHLS/UploadFile do not use buildUploader → almost doesn't apply. |
| S3_UPLOAD_MAX_RETRIES | Exists: aws.Config.MaxRetries (SDK). | Helper exists but is not attached to the current upload thread. |
| S3_UPLOAD_BASE_DELAY_MS, S3_UPLOAD_MAX_DELAY_MS | Separate S3 backoff is not used; shared loading uses UPLOAD_DOWNLOAD_*. | In config/helper but not called by UploadHLS/UploadFile. |
In summary: Adjust S3_* in the .env of worker to affect S3 uploads and SDK retries.
3. Retry loading / network (API and Worker)
| Variable | Meaning |
|---|---|
| UPLOAD_DOWNLOAD_MAX_RETRIES | Loop 0…MaxRetries → maximum MaxRetries + 1 attempts (e.g. 3 → 4 times). |
| UPLOAD_DOWNLOAD_BASE_DELAY_MS | Base delay; backoff base × 2^attempt, with a ceiling. |
| UPLOAD_DOWNLOAD_MAX_DELAY_MS | Maximum wait time between attempts. |
- API (upload_service.go): source loading, retry for temporary errors.
- Worker (retryOperation): similar backoff, log Retrying … attempt.
Quick Checklist
- API: parallel and upload/transcode queue; retry source loading.
- Worker: S3_UPLOADER_CONCURRENCY, S3_UPLOAD_MAX_RETRIES; UPLOAD_DOWNLOAD_* for general retry.
- Note: Align MAX_CONCURRENT_TRANSCODES to the correct process and the .env file of the running binary.