Configuring custom AWS S3 storage backends in django-storages

In the article, we will show how to configure custom storage backends in the django-storages package with the AWS S3. We will have one S3 bucket containing the project’s static files, media and private files that are not going to be publicly accessible.

Let’s start with the settings configuration. We assume that the django-storages was successfully installed and it is available in INSTALLED_APPS variable in settings.py. When working with the settings below, make sure that you configure them only for application instance using S3 because probably on your local machine you are not going to use S3 for storing static files.

# settings.py

AWS_ACCESS_KEY_ID = 'REPLACE_GET_FROM_ENV_FILE'

AWS_SECRET_ACCESS_KEY = 'REPLACE_GET_FROM_ENV_FILE'

AWS_STORAGE_BUCKET_NAME = 'REPLACE_BUCKET_NAME'

AWS_S3_REGION = 'eu-central-1'

AWS_S3_CUSTOM_DOMAIN = 's3.{}.amazonaws.com/{}'.format(
    AWS_S3_REGION, AWS_STORAGE_BUCKET_NAME
)

AWS_S3_HOST = 's3.{}.amazonaws.com'.format(AWS_S3_REGION)

AWS_STATIC_LOCATION = 'static'

AWS_PUBLIC_MEDIA_LOCATION = 'media'

AWS_PRIVATE_MEDIA_LOCATION = 'private'

STATIC_URL = 'https://{}/{}/'.format(
    AWS_S3_CUSTOM_DOMAIN, AWS_STATIC_LOCATION
)

MEDIA_URL = 'https://{}/{}/'.format(
    AWS_S3_CUSTOM_DOMAIN, AWS_PUBLIC_MEDIA_LOCATION
)

Now define custom storages. We are having 3 storage backends: public static files (CSS, JavaScript …), public media files (uploaded images like user avatars) and private files that are not publicly accessible.

# storage_backends.py

from django.conf import settings

from storages.backends.s3boto3 import S3Boto3Storage


class StaticStorage(S3Boto3Storage):
    location = settings.AWS_STATIC_LOCATION
    default_acl = 'public-read'


class MediaStorage(S3Boto3Storage):
    location = settings.AWS_PUBLIC_MEDIA_LOCATION
    default_acl = 'public-read'


class PrivateStorage(S3Boto3Storage):
    location = settings.AWS_PRIVATE_MEDIA_LOCATION
    default_acl = 'private'

Please note that for each storage we have to specify the default_acl attribute. In older versions of the django-storages it was by default set to public_read but now we must specify it. For the private storage, we are going to set the value to private to make the files in this storage not publicly accessible. After defining new storages we must set proper storage values in the settings.py

# settings.py

STATICFILES_STORAGE = 'application_directory.storage_backends.StaticStorage'

DEFAULT_FILE_STORAGE = 'application_directory.storage_backends.MediaStorage'

As you can see we didn’t configure any option with the private storage. That’s because we must create a new type of model field that will use the private storage when the file is uploaded.

# fields.py

from django.conf import settings
from django.db import models

from .storage_backends import PrivateStorage


class PrivateFileField(models.FileField):
    def __init__(
        self, verbose_name=None, name=None, upload_to='', storage=None, **kwargs
    ):
        storage = None
        if hasattr(settings, 'AWS_PRIVATE_MEDIA_LOCATION'):
            storage = PrivateStorage()
        super().__init__(verbose_name, name, upload_to, storage, **kwargs)

In the code above we are inheriting the FileField class (you can inherit from ImageField as well) and then a new value for storage is set. Now the question is why we are not passing proper storage backend in storage attribute which we can see in the init method. The reason is that the backend change depending on the current working environment. If we are on the local machine we are probably using default backends and on the production machine, we are uploading files into AWS S3.

After implementing all the mentioned classes our directory structure may look like an example below. Of course, it is up to you how you structure the files. Most of the time we are creating custom Django application which is containing application wide tools and helpers. In this case, we can move fields.py and storage_backends.py files into a separate application.

| application_directory
|- __init__.py
|- fields.py
|- models.py
|- storage_backends.py

Here is the final model using a new PrivateFileField. All files uploaded using this model will be available in the private folder in S3.

# models.py

from django.db import models
from django.utils.translation import ugettext_lazy as _

from .fields import PrivateFileField


class PrivateFile(models.Model):
    some_file = PrivateFileField(_('file'))

After implementing all logic it is possible to upload files into a private directory in S3. Files are not going to be publicly accessible. To serve them, you can write custom CBS (class-based view) which authenticates a user and then serves the proper file.