Confirming password while registering

To add the confirm_password field when registering a new user, the simplest way seems to be to define another serializer RegisterUserSerializer that inherits the UserSerializer:

class RegisterUserSerializer(UserSerializer):
    '''Serializer for registering a new user'''
    confirm_password = serializers.CharField(
        style={'input_type': 'password'},
        write_only=True
    )

    def validate(self, data):
        '''Validate that password and confirm_password are the same'''
        if not data['password'] == data['confirm_password']:
            raise serializers.ValidationError('Passwords are not matching.')
        return data

    class Meta(UserSerializer.Meta):
        model = User
        fields = ['username', 'password', 'confirm_password', 'is_active']

A new field confirm_password has been defined along with a validate method that performs object level checking on the serializer to ensure that the password and confirm_password field are identical. The view method will now use request.data to create a RegisterUserSerializer instance instead of UserSerializer. The only problem is that the error for passwords not matching is not very nice – “non_field_errors – Passwords not matching”. This is because errors found on the object using the validate method are sent out in the dictionary with non_field_errors key. This means that the error messages will need to be customized rather than processed in the view method.

class Meta:
    model = User
    fields = ['username', 'password', 'is_active']
    read_only_fields = ['is_active', ]
    extra_kwargs = {
        'username': {
            'error_messages':
            {
                'blank': 'The username field is required',
                'required': 'The username field is required'
            }
        }
    }

I included both blank and required error messages for the browser form as well as an API request sent from Postman. The password fields can have similar error messages in their serializer field definitions.

password = serializers.CharField(
    style={'input_type': 'password'},
    write_only=True,
    error_messages={
        'blank': 'The password field is required',
        'required': 'The password field is required'
    }
)

confirm_password = serializers.CharField(
    style={'input_type': 'password'},
    write_only=True,
    error_messages={
        'blank': 'The confirm password field is required',
        'required': 'The confirm password field is required'
    }
)

The tests for the API endpoint can be updated:

def test_register_new_user():
    '''Test API to register new user'''
    client = APIClient()

    # Should result in a user created in db
    api_response = client.post(
        '/user/register-user',
        {
            'username': 'someuser@domain.com',
            'password': 'somepass',
            'confirm_password': 'somepass'
        },
        format='json'
    )
    assert api_response.status_code == 201
    assert api_response.data['username'] == 'someuser@domain.com'
    assert hasattr(api_response.data, 'password') == False
    assert api_response.data['is_active'] == False

    users_in_db = User.objects.all().count()
    assert users_in_db == 1

    # Should fail model validation
    api_response = client.post(
        '/user/register-user',
        {
            'username': 'someuser',
            'password': 'somepass'
        },
        format='json'
    )
    assert api_response.status_code == 400

    # Should fail because of missing password field
    api_response = client.post(
        '/user/register-user',
        {
            'username': 'someuser@domain.com',
        },
        format='json'
    )
    assert api_response.status_code == 400

    # Should fail because of missing username field
    api_response = client.post(
        '/user/register-user',
        {
            'password': 'somepass',
        },
        format='json'
    )
    assert api_response.status_code == 400

    # Should fail become of missing username and password
    api_response = client.post(
        '/user/register-user',
        format='json'
    )
    assert api_response.status_code == 400

    # Should fail because of missing confirm_password field
    api_response = client.post(
        '/user/register-user',
        {
            'username': 'someuser@domain.com',
            'password': 'somepass',
        },
        format='json'
    )
    assert api_response.status_code == 400

    # Should fail because the passwords do not match
    api_response = client.post(
        '/user/register-user',
        {
            'username': 'someuser1@domain.com',
            'password': 'somepass',
            'confirm_password': 'somepass1'
        },
        format='json'
    )
    assert api_response.status_code == 400

    users_in_db = User.objects.all().count()
    assert users_in_db == 1

Testing the register user API endpoint

Writing tests for the API call threw up a few basic errors that I had overlooked along with the fact that I had forgotten to add the confirm password field. The confirm password field will come next, but first the tests and the changes in the code to make the tests pass.

To begin with the most basic test for the API:

from rest_framework.test import APIClient

def test_register_new_user():
    '''Test API to register new user'''
    client = APIClient()

    # Should result in a user created in db
    api_response = client.post(
        '/user/register-user',
        {
            'username': 'someuser@domain.com',
            'password': 'somepass'
        },
        format='json'
    )
    assert api_response.status_code == 201
    assert api_response.data['username'] == 'someuser@domain.com'
    assert hasattr(api_response.data, 'password') == False
    assert api_response.data['is_active'] == False

    users_in_db = User.objects.all().count()
    assert users_in_db == 1

However, even something as this fails. Examining the api_response.data shows Bad request. The reason is that I have been using request.POST for populating the UserSerializer. Since I am using Request from DRF, I do not need to use request.POST as that still contains the encodings with a submitted form. Instead I can use request.data which contains a dictionary with the fields. This way, the API extracts the username and password whether a form was filled with the link accessed in a browser or the data submitted through Postman or in this case the test.

The tests for the API can be expanded to a few other cases:

# Should fail model validation
api_response = client.post(
    '/user/register-user',
    {
        'username': 'someuser',
        'password': 'somepass'
    },
    format='json'
)
assert api_response.status_code == 400

# Should fail because of missing password field
api_response = client.post(
    '/user/register-user',
    {
        'username': 'someuser1@domain.com',
    },
    format='json'
)
assert api_response.status_code == 400

# Should fail because of missing username field
api_response = client.post(
    '/user/register-user',
    {
        'password': 'somepass',
    },
    format='json'
)
assert api_response.status_code == 400

# Should fail become of missing username and password
api_response = client.post(
    '/user/register-user',
    format='json'
)
assert api_response.status_code == 400

# Should fail because user has been created above
api_response = client.post(
    '/user/register-user',
    {
        'username': 'someuser@domain.com',
        'password': 'somepass'
    },
    format='json'
)
assert api_response.status_code == 400

users_in_db = User.objects.all().count()
assert users_in_db == 1

In the case of certain tests such as when an API request is submitted without a password field, an error is returned that this field is required. However, the error message merely is “This field is required” as the error returned is from the UserSerializer validator for the password field. To make it clear that the field is missing, the error response returned in the view has to be changed to:

error_list = [
    '{error_field}-{error_text}'.format(
        error_field=e,
        error_text=user.errors[e][0].title()
    ) for e in user.errors
]

This formatting of the error string will make it clear with the field before the error message. Something that is interesting in the errors in the API responses is the error when the test attempts to create a duplicate user. The error is generated from the User model that is in-built with Django. However, examining the error message gives “username-A User With That Username Already Exists.” This shows that the error is generated by the serializer rather than the model. How does the error from the model get raised to the level of the Serializer that is created from the model? This is something that is enforced at the level of the database as the username is a unique field.

The next change that needs to be made is that a confirm_password field needs to be added when registering a user.

Registering a new user

With the custom User model created and the additional code to enforce the username to be a valid email, the next step is to allow a new user to register with an API call. For this, I get started with Django Rest Framework. After the usual installation and inclusion of ‘rest_framework’ in INSTALLED_APPS in settings.py file, DRF is ready to be used.

The first step is to create a UserSerializer class based on the custom User model:

class UserSerializer(serializers.ModelSerializer):
    '''Serializer for User model'''
    password = serializers.CharField(
        style={'input_type': 'password'},
        write_only=True
    )

    # def validate_username(self, value):
    #     '''Check that username is a valid email'''
    #     try:
    #         validate_email(value)
    #     except:
    #         raise serializers.ValidationError('Username must be a valid email')
    #     else:
    #         return value

    def create(self, validated_data):
        '''Create new user in db'''
        new_user = User(
            username=validated_data['username']
        )
        new_user.set_password(validated_data['password'])
        # User should be inactive until email is validated
        new_user.is_active = False
        new_user.save()
        return new_user

    class Meta:
        model = User
        fields = ['username', 'password', 'is_active']
        read_only_fields = ['is_active', ]
        # extra_args = {
        #     'password': {
        #         'write_only': True
        #     }
        # }

Though documentation on Serializers and ModelSerializers is fairly extensive, to be able to code it needs a bit of googling and experimenting. The simplest implementation is merely to define:

class UserSerializer(serializers.ModelSerializer):
    '''Serializer for User model'''
    class Meta:
        model = User
        fields = ['username', 'password', 'is_active']

Next is to use the CreateAPIView with DRF to create a new model instance – in this case to register a new user.

class RegisterUserView(CreateAPIView):
    '''Register a new user'''
    serializer_class = UserSerializer

To be able to access this, we need to register a URL for the same. Since several urls will be related to the User model, a separate urls.py file can be created in the user_auth app directory. This will have the url:

urlpatterns = [
    path('register-user', RegisterUserView.as_view(), name='register-user'),
]

The urls.py file in user_auth app directory can be included in the main urls.py file as:

from user_auth.urls import urlpatterns as user_urlpatterns

urlpatterns = [
    path('admin/', admin.site.urls),
    path('user/', include(user_urlpatterns)),
]

Now this URL can be accessed in the web-browser with the development server on the link user/register-user/ and this will give a form for the user. This form will have three fields – username, password and is_active. To begin with, the is_active field is not for the user to enter, but to be handled by the backend. The intention is to create a new user upon registration, but to set the is_active to False. When an email is set to the user’s email, the user will confirm the registration and only then will the is_active be set to True. Of course, the email integration will come later.

But first, this means that the is_active field should not appear in the default form. For that, the read_only_fields attribute need to be specified in class Meta. The next change that needs to be made is that the password is something that should only be written and should not be returned in an API response. For that, we can use the attribute extra_args and specify attributes for any of the fields. This is done with:

extra_args = {
    'password': {
        'write_only': True
    }
}

The problem however is that the password form field in the browser does not hide the password and it appears in plain text. It might be possible to add more parameters to the password field in the extra_args. But, in such a case, it might be best to redefine the password field as a serializer field:

password = serializers.CharField(
    style={'input_type': 'password'},
    write_only=True
)

Here, the style has been defined with the input_type being password. In such a case of redefinition, the parameters specified in extra_args gets overwritten and so it is best to define all parameters and attributes in this redefinition itself.

With this, the form for registering a user looks OK – the password is hidden and is_active is not shown to the user. To be able to handle the form submission, the CreateAPIView provides a post method handler:

def post(self, *args, **kwargs):
    '''Create new user from API request'''
    user = UserSerializer(data=self.request.POST)
    if user.is_valid():
        try:
            user.save()
        except Exception as e:
            return Response(
                data=e,
                status=status.HTTP_400_BAD_REQUEST
            )
    else:
        error_list = [user.errors[e][0].title() for e in user.errors]
        return Response(
            data=error_list[0],
            status=status.HTTP_400_BAD_REQUEST
        )
    return Response(user.data, status=status.HTTP_201_CREATED)

The advantage of specifying the serializer_class is that the data in the form can be directly dumped into the serializer from the POST data. One can check whether the serializer data is valid and only then create the model. But here there is the catch. The validation that has been created at the User model does not directly translate to the UserSerializer class even though it is based of the User model. I suppose one would have to redefine the fields of the UserSerializer with validation parameters. But it would defeat the purpose of redefining all fields of a User. So this means that checking for validity of UserSerializer data will not provide errors such as the username not being a valid email.

To be able to check for validity of UserSerializer data, it is also possible to define a validate_username method:

def validate_username(self, value):
    '''Check that username is a valid email'''
    try:
        validate_email(value)
    except:
        raise serializers.ValidationError('Username must be a valid email')
    else:
        return value

Defining such a method seems worse that redefining the serializer class fields as now this is repeating code. So, I have commented it out for now as the model validation should be good enough. This implies that at this moment the is_valid() check is useless as the serializer does not have any validation. I have kept it for now in case I add other validation in UserSerializer.

To be able to create a new user while also ensuring that the password is hashed and not stored directly as raw password, the create method of UserSerializer needs to be overridden:

def create(self, validated_data):
    '''Create new user in db'''
    new_user = User(
        username=validated_data['username']
    )
    new_user.set_password(validated_data['password'])
    # User should be inactive until email is validated
    new_user.is_active = False
    new_user.save()
    return new_user

Here, the checking that the username has to be valid email is performed and it will raise a ValidationError if that fails. So, the create method either returns the successfully created user or raises a ValidationError. The view then returns a 201 response with the new user or the validation error as error message with 400 response.

This works well with manual testing. Next, to clean up code and write tests.

Custom User model

With this blog post, I will get started with the code. The code for the project can be found in the repository:

 Github repo

Besides the usual Django project setup, for now the environment variables such as SECRET_KEY have been defined in a file env.py file which is importing in settings.py file. The variables that need to be defined are specified in env_example.py file. There are definitely more elegant ways to do this, but that can come later when I am about to deploy a first version.

Though I intend to only collect the email of a user as required info, I thought it best to define a custom user model rather than use the User model that is present in django.contrib.auth.models. This is done with:

class User(AbstractUser):
    '''User model for authentication and authorization'''

    def save(self, *args, **kwargs):
        '''Username validation during save to db'''
        try:
            validate_email(self.username)
        except:
            raise ValidationError('Username must be a valid email')
        else:
            super().save(*args, **kwargs)

    def clean_fields(self, exclude=None):
        '''Ensure that username is a valid email in Admin dashboard.'''
        try:
            validate_email(self.username)
        except:
            raise ValidationError('Username must be a valid email')

The custom User model being defined imports AbstractUser model from django.contrib.auth.models. This provides all the working features of the in-built User model without creating a database table. For now, all I wish to do is enforce the requirement that the username should be a valid email. For that, I overrode two methods – clean_fields and save. The clean_fields method works when a form is filled such as in the Admin dashboard. Since this will be a REST API, users will be created using User.objects.create() method, and this method does not call the clean_fields model method. So, it is necessary to override the save() model method as well. Having only the save() method and omitting the clean_fields() method is possible, but when using the Admin dashboard, this will result in an exception that breaks the server rather than a nice error message in the form field.

I wrote a test for this right away in the tests.py file:

def test_username_only_email():
    '''Test that the username can only be a valid email'''

    # Passing test with valid email
    user1 = User(username='someuser@domain.com')
    user1.set_password('somepasswordfortest')
    user1.save()
    users_in_db = User.objects.all().count()
    assert users_in_db == 1

    # Failing test with normal text instead of email
    with pytest.raises(Exception):
        user2 = User(username='someuser')
        user2.set_password('someotherpassword')
        user2.save()

    users_in_db = User.objects.all().count()
    assert users_in_db == 1

The test creates a user with the username being a valid email and verifies that there is now one user created in the database. The test then tries to create a user with a username not being an email address and verifies that an exception is raised.

Launching the online course web application

 After making a few attempts at building web apps and abandoning them after a few months, I finally found a project that I feel I can effectively use. As the creator of online courses, their hosting and management is fairly critical to the Python Power Electronics project. The courses are currently hosted on Udemy which has done quite a good job in terms of providing instructors a nice interface for hosting courses. However, to begin with, a fairly significant share of the course revenue goes to Udemy. Moreover, with Udemy choosing a new pricing strategy that changes dynamically, my courses are unaffordable to most students unless they are on sale. For this reason, a very useful web app will be one that allows me to host courses on my own website while providing most of the facilities that Udemy does.

As always, this project will be completely open source as that is largely my philosophy behind software. I have been using open-source software for close to twenty years and for the past decade almost exclusively open-source software. The hope is that the code can be used by another instructor to host his or her courses online.

The backend for the app will be Django and Django Rest Framework (DRF). In terms of the database, I have not decided yet, but will probably be MySQL or Postgres. For the frontend, I would like to experiment. I have sworn by Angular for the past several years, but have been forced to use React for a few projects. I have always preferred Angular over React, but there are a few aspects to React that are interesting and might be worth digging deeper with a side-project. I learned Flutter for mobile app development a while back, but did nothing with it. This might be the time to start using Flutter, and if a React app is ready, some of the state management can be transferred to the Flutter app.

The last bits are testing, DevOps and security. All of these are domains where I have dabbled in over the past few years with many projects. I would like to use them extensively with this project. So, the hope is to test every bit of code. For the backend, I will be using Pytest, for React it will be Jest, for Angular it will be Jasmine. In terms of DevOps, all deployment will be done using Docker containers. But besides, I would also like to automate the deployment by using Jenkins. In terms of security, I will be using a number of tools. For Static Application Security Testing (SAST), I will be using add-ons with Visual Studio Code such as HCL AppScan which highlights security issues as files are saved. For Penetration Testing, I will be using OWASP ZAP Penetration tester and also some other tools as I learn more on web application security.

Since I do see myself using this web app, I hope I will be motivated to build it consistently over the next few months. I hope it will also be a good learning experience just like the circuit simulator was.