Setting up multi-languages in Django

Following the last post where I had briefly covered the idea behind multiple languages through Django, I finally setup the app to support for now just two languages - the first being English and the second German as that is the only foreign language I speak to some extent. Other than default settings with Django, it also needs django-modeltranslation which can be found from:

 https://github.com/deschler/django-modeltranslation

In the settings.py file, the model translation app will appear before all other apps, as most other apps will be modified because of multi language support.

INSTALLED_APPS = [
    'modeltranslation',
    #
]

It needs the LocaleMiddleware to be able to handle multi language content. This needs to come after SessionMiddleware as sessions will store data and this will need to be translated as well, but should come before CommonMiddleware. So far I am not using CacheMiddleware, but I will need to enable cache later. In case CacheMiddleware is being used, LocaleMiddleware must come after it.

Finally, comes the language settings. Creating a Django project usually produces LANGUAGE_CODE, USE_I18N and USE_TZ variables. The LANGUAGE_CODE is usually en_US, but I changed it to en to keep things simple.

LANGUAGE_CODE = 'en'

LANGUAGES = [
    ('en', _('English')),
    ('de', _('German')),
]

TIME_ZONE = 'UTC'

USE_I18N = True

USE_TZ = True

# Where translations will be stored
LOCALE_PATHS = [
    BASE_DIR / 'locale',
]

After creating the locale directory in the root of the project, we can now start generating translation files. The very first time, it is necessary to specify the exact locales that you want to create translations for:

django-admin makemessages --locale=de

This will create the .po files where translations need to be inserted. To start inserting translations, all the text that needs to be translated needs to wrapped with:

from django.utils.translation import gettext_lazy as _
#
_('Course not found from URL')

The normal text 'Course not found from URL' will be wrapped with gettext_lazy from the translation module.

The makemessages command will create a folder for each language in the locale directory and create a django.po file. All that is need to be done is go through the file and insert translations:

msgid "Course not found from URL"
msgstr "Translation of Course not found from URL"

To get translations enabled at the database level, a new file translation.py file needs to be created in each app that contains a models.py file. As an example, for the courses app:

from modeltranslation.translator import translator, TranslationOptions

from .models import Course


class CourseTranslation(TranslationOptions):
    '''
    Translation of Course model
    '''
    fields = ('title', 'subtitle', 'description')


translator.register(Course, CourseTranslation)

We need to specify the fields for which translation needs to be enabled, and register these fields along with the model. This will now create additional columns for those fields for each language that is supported  - so title becomes title, title_en and title_de. The default title will be the same as the default language title_en.

To ensure that multiple languages can also be edited in Django admin, the admin.py file also needs modification:

from django.contrib import admin
from modeltranslation.admin import TranslationAdmin

from .models import Course


class CourseAdmin(TranslationAdmin):
    '''
    Translated model of Course model for admin
    '''
    pass


admin.site.register(Course, CourseAdmin)

I had to delete the database and run the migrations again to ensure that there were no violations. But once that was done, the rest was pretty smooth - the admin panel showed multiple language fields.

What I was surprised about was that not much change needed to made for the API and the hook between the API and the database. To enable multi-language support in the API, the API calls will need to have an Accept-Language header. If this header is absent, the default English is used. To now hook this up with the rest, I needed to create a base view which then is inherited by all other views:

from django.utils import translation
from rest_framework.generics import GenericAPIView
class BaseAPIView(GenericAPIView):
    '''
    Base API view for handling language content
    '''

    def initial(self, request, *args, **kwargs):
        super().initial(request, *args, **kwargs)
        lang = request.META.get('HTTP_ACCEPT_LANGUAGE')
        if lang is not None:
            translation.activate(lang)

The initial method is defined in the APIView of Django Rest Framework and

"Performs any actions that need to occur before the handler method gets called. This method is used to enforce permissions and throttling, and perform content negotiation."

The documentation also adds that one would typically not need to override this method. However, enabling language support does seem like a legitimate override of this method. Once the language is enabled with translation.activate(), the bridge is complete. Not only is it possible to retrieve data in different languages, but also to create and update data in different languages using POST and PUT methods.

So far this seems like a very convenient way to enable multi-language support in the app. It would be interesting to see if any issues crop up when many languages are enabled as then the columns of the tables will increase in number.

Multi language support for the app

I took a break from back-end dev to do some front-end dev using React. One part of this app will be interactive videos, and for that purpose, I was planning to build an interactive video player. This would be a custom video player that can play multiple videos according to a configuration file, and can accept feedback from the user/student.

This app has now reached a decent level of maturity though it needs more work:

https://github.com/shivkiyer/accordion-player

Now that the interactive video player can be augmented at a gradual pace, it is now time to return to the Django app. So far, the only language supported has been English. For an app to be practically useful, it must support multiple languages, and this would be even more so in the case of an educational app. As an example, most of my students of electrical engineering would have a language other than English as their first language. From the demographics, the biggest language groups would be Spanish, Arabic, Portugese and maybe French and Russian to some extent. This makes the app design interesting for several reasons.

The first is that the webpage has to be in other languages. Though one could argue that most modern browsers come with extensions that perform automatic translation, these extensions do have some limitations. For example, dynamic content that appears in drop down boxes or modals may not be translated. In such cases, where an interaction occurs with the user, it is best that content be generated in the user's preferred language.

The second is with respect to the actual educational content - the video and audio. Here, things get interesting, as many MOOC websites offer an automatic generation of subtitles of uploaded videos, and subsequently translate these subtitles in other languages. One major flaw is that the subtitles generated can be fairly incorrect especially if the speaker is not a native language speaker or has an accent - such as my case. I have found that automatic subtitles generated from my videos are usually full of errors.

Though subtitles are important especially to make the content accessible to those with hearing impairments, learning from reading the subtitles alone would make the videos a bit boring. As of now, there are many AI based translation tools that also allow for translation of audio into other languages. This implies that video content can be made separately available in different languages. Of course the translation may be glitchy, but it will get better with time.

So, for this the first would be to enable a language preference on the front-end. This is quite simple, it can be done on the home page or with a user's account. Once done, all requests made by the front-end to the back-end will contain the "Accept-Language" header, and the back-end will provide content in the language specified in the header. The default language will be English, so if the translated content does not exist, the default English content will be returned.

With Django, this actually turned out much easier than expected. I used the gettext package available with Linux. Apparently, the python-gettext package through pip did not work. Once this was done, the gettext  or gettext_lazy can be used to indicate translations to be expected. More on this in a separate post.

Another feature will be to provide translations in the database, and for this I plan to use django-modeltranslation. Both configurations and set up to be described in another post.

Lecture detail view with video list

To get started with videos, I need a new model VideoContent that has the name of the video, the course to which the video belongs and the video file. A video is associated with a course and then can be added to a lecture. This association between course and video is loose, it is merely to have some association while uploading them.

def video_file_path(instance, filename):
    '''
    Generate path that has course slug.
    Called when a video file is uploaded to Video model instance.

    Attributes
    ----------------
    instance : Video model instance
    filename : str

    Raises
    ----------------
    400 error
        If uploading for a course that does not exist

    Returns
    ----------------
    String with file path containing course slug
    '''

    course = instance.course
    if course is None:
        raise CustomAPIError(
            status_code=status.HTTP_400_BAD_REQUEST,
            detail='Associated course not found'
        )
    dir_name = ''.join(filter(str.isalnum, course.slug))
    return f'{dir_name}/{filename}'


class VideoContent(models.Model):
    '''
    Video model for lectures

    Attributes
    ---------------
    course : Reference to a course model instance
    video_file : File
    created_at: Datetime
        Autogenerated when model is created
    updated_at: Datetime
        Autoupdated when model is updated
    '''
    course = models.ForeignKey(
        'courses.Course',
        related_name='videos',
        null=True,
        on_delete=models.SET_NULL
    )
    name = models.CharField(
        max_length=300,
        unique=True,
        default='Video name'
    )
    video_file = models.FileField(upload_to=video_file_path, max_length=300)
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)

    @property
    def video_file_path(self):
        return f'{settings.BASE_URL}{self.video_file.url}'

    objects = VideoContentManager()

    def __str__(self):
        return self.video_file.url

The VideoContent model is quite simple - there is a name field which is unique, but that is just to be able to search for videos while associating them with lectures. There is mainly the video_file field which contains the uploaded file contents. I defined a property video_file_path that returns the URL of the uploaded file - this will be used when returning the video details so that the frontend can fetch the video and play it. To upload the video file, I created an uploader method that uploads the file into a directory according to the course slug.

To upload this video through an API, Django Rest Framework recommends using the FormParser and the MultiPartParser. Also, the file name needs to be a part of the API url. Therefore, the definition of the URL for uploading videos will be:

/api/courses/course-slug/lectures/id/add-video/filename

To be able to use Postman to upload files, the body of the request needs to be form-data with name as a parameter and the second parameter will be of type File. How this will be done with React will need to be checked out later.

The view class for video upload will be:

class VideoContentView(APIView, UserAuthentication):
    '''
    View for uploading videos

    Attributes
    -----------------
    parser_classes : list
        FormParser and MultiPartParser used for file uploads

    Methods
    -----------------
    post(request, filename, *args, **kwargs):
        Creates VideoContent model instance and handles file uploads
    '''

    parser_classes = [FormParser, MultiPartParser, ]
    user_model = User

    def post(self, request, filename, *args, **kwargs):
        '''
        Creates VideoContent model instance and handles file uploads

        Parameters
        ---------------
        request : Request object
        filename : str
            Name of the video file

        Raises
        ---------------
        400 error:
            If course slug is not present in url
            If video name is not provided in request body
            If video name is not unique
        403 error:
            If no credentials are provided in header
            If non-admin credentials are provided in header
            If non-instructor credentials are provided in header
        404 error:
            If lecture cannot be found

        Returns
        ---------------
        201 with VideoContentSerializer data
        '''
        user = self.authenticate(request)
        file_obj = request.data['File']

        course_slug = self.kwargs.get('slug', None)
        course_obj = Course.objects.get_course_by_slug(
            course_slug,
            admin_only=True
        )
        if not course_obj.check_user_is_instructor(user):
            raise CustomAPIError(
                status_code=status.HTTP_403_FORBIDDEN,
                detail='Only an instructor can add videos'
            )

        video_name = request.data.get('name')
        if video_name is None:
            raise CustomAPIError(
                status_code=status.HTTP_400_BAD_REQUEST,
                detail='Video name is required'
            )

        VideoContent.objects.is_video_name_unique(video_name)

        video_obj = VideoContent.objects.create(
            name=video_name,
            course=course_obj,
            video_file=file_obj
        )

        lecture_id = self.kwargs.get('id')
        Lecture.objects.add_video_to_lecture(lecture_id, video_obj)

        serializer = VideoContentSerializer(video_obj)

        return Response(
            data=serializer.data,
            status=status.HTTP_201_CREATED
        )

After the video is uploaded, the video is added to the lecture with the manager method add_video_to_lecture which will connect the video model instance to the videos field in the lecture instance with the ID specified in the API URL. The API returns the serializer which is:

class VideoContentSerializer(serializers.ModelSerializer):
    '''
    Serializer for model VideoContent
    '''
    video_file_path = serializers.ReadOnlyField()

    class Meta:
        model = VideoContent
        fields = ['name', 'video_file_path']

Now that the video has been uploaded, the videos have to show up in the lecture detail view. The lecture list view will not show the video list as the lecture list is an open end-point while the detail view is only for registered users or admin.

Therefore, defining a separate LectureDetailSerializer:

class LectureDetailSerializer(serializers.ModelSerializer):
    '''
    Serializer for detail view of Lecture including related videos
    '''

    videos = VideoContentSerializer(many=True, read_only=True)

    class Meta:
        model = Lecture
        fields = ['id', 'title', 'description', 'seq_no', 'videos']

The LectureDetailSerializer has an embedded serializer with video details. This now needs to be connected to the lecture view class. the only change that needs to be made in the LectureView class is to add a get_serializer_class that returns which serializer class to use according to the request type:

def get_serializer_class(self):
    '''
    Return the serializer_class according to the view.

    Returns
    ------------------
    LectureDetailSerializer for detail views and
    LectureSerializer for all other views
    '''
    if self.request.method == 'GET' and self.kwargs.get('id') is not None:
        return LectureDetailSerializer
    return LectureSerializer

Only when the request type is a GET and there is a lecture ID present in the URL parameters, should the detail serializer be used - in all other cases, the basic serializer should be used.

With this, now there is separate behavior for list and detail views as intended. And, a basic structure of the backend now exists - user registration, user login, course creation, course update, course fetch, lecture creation, lecture update, lecture fetch, video upload. Now to start coding the frontend and add features to the backend as needed.

Handling videos

In this app, I would like to handle videos a little differently than regular online education websites. In any other MOOC website, a lecture may have video content or may just be some text instructions, and the lecture may have some downloadable resources/links. In my online app, I would like to introduce a new concept - interactive videos.

Interactive videos are typically used in marketing when a company produces promotional videos for their products, and if they offer a range of products and services, the videos might also be quite a few. In such cases, they would make their videos interactive, where the user decides which products or services they are interested in, and only watch the promo videos relevant to that. This user feedback can be from various means - a simple table with choices asking the user to select the products they are interested in, or a little more graphic - for example, a sports company showing the photo of a famous sports person and clicking on different clothing or equipment can take the user to the relevant products.

The reason for trying to bring this into online education is that most students do not have time to study regularly. Some might find time only on weekends, while for others it may be just a couple of times a month. In such a case, coming back to a course after a week, and especially a course that is heavily technical, would mean they might have to go back and review some of the older lectures and re-watch previous videos. Here comes the need for interactivity - if every lecture is completely self-sufficient, and this implies if a lecture uses a certain number of concepts, and there is specific review material for all these prerequisites, then the student can watch every lecture without any need to go back and forth - just select the prerequisites that need to be refreshed before watching the lecture. As for the students who are regularly watching and do not need review, they can skip the refreshers and go on to watch the main lecture video.

To enable the above, the video content for lectures needs to be reviewed. Instead of having just a single video for a lecture, there can be many videos for a lecture - the main video and also the prerequisites. Moreover, videos themselves can be reused, especially the prerequisites, and this implies that a video can be present in many lectures. This points to a ManyToMany relationship between videos and lectures. Furthermore, to make the handling of file content efficient, it would be best to reduce the duplication of video files and potentially even make them unique. If you want to reuse a video from another lecture or even course, no need to upload it again - merely refer to it.

Therefore, to structure the VideoContent model, I have been making a number of commits. The first was to associate a VideoContent with a Lecture as a foreign key which implies a lecture can have many videos. Next, I thought it might be better to add reference to Course as foreign key which means a course can have many videos, and add a reference to Lecture as a ManyToMany field, which implies that a lecture can have many videos, and a video can belong to many videos. However, going further, it looks like the ManyToMany relationship between lectures and videos is better handled in the Lecture model rather than the VideoContent model. While uploading a video, all that is needed is a primary identifier - which course the video belongs to, as this will also help to store the video in a folder that resembles the course slug. The video of course will be uploaded when a lecture is created, and therefore, it will be added to the videos list. There can be only a single video, if the instructor does not want any interactivity, or there could be multiple videos. How these multiple videos need to be played in the lecture will be decided by a configuration table - something that will come soon.

After documentation

I spent the last couple of weeks learning how to document Python code as well as Angular and React projects. I figured this is best done when the project is in a nascent stage, so as to establish this as the baseline going forward.

 The best documentation experience was with Angular. I used Jsdoc to document the Typescript code, but for Angular, there is a package compodoc that produces an elegant web document. This is the typical documentation for a Angular component:

/**
 * Generates a container for the video player
 *
 * @param {number} width The width of the container (optional)
 * @param {number} height The height of the container (optional)
 * @returns A container for the video and controls
 *
 * @example
 * Without any inputs, container adjusts to browser window
 * 
 *
 * @example
 * With only width, container has fixed width and
 * aspect ratio of 16:9
 * 
 *
 * @example
 * With only height, container has fixed width and
 * aspect ratio of 16:9
 * 
 *
 */

After documenting all components and utility functions, I created a very basic config file for the documentation and saved it in tsconfig.doc.json:

{
  "include": ["src/**/*.ts"],
  "exclude": ["src/test.ts", "src/**/*.spec.ts", "src/app/file-to-exclude.ts"]
}

Running compodoc produces an entire directory of html files that depicts the entire app graphically.

 

Very impressed to see documentation arranged in such a visually appealing manner.

In comparison, documentation for the React app was with Jsdoc and another package better-docs which has a special addition for components. Though running jsdoc produces another directory with html files, the links are broken and navigation seems a bit jumpy.

For Python, I used almost a markdown approach. Classes are documented as this example:

    '''
    Base view for a course based on course URL

    Attributes
    -------------
    serializer_class : class
        CourseSerializer class
    user_model : class
        User class
    lookup_field : str
        The field in URL used to look up model instance

    Methods
    -------------
    get_queryset() : Base method for course list view
    get_object() : Returns course model instance
    '''

The VSCode IDE displays the documentation once you hover over a class. A function has documentation similar to JS:

'''
Create a new course - POST request

Parameters
--------------
request - dict

Raises
--------------
400 error
    Course title missing or not unique
    Course price missing for non-free course
403 error
    If course created by non-admin user

Returns
--------------
201 response with course data
'''

Next step is to find a package that will convert these doc strings into html viewable pages that can be committed into Git.

Continuing with video contents and getting started with the frontend

With some basic course and lecture end-points created and tested, the next step is the actual video contents of the course along with other downloadable resources like .pdf, .docx, .zip and other files. Since the backend is also now around 1/4th done, it is time to slowly get started with the frontend as well. Rather than start with the frontend for the app directly, I am starting with the video player.

Instead of using the regular video player, I have decided to build a custom video player for a few reasons. First, and quite obviously, I would like to have more control over the videos rather than what comes with a default video player. Second, a year or so back, I was working on another interesting project with a friend that involved building an interactive video player. This was a video player that would take user inputs and play certain parts of the video accordingly. The application for this is primarily in advertising and marketing as the user can choose to watch only what they are interested in watching. With respect to online courses, I would like to use this interactivity in asking students if they would like to review certain concepts necessary for the particular lecture they are about to watch. This is due to the fact that not all students can invest time regularly, and if they are watching videos after several days, they might need a refresher.

This project was called the Accordion player, a name that my friend came up with. I have started hosting this project at this Github link:

https://github.com/shivkiyer/accordion-player

I would definitely like to build the app in React, and would like to also use Angular as it is something I have used in the past. So, there are currently two sub-projects in this repo - one with React and the other with Angular.

Since the app is in the nascent stage, another thing which I would like to go back and work on is documentation. This time, with the Django backend, I have done my best to make the code as modular as possible. This implies classes and functions that are as small as possible. This has the advantage that the single doc string at the beginning of the function or class is usually good enough and there doesn't have to be comments all over the code which can become a little bothersome.

However, with both Python and JavaScript, it might be worthwhile to write proper doc strings which can then be used to create nice understandable documentation of the code. So there might be a few commits only related to documentation over the next week.

Additions to the lecture API endpoints

With the basic CRUD for lecture endpoints done, I will need to plan some other details as well as the lecture contents and resources (or attachments). For now, a lecture must have a title which must be unique in a course, and an optional description. Another field that a lecture must have is whether a preview is enabled, and in that case anyone can view it for free. Also, if the lecture has video content, the duration of the video needs to be displayed even in the list view.

Next comes the sequence of lectures in a course. There has to be a sequence ID to indicate how the lectures are to appear, and an instructor can move lectures around and change the sequence. This sequence ID is also unique within a course, and must not be set directly by the instructor, but rather must be generated during the creation process. In the case of a single lecture creation, it might be fairly easy, as one only needs to check the largest sequence number in the course and add the new lecture. In the case of bulk lecture creation, it might need some kind of counter as the contents of a spreadsheet might be read. Finally, in the lecture list view, the lectures should appear in the ascending order of the sequence ID.

There has been no difference between the Lecture List view and the Detail view. However, these two should be different - the list view should only display the title, description and time duration of a lecture. The detail view should provide everything - details of the video content, resources for download etc.

Some changes such as the detail view can wait until the other models such as videos and downloads are ready. But the sequence ID can be handled right away. Along with creating the sequence ID and retrieving lectures according to sequence ID, an API call for changing the sequence ID of a lecture must also be created. So, if lecture 15 in a course needs to be inserted before lecture 9, the sequence ID of lecture 15 will become 9, and the sequence IDs of lectures 9 to 14 will increase by 1. And the reverse process if a lecture needs to be moved down the list.