Bad UX in Web Apps that Perform Intensive Tasks (and How to Avoid it with Queues)

Processing CSVs, resizing images, converting video...these are all intensive, time-consuming tasks that may take a computer seconds, minutes or hours to complete.

When the client requests something in a typical web app, the web server is able to handle the request in a few seconds or less. A response is then sent to the client to inform it of the outcome.

This is the familiar HTTP "request/response loop", which is summarized in this diagram:

Good UX dictates that web servers should respond fast. For that reason, an intensive task shouldn't be squeezed into the request/response loop.

We'll need a different architecture where the intensive task can be processed asynchronously, not only to prevent your web server from being overworked but also to allow the user to continue browsing rather than sitting there wondering if the website is broken or if it's just really slow.

We can add a message queue to the architecture to achieve this effectively.

In this article, we'll go through the high-level steps of implementing a message queue in a web app using Vue and Laravel.

Breaking out of the request/response loop

Say we're building an app which processes CSVs and writes the data to a database. A particularly large CSV may take several minutes to process.

Once a user uploads one to the server via the client app, we want to move the processing task into an asynchronous process. Let's look at how.

Client

Here's the relevant part of the client app where we upload the CSV. You can see we have a file input and a submit button with a message we can display.

CSVUpload.vue

<template>
  <div>
    <div v-if="message">{{ message }}</div>
    <form id="upload" enctype="multipart/form-data" @submit.prevent="submit">
      <p>Please select the file you'd like to upload.</p>
      <input type="file" name="csv" />
      <input type="submit" value="Upload" />
    </form>
  </div>
</template>

We'll use HTTP POST to submit the file. Since we're not going to process the CSV in the request/response loop, we're not expecting the final result from the response. We instead just want the server to tell us that file has been received.

submit(event) {
  axios.post("/upload", new FormData(event.target))
    .then(res => {
      this.message = res.status;
    });
}

Server

Over on the server, we'll have a controller that handles this file upload request. We'll flesh out the logic in the next section, but the important thing to note is that we attach the HTTP code 202 (Accepted) to the response. This is appropriate when you want to tell the client that the request has been received, but has not yet completed.

App/Http/Controllers/CSVUploadController.php

public function store(Request $request) 
{
  if ($request->hasFile('csv')) {
    // TODO: logic for async processing
    return response("File received for processing.", 202);
  } else {
    return response("No file provided.", 400);
  }
}

Using a message queue

Once the file is received by the web server, how do we process it outside the request/response loop? This is where we want to use a message queue.

A message queue is software that runs on a separate process to your web server (or possibly even on a separate machine) and its job is to manage asynchronous tasks. In a typical scenario, the web server will tell the message queue we have a "job" for it, the message queue will do the job (i.e. execute the code), and then it will report the results when it's done.

Message queues are handy not just because they take the load off our web server - they typically have other useful features like helping us ensure we don't lose jobs by allowing us to retry them if they fail, prioritizing important jobs etc.

Examples of message queue software include:

• Beanstalkd
• Amazon SQS (cloud-based message queue)
• Redis (not intrinsically a message queue but works great as one)

Another bonus of processing with message queues is that you can scale the message queue without having to scale your web app. If your message queue is responsible for processing intensive tasks, it will probably hit limits before the rest of your web app.

Laravel Queues

Laravel Queues make it really easy to interface a Laravel web app with a message queue.

Here's a high-level overview of how they work - I'll give a concrete example afterward.

1. Run a message queue. Tell Laravel where is and how to access it via the config/queues.php config file.
2. Run a queue worker process. This is the intermediary between the web app and the message queue that will listen for new jobs and push them to the queue. Since we need to process queue tasks asynchronously, this will run as a separate process to your web app.
3. Dispatch a "job" and the queue worker process (i.e. some code you want to execute - we'll better define jobs below)
4. Listen for an event that contains the outcome of the job (optional).

For example, we can use Redis as the message queue. Laravel includes drivers for this out-of-the-box, so it's simply a matter of running Redis on the server and telling Laravel the port/password in config/queues.php.

Laravel provides a queue worker process out of the box via the Artisan console. Open a terminal tab and run:

$ php artisan queue:work redis

Next, we'll see how to pass jobs to the message queue that can be processed asynchronously.

Job

Now we can create a job, which is the code you want run by the message queue. This will usually be an intensive or time-consuming task like CSV processing.

Laravel provides a Job class that you put your code in. Use Artisan to create one:

$ php artisan make:job ProcessCSV

The handle method gets called when this job is run, so that's where we put the task logic.

App/Jobs/ProcessCSV.php

public function handle()
{
  // Logic for processing CSV
}

We can then use the static dispatch method of this job class in our web app. This will tell the queue worker process that we want this handled by the message queue:

App/Http/Controllers/CSVUploadController.php

public function store(Request $request) 
{
  if ($request->hasFile('csv')) {
    ProcessCSV::dispatch($request->file("csv"));
    return response("File received for processing!", 202);
  } else {
    return response("No file provided.", 400);
  }
}

Using an async protocol to inform the user of the result

Our initial 202 Accepted told the client we were working on the task, but we probably need to tell them outcome when the task is complete.

Since the task may take a long time to complete, it'd be better UX to use an async protocol like email or SMS to inform of the outcome, so the user can keep using their browser to scroll Facebook or Reddit for a few minutes and don't have to sit there waiting.

You could also open a web socket connection between the client and server, and send the response that way. I still think email or SMS is better as it doesn't require the user to keep the tab open and remember to check.

Client

Let's modify the form on the client so the user can specify their email address:

<form id="upload" enctype="multipart/form-data" @submit.prevent="submit">
  <p>Please select the file you'd like to upload. Provide an email address and we'll inform you of the result and spam you later.</p>
  <input type="file" name="csv" />
  <input type="email" name="email" />
  <input type="submit" value="Upload" />
</form>

Server

Now, when we handle the initial request, we can pass the email address to the job:

public function store(Request $request) 
{
  if ($request->hasFile('csv')) {
    ProcessCSV::dispatch($request->file("csv"), $request->email);
    return response("File received for processing!", 202);
  } else {
    return response("No file provided.", 400);
  }
}

Laravel's queue worker process will send an event when a job is complete, telling you what happened, if it failed, etc.

We can listen to that event and use it to dispatch a notification. And why not create another job for sending the email!

App/Providers/AppServiceProvider.php

Queue::after(function (JobProcessed $event) {
  $result = ... // get the job result from the DB
  SendEmail::dispatch($event->data["email"], $result);
});

Wrap up

If your web app needs to complete an intensive or time-consuming task for a user, don't try and squeeze it into the request/response loop. Dispatch to a message queue so that not only can you give a fast response to the user, but you also prevent your web server from being overburdened.

Laravel Queues are fantastic for bringing the power of message queues to a web app. There are many more features I didn't cover here, including Laravel's free Horizon dashboard for managing your queue.