Chapter 6. Getting to the Minimum Viable Site

In this chapter we’re going to address the problems we discovered at the end of the last chapter. In the immediate, the problem of cleaning up after functional test runs. Later, the more general problem, which is that our design only allows for one global list. I’ll demonstrate a critical TDD technique: how to adapt existing code using an incremental, step-by-step process which takes you from working code to working code. Testing Goat, not Refactoring Cat.

Ensuring Test Isolation in Functional Tests

We ended the last chapter with a classic testing problem: how to ensure isolation between tests. Each run of our functional tests was leaving list items lying around in the database, and that would interfere with the test results when you next ran the tests.

When we run unit tests, the Django test runner automatically creates a brand new test database (separate from the real one), which it can safely reset before each individual test is run, and then throw away at the end. But our functional tests currently run against the "real" database, db.sqlite3.

One way to tackle this would be to "roll our own" solution, and add some code to functional_tests.py which would do the cleaning up. The setUp and tearDown methods are perfect for this sort of thing.

Since Django 1.4 though, there’s a new class called LiveServerTestCase which can do this work for you. It will automatically create a test database (just like in a unit test run), and start up a development server for the functional tests to run against. Although as a tool it has some limitations which we’ll need to work around later, it’s dead useful at this stage, so let’s check it out.

LiveServerTestCase expects to be run by the Django test runner using manage.py. As of Django 1.6, the test runner will find any files whose name begins with test. To keep things neat and tidy, let’s make a folder for our functional tests, so that it looks a bit like an app. All Django needs is for it to be a valid Python package directory (ie, one with a __init__.py in it):

$ mkdir functional_tests
$ touch functional_tests/__init__.py

Then we move our functional tests, from being a standalone file called functional_tests.py, to being the tests.py of the functional_tests app. We use git mv so that Git notices that we’ve moved the file:

$ git mv functional_tests.py functional_tests/tests.py
$ git status # shows the rename to functional_tests/tests.py and __init__.py

At this point your directory tree should look like this:

.
├── db.sqlite3
├── functional_tests
│   ├── __init__.py
│   └── tests.py
├── lists
│   ├── admin.py
│   ├── __init__.py
│   ├── migrations
│   │   ├── 0001_initial.py
│   │   ├── 0002_item_text.py
│   │   ├── __init__.py
│   │   └── __pycache__
│   ├── models.py
│   ├── __pycache__
│   ├── templates
│   │   └── home.html
│   ├── tests.py
│   └── views.py
├── manage.py
└── superlists
    ├── __init__.py
    ├── __pycache__
    ├── settings.py
    ├── urls.py
    └── wsgi.py

functional_tests.py is gone, and has turned into functional_tests/tests.py. Now, whenever we want to run our functional tests, instead of running python3 functional_tests.py, we will use python3 manage.py test functional_tests.

You could mix your functional tests into the tests for the lists app. I tend to prefer to keep them separate, because functional tests usually have cross-cutting concerns that run across different apps. FTs are meant to see things from the point of view of your users, and your users don’t care about how you’ve split work between different apps!

Now let’s edit functional_tests/tests.py and change our NewVisitorTest class to make it use LiveServerTestCase:

functional_tests/tests.py (ch06l001). 

from django.test import LiveServerTestCase
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

class NewVisitorTest(LiveServerTestCase):

    def setUp(self):
        [...]

Next,[9] instead of hardcoding the visit to localhost port 8000, LiveServerTestCase gives us an attribute called live_server_url:

functional_tests/tests.py (ch06l002). 

    def test_can_start_a_list_and_retrieve_it_later(self):
        # Edith has heard about a cool new online to-do app. She goes
        # to check out its homepage
        self.browser.get(self.live_server_url)

We can also remove the if __name__ == '__main__' from the end if we want, since we’ll be using the Django test runner to launch the FT.

Now we are able to run our functional tests using the Django test runner, by telling it to run just the tests for our new functional_tests app:

$ python3 manage.py test functional_tests
Creating test database for alias 'default'...
F
======================================================================
FAIL: test_can_start_a_list_and_retrieve_it_later
(functional_tests.tests.NewVisitorTest)
 ---------------------------------------------------------------------
Traceback (most recent call last):
  File "/workspace/superlists/functional_tests/tests.py", line 61, in
test_can_start_a_list_and_retrieve_it_later
    self.fail('Finish the test!')
AssertionError: Finish the test!

 ---------------------------------------------------------------------
Ran 1 test in 6.378s

FAILED (failures=1)
Destroying test database for alias 'default'...

The FT gets through to the self.fail, just like it did before the refactor. You’ll also notice that if you run the tests a second time, there aren’t any old list items lying around from the previous test—it has cleaned up after itself. Success! We should commit it as an atomic change:

$ git status # functional_tests.py renamed + modified, new __init__.py
$ git add functional_tests
$ git diff --staged -M
$ git commit  # msg eg "make functional_tests an app, use LiveServerTestCase"

The -M flag on the git diff is a useful one. It means "detect moves", so it will notice that functional_tests.py and functional_tests/tests.py are the same file, and show you a more sensible diff (try it without the flag!).

Running Just the Unit Tests

Now if we run manage.py test, Django will run both the functional and the unit tests:

$ python3 manage.py test
Creating test database for alias 'default'...
.......F
======================================================================
FAIL: test_can_start_a_list_and_retrieve_it_later
[...]
AssertionError: Finish the test!

 ---------------------------------------------------------------------
Ran 8 tests in 3.132s

FAILED (failures=1)
Destroying test database for alias 'default'...

In order to run just the unit tests, we can specify that we want to only run the tests for the lists app:

$ python3 manage.py test lists
Creating test database for alias 'default'...
.......
 ---------------------------------------------------------------------
Ran 7 tests in 0.009s

OK
Destroying test database for alias 'default'...

Now let’s move on to thinking about how we want support for multiple lists to work. Currently the FT (which is the closest we have to a design document) says this:

functional_tests/tests.py. 

    # Edith wonders whether the site will remember her list. Then she sees
    # that the site has generate a unique URL for her -- there is some
    # explanatory text to that effect.
    self.fail('Finish the test!')

    # She visits that URL - her to-do list is still there.

    # Satisfied, she goes back to sleep

But really we want to expand on this, by saying that different users don’t see each other’s lists, and each get their own URL as a way of going back to their saved lists. Let’s think about this a bit more.

Small Design When Necessary

TDD is closely associated with the agile movement in software development, which includes a reaction against Big Design Up Front the traditional software engineering practice whereby, after a lengthy requirements gathering exercise, there is an equally lengthy design stage where the software is planned out on paper. The agile philosophy is that you learn more from solving problems in practice than in theory, especially when you confront your application with real users as soon as possible. Instead of a long up-front design phase, we try and put a minimum viable application out there early, and let the design evolve gradually based on feedback from real-world usage.

But that doesn’t mean that thinking about design is outright banned! In the last chapter we saw how just blundering ahead without thinking can eventually get us to the right answer, but often a little thinking about design can help us get there faster. So, let’s think about our minimum viable lists app, and what kind of design we’ll need to deliver it.

  • We want each user to be able to store their own list—at least one, for now.
  • A list is made up of several items, whose primary attribute is a bit of descriptive text.
  • We need to save lists from one visit to the next. For now, we can give each user a unique URL for their list. Later on we may want some way of automatically recognising users and showing them their lists.

To deliver the "for now" items, it sounds like we’re going to store lists and their items in a database. Each list will have a unique URL, and each list item will be a bit of descriptive text, associated with a particular list.

YAGNI!

Once you start thinking about design, it can be hard to stop. All sorts of other thoughts are occurring to us—we might want to give each list a name or title, we might want to recognise users using usernames and passwords, we might want to add a longer notes field as well as short descriptions to our list, we might want to store some kind of ordering, and so on. But we obey another tenet of the agile gospel: "YAGNI" (pronounced yag-knee), which stands for "You aint gonna need it!" As software developers, we have fun creating things, and sometimes it’s hard to resist the urge to build things just because an idea occurred to us and we might need it. The trouble is that more often than not, no matter how cool the idea was, you won’t end up using it. Instead you have a load of unused code, adding to the complexity of your application. YAGNI is the mantra we use to resist our overenthusiastic creative urges.

REST

We have an idea of the data structure we want—the Model part of Model-View-Controller (MVC). What about the View and Controller parts? How should the user interact with Lists and their Items using a web browser?

Representational State Transfer (REST) is an approach to web design that’s usually used to guide the design of web-based APIs. When designing a user-facing site, it’s not possible to stick strictly to the REST rules, but they still provide some useful inspiration.

REST suggests that we have a URL structure that matches our data structure, in this case lists and list items. Each list can have its own URL:

    /lists/<list identifier>/

That will fulfill the requirement we’ve specified in our FT. To view a list, we use a GET request (a normal browser visit to the page).

To create a brand new list, we’ll have a special URL that accepts POST requests:

    /lists/new

To add a new item to an existing list, we’ll have a separate URL, to which we can send POST requests:

    /lists/<list identifier>/add_item

(Again, we’re not trying to perfectly follow the rules of REST, which would use a PUT request here—we’re just using REST for inspiration.)

In summary, our scratchpad for this chapter looks something like this:

Implementing the New Design Using TDD

How do we use TDD to implement the new design? Let’s take another look at the flowchart for the TDD process in Figure 6-1.

At the top level, we’re going to use a combination of adding new functionality (by extending the FT and writing new application code), and refactoring our application—ie, rewriting some of the existing implementation so that it delivers the same functionality to the user but using aspects of our new design. At the unit test level, we’ll be adding new tests or modifying existing ones to test for the changes we want, and we’ll be able to use the untouched unit tests to make sure we don’t break anything in the process.

A flowchart showing functional tests as the overall cycle
Figure 6-1. The TDD process with functional and unit tests

Let’s translate our scratchpad into our functional test. As soon as Edith submits a first list item, we’ll want to create a new list, adding one item to it, and take her to the URL for her list. Look for the point at which we say inputbox.send_keys('Buy peacock feathers'), and amend the next block of code like this:

functional_tests/tests.py. 

    inputbox.send_keys('Buy peacock feathers')

    # When she hits enter, she is taken to a new URL,
    # and now the page lists "1: Buy peacock feathers" as an item in a
    # to-do list table
    inputbox.send_keys(Keys.ENTER)
    edith_list_url = self.browser.current_url
    self.assertRegex(edith_list_url, '/lists/.+') #1
    self.check_for_row_in_list_table('1: Buy peacock feathers')

    # There is still a text box inviting her to add another item. She
    [...]

1

assertRegex is a helper function from unittest that checks whether a string matches a regular expression. We use it to check that our new REST-ish design has been implemented. Find out more in the unittest documentation.

Let’s also change the end of the test and imagine a new user coming along. We want to check that they don’t see any of Edith’s items when they visit the home page, and that they get their own unique URL for their list.

Delete everything from the comments just before the self.fail (they say "Edith wonders whether the site will remember her list …") and replace them with a new ending to our FT:

functional_tests/tests.py. 

    [...]
    # The page updates again, and now shows both items on her list
    self.check_for_row_in_list_table('2: Use peacock feathers to make a fly')
    self.check_for_row_in_list_table('1: Buy peacock feathers')

    # Now a new user, Francis, comes along to the site.

    ## We use a new browser session to make sure that no information
    ## of Edith's is coming through from cookies etc #1
    self.browser.quit()
    self.browser = webdriver.Firefox()

    # Francis visits the home page.  There is no sign of Edith's
    # list
    self.browser.get(self.live_server_url)
    page_text = self.browser.find_element_by_tag_name('body').text
    self.assertNotIn('Buy peacock feathers', page_text)
    self.assertNotIn('make a fly', page_text)

    # Francis starts a new list by entering a new item. He
    # is less interesting than Edith...
    inputbox = self.browser.find_element_by_id('id_new_item')
    inputbox.send_keys('Buy milk')
    inputbox.send_keys(Keys.ENTER)

    # Francis gets his own unique URL
    francis_list_url = self.browser.current_url
    self.assertRegex(francis_list_url, '/lists/.+')
    self.assertNotEqual(francis_list_url, edith_list_url)

    # Again, there is no trace of Edith's list
    page_text = self.browser.find_element_by_tag_name('body').text
    self.assertNotIn('Buy peacock feathers', page_text)
    self.assertIn('Buy milk', page_text)

    # Satisfied, they both go back to sleep

1

I’m using the convention of double-hashes (##) to indicate "meta-comments"—comments about how the test is working and why—so that we can distinguish them from regular comments in FTs which explain the User Story. They’re a message to our future selves, which might otherwise be wondering why the heck we’re quitting the browser and starting a new one…

Other than that, the changes are fairly self-explanatory. Let’s see how they do when we run our FTs:

[...]
    self.assertRegex(edith_list_url, '/lists/.+')
AssertionError: Regex didn't match: '/lists/.+' not found in
'http://localhost:8081/'

As expected. Let’s do a commit, and then go and build some new models and views:

$ git commit -a

I found the FTs hung when I tried to run them today. It turns out I needed to upgrade Selenium, with a pip3 install --upgrade selenium. You may remember from the preface that it’s important to have the latest version of Selenium installed—it’s only been a couple of months since I last upgraded, and Selenium had gone up by six point versions. If something weird is happening, always try upgrading Selenium!

Iterating Towards the New Design

Being all excited about our new design, I had an overwhelming urge to dive in at this point and start changing models.py, which would have broken half the unit tests, and then pile in and change almost every single line of code, all in one go. That’s a natural urge, and TDD, as a discipline, is a constant fight against it. Obey the Testing Goat, not Refactoring Cat! We don’t need to implement our new, shiny design in a single big bang. Let’s make small changes that take us from a working state to a working state, with our design guiding us gently at each stage.

There are four items on our to-do list. The FT, with its Regexp didn't match, is telling us that the second item—giving lists their own URL and identifier—is the one we should work on next. Let’s have a go at fixing that, and only that.

The URL comes from the redirect after POST. In lists/tests.py, find test_home_page_redirects_after_POST, and change the expected redirect location:

lists/tests.py. 

self.assertEqual(response.status_code, 302)
self.assertEqual(response['location'], '/lists/the-only-list-in-the-world/')

Does that seem slightly strange? Clearly, /lists/the-only-list-in-the-world isn’t a URL that’s going to feature in the final design of our application. But we’re committed to changing one thing at a time. While our application only supports one list, this is the only URL that makes sense. We’re still moving forwards, in that we’ll have a different URL for our list and our home page, which is a step along the way to a more REST-ful design. Later, when we have multiple lists, it will be easy to change.

Another way of thinking about it is as a problem-solving technique: our new URL design is currently not implemented, so it works for 0 items. Ultimately, we want to solve for n items, but solving for 1 item is a good step along the way.

Running the unit tests gives us an expected fail:

$ python3 manage.py test lists
[...]
AssertionError: '/' != '/lists/the-only-list-in-the-world/'

We can go adjust our home_page view in lists/views.py:

lists/views.py. 

def home_page(request):
    if request.method == 'POST':
        Item.objects.create(text=request.POST['item_text'])
        return redirect('/lists/the-only-list-in-the-world/')

    items = Item.objects.all()
    return render(request, 'home.html', {'items': items})

Of course, that will now totally break the functional tests, because there is no such URL on our site yet. Sure enough, if you run them, you’ll find they fail just after trying to submit the first item, saying that they can’t find the list table; it’s because URL /the-only-list-in-the-world/ doesn’t exist yet!

    self.check_for_row_in_list_table('1: Buy peacock feathers')
[...]
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate
element: {"method":"id","selector":"id_list_table"}

So, let’s build a special URL for our one and only list.

Testing Views, Templates, and URLs Together with the Django Test Client

In previous chapters we’ve used unit tests that check the URL resolution explicitly, that test view functions by actually calling them, and that check that views render templates correctly too. Django actually provides us with a little tool that can do all three at once, which we’ll use now.

I wanted to show you how to "roll your own" first, partially because it’s a better introduction to how Django works, but also because those techniques are portable—you may not always use Django, but you’ll almost always have view functions, templates, and URL mappings, and now you know how to test them.

A New Test Class

So let’s use the Django test client. Open up lists/tests.py, and add a new test class called ListViewTest. Then copy the method called test_home_page_displays_all_ list_items across from HomePageTest into our new class, rename it, and adapt it slightly:

lists/tests.py (ch06l009). 

class ListViewTest(TestCase):

    def test_displays_all_items(self):
        Item.objects.create(text='itemey 1')
        Item.objects.create(text='itemey 2')

        response = self.client.get('/lists/the-only-list-in-the-world/') #1

        self.assertContains(response, 'itemey 1') #2
        self.assertContains(response, 'itemey 2') #3

1

Instead of calling the view function directly, we use the Django test client, which is an attribute of the Django TestCase called self.client. We tell it to .get the URL we’re testing—it’s actually a very similar API to the one that Selenium uses.

2 3

Instead of using the slightly annoying assertIn/response.content.decode() dance, Django provides the assertContains method which knows how to deal with responses and the bytes of their content.

Some people really don’t like the Django test client. They say it provides too much magic, and involves too much of the stack to be used in a real "unit" test—you end up writing what are more properly called integrated tests. They also complain that it is relatively slow (and relatively is measured in milliseconds). We’ll explore this argument further in a later chapter. For now we’ll use it because it’s extremely convenient!

Let’s try running the test now:

AssertionError: 404 != 200 : Couldn't retrieve content: Response code was 404

A New URL

Our singleton list URL doesn’t exist yet. We fix that in superlists/urls.py.

Watch out for trailing slashes in URLs, both here in the tests and in urls.py—They’re a common source of bugs.

superlists/urls.py. 

urlpatterns = [
    url(r'^$', views.home_page, name='home'),
    url(r'^lists/the-only-list-in-the-world/$', views.view_list, name='view_list'),
    # url(r'^admin/', include(admin.site.urls)),
]

Running the tests again, we get:

AttributeError: 'module' object has no attribute 'view_list'
[...]
FAILED (errors=4)

A New View Function

Nicely self-explanatory. Let’s create a dummy view function in lists/views.py:

lists/views.py. 

def view_list(request):
    pass

Now we get:

ValueError: The view lists.views.view_list didn't return an HttpResponse
object. It returned None instead.

Let’s copy the two last lines from the home_page view and see if they’ll do the trick:

lists/views.py. 

def view_list(request):
    items = Item.objects.all()
    return render(request, 'home.html', {'items': items})

Rerun the tests and they should pass:

Ran 8 tests in 0.016s
OK

And the FTs should get a little further on:

AssertionError: '2: Use peacock feathers to make a fly' not found in ['1: Buy
peacock feathers']

Green? Refactor

Time for a little tidying up.

In the Red/Green/Refactor dance, we’ve arrived at green, so we should see what needs a refactor. We now have two views, one for the home page, and one for an individual list. Both are currently using the same template, and passing it all the list items currently in the database. If we look through our unit test methods, we can see some stuff we probably want to change:

$ grep -E "class|def" lists/tests.py
class HomePageTest(TestCase):
    def test_root_url_resolves_to_home_page_view(self):
    def test_home_page_returns_correct_html(self):
    def test_home_page_displays_all_list_items(self):
    def test_home_page_can_save_a_POST_request(self):
    def test_home_page_redirects_after_POST(self):
    def test_home_page_only_saves_items_when_necessary(self):
class ListViewTest(TestCase):
    def test_displays_all_items(self):
class ItemModelTest(TestCase):
    def test_saving_and_retrieving_items(self):

We can definitely delete the test_home_page_displays_all_list_items method, it’s no longer needed. If you run manage.py test lists now, it should say it ran 7 tests instead of 8:

Ran 7 tests in 0.016s
OK

Next, we don’t actually need the home page to display all list items any more; it should just show a single input box inviting you to start a new list.

A Separate Template for Viewing Lists

Since the home page and the list view are now quite distinct pages, they should be using different HTML templates; home.html can have the single input box, whereas a new template, list.html, can take care of showing the table of existing items.

Let’s add a new test to check that it’s using a different template:

lists/tests.py. 

class ListViewTest(TestCase):

    def test_uses_list_template(self):
        response = self.client.get('/lists/the-only-list-in-the-world/')
        self.assertTemplateUsed(response, 'list.html')


    def test_displays_all_items(self):
        [...]

assertTemplateUsed is one of the more useful functions that the Django test client gives us. Let’s see what it says:

AssertionError: False is not true : Template 'list.html' was not a template
used to render the response. Actual template(s) used: home.html

Great! Let’s change the view:

lists/views.py. 

def view_list(request):
    items = Item.objects.all()
    return render(request, 'list.html', {'items': items})

But, obviously, that template doesn’t exist yet. If we run the unit tests, we get:

django.template.base.TemplateDoesNotExist: list.html

Let’s create a new file at lists/templates/list.html:

$ touch lists/templates/list.html

A blank template, which gives us this error—good to know the tests are there to make sure we fill it in:

AssertionError: False is not true : Couldn't find 'itemey 1' in response

The template for an individual list will reuse quite a lot of the stuff we currently have in home.html, so we can start by just copying that:

$ cp lists/templates/home.html lists/templates/list.html

That gets the tests back to passing (green). Now let’s do a little more tidying up (refactoring). We said the home page doesn’t need to list items, it only needs the new list input field, so we can remove some lines from lists/templates/home.html, and maybe slightly tweak the h1 to say "Start a new To-Do list":

lists/templates/home.html. 

<body>
    <h1>Start a new To-Do list</h1>
    <form method="POST">
        <input name="item_text" id="id_new_item" placeholder="Enter a to-do item" />
        {% csrf_token %}
    </form>
</body>

We rerun the unit tests to check that hasn’t broken anything … good…

There’s actually no need to pass all the items to the home.html template in our home_page view, so we can simplify that:

lists/views.py. 

def home_page(request):
    if request.method == 'POST':
        Item.objects.create(text=request.POST['item_text'])
        return redirect('/lists/the-only-list-in-the-world/')
    return render(request, 'home.html')

Rerun the unit tests; they still pass. Let’s run the functional tests:

AssertionError: '2: Use peacock feathers to make a fly' not found in ['1: Buy
peacock feathers']

We’re still failing to input the second item. What’s going on here? Well, it’s not immediately obvious, but it looks like our POST requests aren’t working the way they should. After a bit of head-scratching and digging through the various views and templates, we will eventually uncover the problem: both our forms are missing the action= attribute, which means that, by default, they submit to the same URL they were rendered from. That works for the home page, because it’s the only one that knows how to deal with POST requests currently, but it won’t work for our view_list function, which is just ignoring the POST.

We can fix that in lists/templates/list.html:

lists/templates/list.html (ch06l019). 

    <form method="POST" action="/">

And try running the FT again:

    self.assertNotEqual(francis_list_url, edith_list_url)
AssertionError: 'http://localhost:8081/lists/the-only-list-in-the-world/' ==
'http://localhost:8081/lists/the-only-list-in-the-world/'

Hooray! We’re back to where we were earlier, which means our refactoring is complete—we now have a unique URL for our one list. It may feel like we haven’t made much headway since, functionally, the site still behaves almost exactly like it did when we started the chapter, but this really is progress. We’ve started on the road to our new design, and we’ve implemented a number of stepping stones without making anything worse than it was before. Let’s commit our progress so far:

$ git status # should show 4 changed files and 1 new file, list.html
$ git add lists/templates/list.html
$ git diff # should show we've simplified home.html,
           # moved one test to a new class in lists/tests.py added a new view
           # in views.py, and simplified home_page and made one addition to
           # urls.py
$ git commit -a # add a message summarising the above, maybe something like
                # "new URL, view and template to display lists"

Another URL and View for Adding List Items

Where are we with our own to-do list?

We’ve sort of made progress on the third item, even if there’s still only one list in the world. Item 2 is a bit scary. Can we do something about items 4 or 5?

Let’s have a new URL for adding new list items. If nothing else, it’ll simplify the home page view.

A Test Class for New List Creation

Open up lists/tests.py, and move the test_home_page_can_save_a_POST_request and test_home_page_redirects_after_POST methods into a new class, then change their names:

lists/tests.py (ch06l021-1). 

class NewListTest(TestCase):

    def test_saving_a_POST_request(self):
        request = HttpRequest()
        request.method = 'POST'
        [...]

    def test_redirects_after_POST(self):
        [...]

Now let’s use the Django test client:

lists/tests.py (ch06l021-2). 

class NewListTest(TestCase):

    def test_saving_a_POST_request(self):
        self.client.post(
            '/lists/new',
            data={'item_text': 'A new list item'}
        )
        self.assertEqual(Item.objects.count(), 1)
        new_item = Item.objects.first()
        self.assertEqual(new_item.text, 'A new list item')


    def test_redirects_after_POST(self):
        response = self.client.post(
            '/lists/new',
            data={'item_text': 'A new list item'}
        )
        self.assertEqual(response.status_code, 302)
        self.assertEqual(response['location'], '/lists/the-only-list-in-the-world/')

This is another place to pay attention to trailing slashes, incidentally. It’s /new, with no trailing slash. The convention I’m using is that URLs without a trailing slash are "action" URLs which modify the database.

Try running that:

    self.assertEqual(Item.objects.count(), 1)
AssertionError: 0 != 1
[...]
    self.assertEqual(response.status_code, 302)
AssertionError: 404 != 302

The first failure tells us we’re not saving a new item to the database, and the second says that, instead of returning a 302 redirect, our view is returning a 404. That’s because we haven’t built a URL for /lists/new, so the client.post is just getting a 404 response.

Do you remember how we split this out into two tests in the last chapter? If we only had one test that checked both the saving and the redirect, it would have failed on the 0 != 1 failure, which would have been much harder to debug. Ask me how I know this.

A URL and View for New List Creation

Let’s build our new URL now:

superlists/urls.py. 

urlpatterns = [
    url(r'^$', views.home_page, name='home'),
    url(r'^lists/new$', views.new_list, name='new_list'),
    url(r'^lists/the-only-list-in-the-world/$', views.view_list, name='view_list'),
    # url(r'^admin/', include(admin.site.urls)),
]

Next we get a no attribute 'new_list', so let’s fix that, in lists/views.py:

lists/views.py. 

def new_list(request):
    pass

Then we get "The view lists.views.new_list didn’t return an HttpResponse object". (This is getting rather familiar!) We could return a raw HttpResponse, but since we know we’ll need a redirect, let’s borrow a line from home_page:

lists/views.py. 

def new_list(request):
    return redirect('/lists/the-only-list-in-the-world/')

That gives:

    self.assertEqual(Item.objects.count(), 1)
AssertionError: 0 != 1
[...]
AssertionError: 'http://testserver/lists/the-only-list-in-the-world/' !=
'/lists/the-only-list-in-the-world/'

Let’s start with the first failure, because it’s reasonably straightforward. We borrow another line from home_page:

lists/views.py. 

def new_list(request):
    Item.objects.create(text=request.POST['item_text'])
    return redirect('/lists/the-only-list-in-the-world/')

And that takes us down to just the second, unexpected failure:

    self.assertEqual(response['location'],
'/lists/the-only-list-in-the-world/')
AssertionError: 'http://testserver/lists/the-only-list-in-the-world/' !=
'/lists/the-only-list-in-the-world/'

It’s happening because the Django test client behaves slightly differently to our pure view function; it’s using the full Django stack which adds the domain to our relative URL. Let’s use another of Django’s test helper functions, instead of our two-step check for the redirect:

lists/tests.py. 

    def test_redirects_after_POST(self):
        response = self.client.post(
            '/lists/new',
            data={'item_text': 'A new list item'}
        )
        self.assertRedirects(response, '/lists/the-only-list-in-the-world/')

That now passes:

Ran 8 tests in 0.030s

OK

Removing Now-Redundant Code and Tests

We’re looking good. Since our new views are now doing most of the work that home_page used to do, we should be able to massively simplify it. Can we remove the whole if request.method == 'POST' section, for example?

lists/views.py. 

def home_page(request):
    return render(request, 'home.html')

Yep!

OK

And while we’re at it, we can remove the now-redundant test_home_page_only_saves_ items_when_necessary test too!

Doesn’t that feel good? The view functions are looking much simpler. We rerun the tests to make sure…

Ran 7 tests in 0.016s
OK

Pointing Our Forms at the New URL

Finally, let’s wire up our two forms to use this new URL. In both home.html and lists.html:

lists/templates/home.html, lists/templates/list.html. 

    <form method="POST" action="/lists/new">

And we rerun our FTs to make sure everything still works, or works at least as well as it did earlier…

AssertionError: 'http://localhost:8081/lists/the-only-list-in-the-world/' ==
'http://localhost:8081/lists/the-only-list-in-the-world/'

Yup, we get to the same point we did before. That’s a nicely self-contained commit, in that we’ve made a bunch of changes to our URLs, our views.py is looking much neater and tidier, and we’re sure the application is still working as well as it did before. We’re getting good at this refactoring malarkey!

$ git status # 5 changed files
$ git diff # URLs for forms x2, moved code in views + tests, new URL
$ git commit -a

And we can cross out an item on the to-do list:

Adjusting Our Models

Enough housekeeping with our URLs. It’s time to bite the bullet and change our models. Let’s adjust the model unit test. Just for a change, I’ll present the changes in the form of a diff:

lists/tests.py. 

@@ -3,7 +3,7 @@ from django.http import HttpRequest
 from django.template.loader import render_to_string
 from django.test import TestCase

-from lists.models import Item
+from lists.models import Item, List
 from lists.views import home_page

 class HomePageTest(TestCase):
@@ -60,22 +60,32 @@ class ListViewTest(TestCase):



-class ItemModelTest(TestCase):
+class ListAndItemModelsTest(TestCase):

     def test_saving_and_retrieving_items(self):
+        list_ = List()
+        list_.save()
+
         first_item = Item()
         first_item.text = 'The first (ever) list item'
+        first_item.list = list_
         first_item.save()

         second_item = Item()
         second_item.text = 'Item the second'
+        second_item.list = list_
         second_item.save()

+        saved_list = List.objects.first()
+        self.assertEqual(saved_list, list_)
+
         saved_items = Item.objects.all()
         self.assertEqual(saved_items.count(), 2)

         first_saved_item = saved_items[0]
         second_saved_item = saved_items[1]
         self.assertEqual(first_saved_item.text, 'The first (ever) list item')
+        self.assertEqual(first_saved_item.list, list_)
         self.assertEqual(second_saved_item.text, 'Item the second')
+        self.assertEqual(second_saved_item.list, list_)

We create a new List object, and then we assign each item to it by assigning it as its .list property. We check the list is properly saved, and we check that the two items have also saved their relationship to the list. You’ll also notice that we can compare list objects with each other directly (saved_list and list)—behind the scenes, these will compare themselves by checking their primary key (the .id attribute) is the same.

I’m using the variable name list_ to avoid "shadowing" the Python built-in list function. It’s ugly, but all the other options I tried were equally ugly or worse (my_list, the_list, list1, listey…).

Time for another unit-test/code cycle.

For the first couple of iterations, rather than explicitly showing you what code to enter in between every test run, I’m only going to show you the expected error messages from running the tests. I’ll let you figure out what each minimal code change should be on your own:

Your first error should be:

ImportError: cannot import name 'List'

Fix that, then you should see:

AttributeError: 'List' object has no attribute 'save'

Next you should see:

django.db.utils.OperationalError: no such table: lists_list

So we run a makemigrations:

$ python3 manage.py makemigrations
Migrations for 'lists':
  0003_list.py:
    - Create model List

And then you should see:

    self.assertEqual(first_saved_item.list, list_)
AttributeError: 'Item' object has no attribute 'list'

A Foreign Key Relationship

How do we give our Item a list attribute? Let’s just try naively making it like the text attribute:

lists/models.py. 

from django.db import models

class List(models.Model):
    pass

class Item(models.Model):
    text = models.TextField(default='')
    list = models.TextField(default='')

As usual, the tests tell us we need a migration:

$ python3 manage.py test lists
[...]
django.db.utils.OperationalError: no such column: lists_item.list

$ python3 manage.py makemigrations
Migrations for 'lists':
  0004_item_list.py:
    - Add field list to item

Let’s see what that gives us:

AssertionError: 'List object' != <List: List object>

We’re not quite there. Look closely at each side of the !=. Django has only saved the string representation of the List object. To save the relationship to the object itself, we tell Django about the relationship between the two classes using a ForeignKey:

lists/models.py. 

from django.db import models

class List(models.Model):
    pass


class Item(models.Model):
    text = models.TextField(default='')
    list = models.ForeignKey(List, default=None)

That’ll need a migration too. Since the last one was a red herring, let’s delete it and replace it with a new one:

$ rm lists/migrations/0004_item_list.py
$ python3 manage.py makemigrations
Migrations for 'lists':
  0004_item_list.py:
    - Add field list to item

Deleting migrations is dangerous. If you delete a migration that’s already been applied to a database somewhere, Django will be confused about what state it’s in, and how to apply future migrations. You should only do it when you’re sure the migration hasn’t been used. A good rule of thumb is that you should never delete a migration that’s been committed to your VCS.

Adjusting the Rest of the World to Our New Models

Back in our tests, now what happens?

$ python3 manage.py test lists
[...]
ERROR: test_displays_all_items (lists.tests.ListViewTest)
django.db.utils.IntegrityError: NOT NULL constraint failed: lists_item.list_id
[...]
ERROR: test_redirects_after_POST (lists.tests.NewListTest)
django.db.utils.IntegrityError: NOT NULL constraint failed: lists_item.list_id
[...]
ERROR: test_saving_a_POST_request (lists.tests.NewListTest)
django.db.utils.IntegrityError: NOT NULL constraint failed: lists_item.list_id

Ran 7 tests in 0.021s

FAILED (errors=3)

Oh dear!

There is some good news. Although it’s hard to see, our model tests are passing. But three of our view tests are failing nastily.

The reason is because of the new relationship we’ve introduced between Items and Lists, which requires each item to have a parent list, which our old tests weren’t prepared for.

Still, this is exactly why we have tests. Let’s get them working again. The easiest is the ListViewTest; we just create a parent list for our two test items:

lists/tests.py (ch06l031). 

class ListViewTest(TestCase):

    def test_displays_all_items(self):
        list_ = List.objects.create()
        Item.objects.create(text='itemey 1', list=list_)
        Item.objects.create(text='itemey 2', list=list_)

That gets us down to two failing tests, both on tests that try to POST to our new_list view. Decoding the tracebacks using our usual technique, working back from error, to line of test code, to the line of our own code that caused the failure, we identify:

File "/workspace/superlists/lists/views.py", line 14, in new_list
Item.objects.create(text=request.POST['item_text'])

It’s when we try and create an item without a parent list. So we make a similar change in the view:

lists/views.py. 

from lists.models import Item, List
[...]
def new_list(request):
    list_ = List.objects.create()
    Item.objects.create(text=request.POST['item_text'], list=list_)
    return redirect('/lists/the-only-list-in-the-world/')

And that gets our tests passing again:

OK

Are you cringing internally at this point? Arg! This feels so wrong, we create a new list for every single new item submission, and we’re still just displaying all items as if they belong to the same list! I know, I feel the same. The step-by-step approach, in which you go from working code to working code, is counterintuitive. I always feel like just diving in and trying to fix everything all in one go, instead of going from one weird half-finished state to another. But remember the Testing Goat! When you’re up a mountain, you want to think very carefully about where you put each foot, and take one step at a time, checking at each stage that the place you’ve put it hasn’t caused you to fall off a cliff.

So just to reassure ourselves that things have worked, we rerun the FT. Sure enough, it gets all the way through to where we were before. We haven’t broken anything, and we’ve made a change to the database. That’s something to be pleased with! Let’s commit:

$ git status # 3 changed files, plus 2 migrations
$ git add lists
$ git diff --staged
$ git commit

And we can cross out another item on the to-do list:

Each List Should Have Its Own URL

What shall we use as the unique identifier for our lists? Probably the simplest thing, for now, is just to use the auto-generated id field from the database. Let’s change ListViewTest so that the two tests point at new URLs.

We’ll also change the old test_displays_all_items test and call it test_displays_only_items_for_that_list instead, and make it check that only the items for a specific list are displayed:

lists/tests.py (ch06l033-1). 

class ListViewTest(TestCase):

    def test_uses_list_template(self):
        list_ = List.objects.create()
        response = self.client.get('/lists/%d/' % (list_.id,))
        self.assertTemplateUsed(response, 'list.html')


    def test_displays_only_items_for_that_list(self):
        correct_list = List.objects.create()
        Item.objects.create(text='itemey 1', list=correct_list)
        Item.objects.create(text='itemey 2', list=correct_list)
        other_list = List.objects.create()
        Item.objects.create(text='other list item 1', list=other_list)
        Item.objects.create(text='other list item 2', list=other_list)

        response = self.client.get('/lists/%d/' % (correct_list.id,))

        self.assertContains(response, 'itemey 1')
        self.assertContains(response, 'itemey 2')
        self.assertNotContains(response, 'other list item 1')
        self.assertNotContains(response, 'other list item 2')

If you’re not familiar with Python string substitutions, or the printf function from C, maybe that %d is a little confusing? Dive Into Python has a good overview, if you want to go look them up quickly. We’ll see an alternative string substitution syntax later in the book too.

Running the unit tests gives an expected 404, and another related error:

FAIL: test_displays_only_items_for_that_list (lists.tests.ListViewTest)
AssertionError: 404 != 200 : Couldn't retrieve content: Response code was 404
(expected 200)
[...]
FAIL: test_uses_list_template (lists.tests.ListViewTest)
AssertionError: No templates used to render the response

Capturing Parameters from URLs

It’s time to learn how we can pass parameters from URLs to views:

superlists/urls.py. 

urlpatterns = [
    url(r'^$', views.home_page, name='home'),
    url(r'^lists/new$', 'lists.views.new_list', name='new_list'),
    url(r'^lists/(.+)/$', 'lists.views.view_list', name='view_list'),
    # url(r'^admin/', include(admin.site.urls)),
]

We adjust the regular expression for our URL to include a capture group, (.+), which will match any characters, up to the following /. The captured text will get passed to the view as an argument.

In other words, if we go to the URL /lists/1/, view_list will get a second argument after the normal request argument, namely the string "1". If we go to /lists/foo/, we get view_list(request, "foo").

But our view doesn’t expect an argument yet! Sure enough, this causes problems:

ERROR: test_displays_only_items_for_that_list (lists.tests.ListViewTest)
ERROR: test_uses_list_template (lists.tests.ListViewTest)
ERROR: test_redirects_after_POST (lists.tests.NewListTest)
[...]
TypeError: view_list() takes 1 positional argument but 2 were given

We can fix that easily with a dummy parameter in views.py:

lists/views.py. 

def view_list(request, list_id):
    [...]

Now we’re down to our expected failure:

FAIL: test_displays_only_items_for_that_list (lists.tests.ListViewTest)
AssertionError: 1 != 0 : Response should not contain 'other list item 1'

Let’s make our view discriminate over which items it sends to the template:

lists/views.py. 

def view_list(request, list_id):
    list_ = List.objects.get(id=list_id)
    items = Item.objects.filter(list=list_)
    return render(request, 'list.html', {'items': items})

Adjusting new_list to the New World

Now we get errors in another test:

ERROR: test_redirects_after_POST (lists.tests.NewListTest)
ValueError: invalid literal for int() with base 10:
'the-only-list-in-the-world'

Let’s take a look at this test then, since it’s whining:

lists/tests.py. 

class NewListTest(TestCase):
    [...]

    def test_redirects_after_POST(self):
        response = self.client.post(
            '/lists/new',
            data={'item_text': 'A new list item'}
        )
        self.assertRedirects(response, '/lists/the-only-list-in-the-world/')

It looks like it hasn’t been adjusted to the new world of Lists and Items. The test should be saying that this view redirects to the URL of the new list it just created:

lists/tests.py (ch06l036-1). 

    def test_redirects_after_POST(self):
        response = self.client.post(
            '/lists/new',
            data={'item_text': 'A new list item'}
        )
        new_list = List.objects.first()
        self.assertRedirects(response, '/lists/%d/' % (new_list.id,))

That still gives us the invalid literal error. We take a look at the view itself, and change it so it redirects to a valid place:

lists/views.py (ch06l036-2). 

def new_list(request):
    list_ = List.objects.create()
    Item.objects.create(text=request.POST['item_text'], list=list_)
    return redirect('/lists/%d/' % (list_.id,))

That gets us back to passing unit tests. What about the functional tests? We must be almost there?

AssertionError: '2: Use peacock feathers to make a fly' not found in ['1: Use
peacock feathers to make a fly']

The functional tests have warned us of a regression in our application: because we’re now creating a new list for every single POST submission, we have broken the ability to add multiple items to a list. This is exactly what we have functional tests for!

And it correlates nicely with the last item on our to-do list:

One More View to Handle Adding Items to an Existing List

We need a URL and view to handle adding a new item to an existing list ( /lists/<list_id>/add_item). We’re getting pretty good at these now, so let’s knock one together quickly:

lists/tests.py. 

class NewItemTest(TestCase):

    def test_can_save_a_POST_request_to_an_existing_list(self):
        other_list = List.objects.create()
        correct_list = List.objects.create()

        self.client.post(
            '/lists/%d/add_item' % (correct_list.id,),
            data={'item_text': 'A new item for an existing list'}
        )

        self.assertEqual(Item.objects.count(), 1)
        new_item = Item.objects.first()
        self.assertEqual(new_item.text, 'A new item for an existing list')
        self.assertEqual(new_item.list, correct_list)


    def test_redirects_to_list_view(self):
        other_list = List.objects.create()
        correct_list = List.objects.create()

        response = self.client.post(
            '/lists/%d/add_item' % (correct_list.id,),
            data={'item_text': 'A new item for an existing list'}
        )

        self.assertRedirects(response, '/lists/%d/' % (correct_list.id,))

We get:

AssertionError: 0 != 1
[...]
AssertionError: 301 != 302 : Response didn't redirect as expected: Response
code was 301 (expected 302)

Beware of Greedy Regular Expressions!

That’s a little strange. We haven’t actually specified a URL for /lists/1/add_item yet, so our expected failure is 404 != 302. Why are we getting a 301?

This was a bit of a puzzler, but it’s because we’ve used a very "greedy" regular expression in our URL:

    url(r'^lists/(.+)/$', 'lists.views.view_list', name='view_list'),

Django has some built-in code to issue a permanent redirect (301) whenever someone asks for a URL which is almost right, except for a missing slash. In this case, /lists/1/add_item/ would be a match for lists/(.+)/, with the (.+) capturing 1/add_item. So Django "helpfully" guesses that we actually wanted the URL with a trailing slash.

We can fix that by making our URL pattern explicitly capture only numerical digits, by using the regular expression \d:

superlists/urls.py. 

    url(r'^lists/(\d+)/$', views.view_list, name='view_list'),

That gives:

AssertionError: 0 != 1
[...]
AssertionError: 404 != 302 : Response didn't redirect as expected: Response
code was 404 (expected 302)

The Last New URL

Now we’ve got our expected 404, let’s add a new URL for adding new items to existing lists:

superlists/urls.py. 

urlpatterns = [
    url(r'^$', views.home_page, name='home'),
    url(r'^lists/new$', views.new_list, name='new_list'),
    url(r'^lists/(\d+)/$', views.view_list, name='view_list'),
    url(r'^lists/(\d+)/add_item$', views.add_item, name='add_item'),
    # url(r'^admin/', include(admin.site.urls)),
]

Three very similar-looking URLs there. Let’s make a note on our to-do list; they look like good candidates for a refactoring.

Back to the tests, we get the usual missing module objects:

AttributeError: 'module' object has no attribute 'add_item'

The Last New View

Let’s try:

lists/views.py. 

def add_item(request):
    pass

Aha:

TypeError: add_item() takes 1 positional argument but 2 were given

lists/views.py. 

def add_item(request, list_id):
    pass

And then:

ValueError: The view lists.views.add_item didn't return an HttpResponse object.
It returned None instead.

We can copy the redirect from new_list and the List.objects.get from view_list:

lists/views.py. 

def add_item(request, list_id):
    list_ = List.objects.get(id=list_id)
    return redirect('/lists/%d/' % (list_.id,))

That takes us to:

    self.assertEqual(Item.objects.count(), 1)
AssertionError: 0 != 1

Finally we make it save our new list item:

lists/views.py. 

def add_item(request, list_id):
    list_ = List.objects.get(id=list_id)
    Item.objects.create(text=request.POST['item_text'], list=list_)
    return redirect('/lists/%d/' % (list_.id,))

And we’re back to passing tests.

Ran 9 tests in 0.050s

OK

But How to Use That URL in the Form?

Now we just need to use this URL in our list.html template. Open it up and adjust the form tag…

lists/templates/list.html. 

    <form method="POST" action="but what should we put here?">

... oh. To get the URL for adding to the current list, the template needs to know what list it’s rendering, as well as what the items are. We want to be able to do something like this:

lists/templates/list.html. 

    <form method="POST" action="/lists/{{ list.id }}/add_item">

For that to work, the view will have to pass the list to the template. Let’s create a new unit test in ListViewTest:

lists/tests.py (ch06l041). 

    def test_passes_correct_list_to_template(self):
        other_list = List.objects.create()
        correct_list = List.objects.create()
        response = self.client.get('/lists/%d/' % (correct_list.id,))
        self.assertEqual(response.context['list'], correct_list)

response.context represents the context we’re going to pass into the render function—the Django test client puts it on the response object for us, to help with testing. That gives us:

KeyError: 'list'

because we’re not passing list into the template. It actually gives us an opportunity to simplify a little:

lists/views.py. 

def view_list(request, list_id):
    list_ = List.objects.get(id=list_id)
    return render(request, 'list.html', {'list': list_})

That, of course, will break because the template is expecting items:

AssertionError: False is not true : Couldn't find 'itemey 1' in response

But we can fix it in list.html, as well as adjusting the form’s POST action:

lists/templates/list.html (ch06l043). 

    <form method="POST" action="/lists/{{ list.id }}/add_item">

    [...]

        {% for item in list.item_set.all %}
            <tr><td>{{ forloop.counter }}: {{ item.text }}</td></tr>
        {% endfor %}

.item_set is called a "reverse lookup"—it’s one of Django’s incredibly useful bits of ORM that lets you look up an object’s related items from a different table…

So that gets the unit tests to pass:

Ran 10 tests in 0.060s

OK

How about the FT?

$ python3 manage.py test functional_tests
Creating test database for alias 'default'...
.
 ---------------------------------------------------------------------
Ran 1 test in 5.824s

OK
Destroying test database for alias 'default'...

Yes! And a quick check on our to-do list:

Irritatingly, the Testing Goat is a stickler for tying up loose ends too, so we’ve got to do this one final thing.

Before we start, we’ll do a commit—always make sure you’ve got a commit of a working state before embarking on a refactor:

$ git diff
$ git commit -am "new URL + view for adding to existing lists. FT passes :-)"

A Final Refactor Using URL includes

superlists/urls.py is really meant for URLs that apply to your entire site. For URLs that only apply to the lists app, Django encourages us to use a separate lists/urls.py, to make the app more self-contained. The simplest way to make one is to use a copy of the existing urls.py:

$ cp superlists/urls.py lists/

Then we replace three lines in superlists/urls.py with an include. Notice that include can take a part of a URL regex as a prefix, which will be applied to all the included URLs (this is the bit where we reduce duplication, as well as giving our code a better structure):

superlists/urls.py. 

from django.conf.urls import include, url
from lists import views as list_views  #1
from lists import urls as list_urls  #2

urlpatterns = [
    url(r'^$', list_views.home_page, name='home'),
    url(r'^lists/', include(list_urls)),
    # url(r'^admin/', include(admin.site.urls)),
]

1 2

While we’re at it, we use the import x as y syntax to alias views and urls. This is good practice in your top-level urls.py, because it will let us import views and urls from multiple apps if we need to—and indeed we will need to later on in the book.

Back in lists/urls.py we can trim down to only include the latter part of our three URLs, and none of the other stuff from the parent urls.py:

lists/urls.py (ch06l045). 

from django.conf.urls import url
from lists import views

urlpatterns = [
    url(r'^new$', views.new_list, name='new_list'),
    url(r'^(\d+)/$', views.view_list, name='view_list'),
    url(r'^(\d+)/add_item$', views.add_item, name='add_item'),
]

Rerun the unit tests to check everything worked. When I did it, I couldn’t quite believe I did it correctly on the first go. It always pays to be skeptical of your own abilities, so I deliberately changed one of the URLs slightly, just to check if it broke a test. It did. We’re covered.

Feel free to try it yourself! Remember to change it back, check the tests all pass again, and then commit:

$ git status
$ git add lists/urls.py
$ git add superlists/urls.py
$ git diff --staged
$ git commit

Phew. A marathon chapter. But we covered a number of important topics, starting with test isolation, and then some thinking about design. We covered some rules of thumb like "YAGNI" and "three strikes then refactor". But, most importantly, we saw how to adapt an existing site step by step, going from working state to working state, in order to iterate towards a new design.

I’d say we’re pretty close to being able to ship this site, as the very first beta of the superlists website that’s going to take over the world. Maybe it needs a little prettification first … let’s look at what we need to do to deploy it in the next couple of chapters.



[9] Are you unable to move on because you’re wondering what those ch06l0xx things are, next to some of the code listings? They refer to specific commits in the book’s example repo. It’s all to do with my book’s correctness tests. You know, the tests for the tests in the book about testing. They have tests of their own, incidentally.