How to read research papers

Photo by ailatan on Flickr

These last couple of weeks I've been taking my groups of final year project students through the process of starting their literature reviews. There is a separate post on literature reviews on this site here and a post on why you should read academic literature in Computer Science here. This post isn't to do with those topics, this post is about how to read research papers. We often find that if students haven't done much of this sort of reading before their get to their final year getting started can be a bit of a shock. So, this post is designed to help you get started with academic literature and, just as importantly, to help you get the most out of the papers you read in the short space of time you have available (and it is a short space of time, believe me).

Remember the structure of a paper is just like the structure of your thesis

Other posts on this site have discussed the overall structure of your thesis, but in outline this is the sort of structure you should be expecting to produce:

  • Introduction should introduce the reader to your research question and the broad context of the research.
  • Literature review should describe the work that other people have carried out to answer your (or similar) research questions.
  • Method should describe what you did to answer your research question (or to support your thesis, if you think of it that way), and how you went about it. 
  • Results should evaluate what you have done, and say what answer (to your research question) you have arrived at. 
  • Conclusions should summarise what you have done and how you answered the research question.  

Academic writing (in the sciences) of all sorts follows something like this structure, including all of the papers that you will be reading for your project. There are a couple of exceptions to this rule. One is theoretical papers which sometimes put their "related work" (or literature review) somewhere towards the end of the paper rather than after the introduction. The second exception is survey papers. Surveys are extended literature reviews and, as such, are a good place to start in your own literature reviews. ACM Computing Surveys is a journal that publishes survey papers or you can sometimes find them in reputable journals.

Briefly review each paper for relevance

You don't have time to read everything, so it's important to make sure that what you do read is really relevant to your thesis. So, to check whether a paper is likely to be relevant to you first read the Abstract. This should give you a brief summary of the whole paper. So, at the very least the abstract should give you a good idea of what research question the authors were trying to answer. Next, read the Conclusions. This is also likely to be a summary and may well give you a better idea of what results the authors obtained and what work they did not finish but left for "future work". If that doesn't give you a good enough idea of the relevance of the paper to your own work, try reading the last part of the Introduction. This is usually where the authors summarise what is written in each of the following sections of the paper, so that should give you a much more detailed view of what the rest of the paper contains. 

If, after all of that, you think the paper is irrelevant to you, then discard it and move on to something else. Otherwise, you are ready to move on with your reading...

Focus your reading on specific questions

If you just go ahead and read a paper from start to finish the chances are that you won't get very much out of your efforts. You are likely to ramble around the paper, not taking very detailed notes and at the end of your efforts you may not have learned much. A much better way to go about your reading is to keep in mind a number of clear, focussed questions and read the paper with the intention of writing down answers to these questions in your notes. That way you will finish with a clear set of notes that you can be confident will be useful to you when you start writing up.

I would recommend you use this set of questions to guide your reading:

  1. What research question were the authors asking?
  2. Why did the authors believe that their research question was important?
  3. How did the authors go about answering their research question?
  4. What results did the authors obtain or, what did the authors learn from answering their research question?

You can find a template for some notes here.

Making use of your notes

When you have finished reading you should have a stack of notes on all the papers you have read. This should be a much more concise way to start writing up than having a much bigger stack of papers and (most likely) not much memory of what was in them! So, the next thing to do is decide on the structure of your literature review chapter.

The first paragraph of your chapter should introduce the rest of the chapter. This is a good place to remind the reader of your research question and explain how the current chapter relates to it. 

The last paragraph of your chapter should summarise what you have reviewed. This is a good chance to help the reader naviagte around your thesis. Briefly review what you have said in the chapter and refer the reader to the next chapter, explaining how the next chapter follows on from the current one.

The middle part of the chapter is more difficult and, since your writing will depend on your particular research question and the literature you have read, there isn't much generic advice to be given here. However, you can start by reading through your notes and looking for common themes. Think about how best to present the ideas to a reader who has not read the same literature. Do you want to take the reader chronologically through the literature, from the earliest point to the present day? Would it be easier to understand if you split the reading into particular topics that are related? When you have what you think is a good structure, write some section headings into your thesis and think about which papers go in which sections (of course, some papers may well go into several sections). Write the citations into each section using something like EndNote, Mendeley or BibTeX to format them for you. Play around with the structure until you are convinced that it will make sense then write in the details of each section. Make sure you check out this post to help you with your writing.

Tagged academic_writing journals literature literature_review papers reading writing

Top 9 writing mistakes made by project students

325752626_69392aa6b1

Photo by athena on Flickr

A large part of any final year thesis project is the write-up, but every stage of the project will involve some form of writing. Whether you are writing a proposal, an interim report, a draft report, documentation or the final thesis, a very large part of your time will be spent composing text. These are the top ten mistakes we see from students year on year, avoid them and you can save yourself a lot of time (and earn a lot of marks)...

1. Unsubstantiated claims

As a general rule, any statement of fact or opinion about your work, or your topic of study should be substantiated in some way. You don't have to worry about the bleeding obvious, we know that 1+1=2 (unless your work is in the "foundations" of mathematics or philosophy, in which case 1+1 may well be undefined), but anything less obvious should be backed up by solid evidence. That evidence can be a reference to literature, an argument you make in your writing, the results of your own experiments, or any other suitably rigorous evidence.

Avoid so-called sweeping statements that are effectively impossible to substantiate "all Object Oriented programs couple algorithms and data". Do they? All of them? "Testing improves the performance of students". Really? These sorts of claims are poor academic practice and suggest that you have been a little sloppy in your thinking about your work, and of course that's not the impression you want to give.

2. Dangling pronouns and other references

This seems to be an almost universal problem with student writing. Consider the following paragraph:

In his 1953 paper, Henry Gordon Rice proved that for any non-trivial property of a partial function there is no general decision method to determine an algorithm that computes a partial function with that property. It has far reaching consequences for compilers, static analysis and other fields in practical computing.

What does the "It" in the second sentence refer to? Rice's theorem, his paper, or something else? It isn't clear from the text, although we can guess that the author meant to discuss the theorem. Better though, to be clear about the meaning in the first place. Every pronoun ("I", "he", "she", "it", "that", "who", etc.) should clearly refer to exactly one noun. The first sentence gets this right, it is clear that "his" refers to the noun "Henry Gordon Rice" and not any other noun in that sentence. So, we could improve the paragraph above by re-writing it:

In his 1953 paper, Henry Gordon Rice proved that for any non-trivial property of a partial function there is no general decision method to determine an algorithm that computes a partial function with that property. Rice's theorem has far reaching consequences for compilers, static analysis and other fields in practical computing.

3. Using a secondary source rather than a primary one

It's much easier to read the popular press, blogs and other "informal" media than research papers. Very few marks will be awarded for this, though. In a final year project you need to show that you can perform a small academic study, so marks will be available for reading peer-reviewed academic literature. There are two major pitfalls to avoid here. Firstly, you will occasionally come across some disreputable conference or journal which does not use peer-review. Worse, it is possible to come across "articles" on the Internet which have citations and publishing records and look, to all intents and purposes, like a genuine piece of academic writing, but have actually never been submitted to a journal or conference. This is very poor practice on the part of anyone who puts this kind of thing up on their own blog or website, but it does occasionally happen. A reputable publishing venue will have some sort of statement on their website stating how articles are reviewed (look for "Instructions to Authors"). This should say that every article is reviewed by at least two people and sent back for corrections before being published. Without this sort of peer-review any poor standard of work can be "published" without anyone checking even basic standard of good practice, such as detecting plagiarism. That said, even with peer-review, some poor practice still slips through the net.

Secondly, whatever references you cite should be primary, rather than secondary sources. A primary source is one where the author(s) reports work that s/he (they) have personally carried out. A secondary source is one where an author reports on work that someone else has completed and published elsewhere. Secondary sources include news reports, magazine articles and blog posts about research completed by others.

4. Confusing structure and use before definition

Any academic writing should generally be written for a reader who is an expert in the general field of the study, but not necessarily in the specific area of the study. If, for example, your final year project is on genetic algorithms, your work might be marked by an expert in artificial intelligence, or in computer science generally, but not necessarily by an expert in GAs. So, write with that in mind, and make sure that you don't use any specific technical terms without defining them. This is often very difficult to get right at first, especially when you have been working with your own ideas for a very long time. A good plan is to swap drafts of your work with a fellow student who is on the same degree course but working in a completely different field for their final year project. If you can both understand each others work, that's fine. If not, make changes.

5. Claims of "proof"

At the beginning and end of your dissertation you will want to set out the aims of your work and describe the conclusions you have reached. Occasionally we see students writing sentences such as "this study proves that ...", "this thesis will prove ...", and so on. "Proof" is specifically a mathematical method, and if you have genuinely proven a theorem, by all means say so. If not, then don't use the word "prove" and be very careful about what you do claim. For example ...

  • If you have tested a piece of software and it has passed all your test cases, then you have shown that your software is free of the specific errors you have checked for. You have not shown that it is "error-free" or "works".
  • If you have performed some sort of user testing, or any other usability / accessibility testing, then you may have demonstrated that the system under study is "usable" or "accessible" as far as you have tested it. However, without a large-scale study that is as much as you can claim. Be very circumspect about reading research in the area of usability; there is much good research but also much which is over-blown. Be especially careful of authors who also run consultancy practices, make sure you cite their academic literature, and not anything that could be considered advertising. Make sure everything you cite is peer-reviewed.
  • Be very, very careful about using questionnaires. I usually tell all my students to just avoid them altogether. It is very, very hard to produce a questionnaire which holds up under academic scrutiny and you will need an amount of statistical sophistication to produce sensible results. Also, you need a very large sample-size because your questions will be circumscribed (and for other reasons). This makes questionnaires very difficult to use in short, single-person projects. If you are in any doubt, then use a "semi-structured interview" to interview test subjects and the "talk-aloud protocol" or "cognitive dimensions of notation" for usability testing. Before you start, read some papers on evaluation methods, such as Hollingsed and Novick (2007) Usability Inspection Methods after 15 Years of Research and Practice, or Hornbaek and Law (2007) Meta-Analysis of Correlations Among Usability Measures.

6. Ad-hominem remarks

We see this very rarely, but just occasionally a student will forget that they should be writing about academic research, and criticise an author directly. Examples include "$X is stupid", "it would be stupid to think that..." and so on. Don't do this, stick with the research and don't criticise the person.

7. Don't be meta

If there's one piece of advise students regularly misunderstand it's that you should be cautious and critical of the literature that you read and cite. Of course, you should be critical and cautious, but you should also be sure that you are writing about the content of the research that you are reading about, not the quality of the writing. I should probably say, this piece of advice does not hold up if you are studying literary criticism, or anything similar, but for science-based subjects, stick with the science. Don't say things like:

(Foo Bar, 2009)  is a poorly written paper. It is confused and hard to follow.

Also, don't say things like:

The problem with this paper is ....

If there's a problem with the research that the paper describes, then by all means discuss that, but not the writing.

8. Don't write a giant list

I've written a separate post on literature reviews, but one common mistake we often see here is to structure a thesis as if it is a list of points, not a single piece of prose. Do not write about the literature in your field one paper at a time, try to tell a story about how the field has developed, culminating in saying that there is clearly a gap in the research where your contribution can fit. So, avoid writing like this:

Foo Bar (2007) On the usability of Flibble widgets

Bar describes the Flibble widget, which is used for ...

instead, work your thoughts on Flibble widgets into a longer piece of writing on widgets. If you do need to break up and structure your literature review, make sure your headings are topics which group together related pieces of research.

9. Don't replace elegant paragraphs with bullet points

A very common error we see in project write ups is where the student writes a series of ‘bullet lists’ instead of connected and well supported prose. The problem with this is that bullet lists do not often demonstrate complex thinking, rather, they simply provide ‘shallow’ summaries of topics (when we need some depth!). For example:

“People use CSS for:

  • Interoperability
  • Future proofing
  • Accessibility
  • Conforming to standards”

In this example – what does ‘interoperability’ mean?  Why does CSS provide ‘future proofing’?  Is CSS the only way to enable accessibility?  There is SO MUCH TO SAY on these subjects!  The bullet list destroys your chance to demonstrate your research and ideas.

Tagged academic_writing dissertation thesis writing

How to write a literature review for your final year thesis project



A long time ago I wrote an article on how to pass your final year thesis project, which several students found helpful. In the same vein, this post deals with a particular aspect of the final year project: the literature review.

Every year I supervise projects I find students tend to ask the same questions about their literature review. Most common are:

  • How many papers should I read?
  • How long should the literature review be?
  • Should I read books, articles, or ...?
  • Is it OK to reference websites such as Wikipedia?
  • Who will read my literature review and what can I assume about their knowledge of the area?
  • When should I start the literature review and when should it be finished?


These questions crop up frequently and will be familiar to any readers who are starting their own project. However, when you fully understand the purpose of the literature and how to go about writing one, you begin to realise that these questions are actually not that important. This post is designed to help students make that transition, from not yet understanding what the literature review is for, to having a thorough understanding of its purpose and a clear idea of how to write it up.

The meaning of a final year project

A lot of students starting off their projects talk about writing a "report" at the end of their "project" and doing some "research" as part of their literature review. This is where the subtleties of the English language tend to cause a lot of confusion. "Research" can have many different meanings, and to complete a really successful final year project the first thing to do it to understand fully what is expected of you in an academic context. Even if your final year at University is the last experience you have of academic work, remember that your work will be marked according to academic criteria, and academia has quite different aims to industry.

So, to be clear: research, in an academic context, means adding something new to the body of knowledge that humans have gathered in your area of interest. Your project as a whole will be a piece of research because you will be creating something new that has not been created before. What that is exactly will depend on the field you are studying. It might be a new perspective on a piece of literature, a new proof of a theorem, a new application of a particular technology, or something else. Since you are still an undergraduate it is likely (although not necessary) that your work will be a small step forward. It is unlikely that you will produce something completely ground breaking, so don't be intimidated by fact that your work has to be novel. That said, it may be that you produce an excellent piece of work and your supervisor may want to turn that into a technical report or conference paper with you, which would be great for your CV (or resume).

A thesis is a statement of belief that is central to your research. Your dissertation will be a piece of writing that defends your thesis, based on your research. So, for example, if your thesis is regular, online tests help University students to learn new material then you will need to implement some sort of online tests for new material, design and run an experiment to test your thesis, and write it up in your dissertation. Equally, if your thesis is water causes cancer in mice then you will need to plan and run an experiment to determine whether or not this is true and write it up. Notice that you may disprove your thesis in your work. It may be that online tests do not help students learn, or that water doesn't cause cancer in mice. This is absolutely fine, so long as your experiments give a clear answer to the question and you can show that your experiments were performed fairly it doesn't matter whether your thesis turns out to be incorrect or correct (in so far as you have tested it). It may also be that your evaluation is inconclusive, which is also acceptable, so long as your experimental method is good and you can say exactly what further work is necessary to produce a definite result, you will be fine.

Alternatively, you might phrase your thesis as a research question. In which case, instead of having a thesis such as water causes cancer in mice you would ask the question does water cause cancer in mice and your dissertation would describe your efforts to answer that question.

The shape of your dissertation and where the literature fits in

Every dissertation is slightly different, but good dissertations will all contain the same elements. I should say that the advice given in this sections is likely only relevant to science based projects. If you are working in the arts or some areas in the humanities then the expectations of you may well be very different. Still, a good dissertation in the sciences will contain roughly the elements listed below. I say "roughly" because, depending on the exact nature of your work, it may be sensible to expand some sections into two chapters rather than one, or to coalesce some elements into a single chapter. Your supervisor can give you more specific advice on this.

  • Introduction: should introduce the reader to the broad context of the research and explain why this is an interesting area to work in. So, if your thesis is something to do with mobile computing, you might say something here about why mobile phones are important, why mobile computing is an interesting and important area, and broadly what other researchers are working on. At the end of the chapter you will want to introduce your specific research question, having said why the area you are working in (and therefore your question) is important.
  • Literature review: Now you have introduced the reader (who will likely not be an expert in your exact area) to the broad research agenda in the field, and your research question, you can start writing more specifically about your own project. In this chapter you will survey the work that other researchers have done to answer your research question, or related questions. At the end of the chapter you should briefly explain how your own work builds on and differs from the work that has gone before it.
  • Method: this chapter should describe what you did to answer your research question (or to support your thesis, if you think of it that way), and how you went about it. You should describe your work in sufficient detail that another researcher could recreate your work to check your results.
  • Evaluation: here, you should evaluate what you have done, and say what answer (to your research question) you have arrived at. It may be that in your method you describe some experiments, and this section records your results and analysis of those results. This is an important section -- most students gain or lose marks in either their literature review or evaluation. Key to producing a convincing evaluation is to plan very early in the project what information you will need to write this section. More on that in another blog post.
  • Conclusions: should summarise what you have done and how you answered the research question. It may be that your work produced a very clear answer to the question, or it may be that your work points to a need for further research to clarify or confirm your answer. You should refer back to the literature review and summarise how your research differs from (hopefully improves on) the work described in the literature. Make sure you also say what research you would do if you were to continue working on your project.
  • References: a list of publications cited in the main text, in Harvard style or similar format.

It is likely that most chapters will be roughly the same size, although the introductory chapter and conclusions are usually slightly shorter than the others. Try to let the lengths of each chapter be guided by the amount of useful and important information you have to convey to the reader, don't impose artificial word limits on yourself.

Summaries and synthesis: what should go in your literature review

Poor literature reviews often take the same form -- they tend to be a (usually short) list of papers that the student has read, briefly summarised. This is not really what is expected and will not gain high marks. Another common mistake is to review literature that has been used to inform some part of the practical work of the project, rather than to review work that has answered the same or related research questions. To do better, your writing needs to not only summarise the prior art in your area, but also synthesise what is in the literature. In fact, synthesis is one of the key skills that we expect to see from final year students.

So, what is synthesis? The main idea is that you should have understood the literature you have read and, more importantly, you should show that you understand the relationships between items of literature. That means what came first in your field, how it influenced later work, how each step forward in the research improved upon what came before it, and so on. Ideally, you will present your own view of the work you are describing. This partly means that you should be critical of the literature you read, and say where the shortcomings of the work are, and how the work could be improved upon (in particular how your work will improve on the prior art). Also, you might have your own view on where your area of research is likely to go in the future.

Critiquing the work of others is something that is often new to students in their final year. One of the most frequent mistakes I see from students is to criticise the style of the papers they read, rather than the research that those papers describe. Avoid writing things like "this paper is not well written" or "this paper is hard to understand". A literature review should really be a review of the research work that has gone before you, not a literary criticism of the style that other authors adopt.

Practical matters: how to start, how to finish and how to do the bit in the middle

Reading and understanding the work of others is a lifetimes work for professional researchers, it is not something that starts and stops on particular dates, according to a Gantt chart. Your final year project will have a hand in deadline, so you need to be a little more circumscribed about how work. 

Ideally, you should be reading some literature in your field very early on in your project, to help you choose a good topic and write an initial research proposal. I would suggest that you consider this to be the starting point for your literature review and keep on reading and adding to your writing all the way through your project until you hand in your final dissertation. To do this, I suggest you do two things. Firstly, keep a careful log of what you read in a format you find easy to work with. This might be a a log book on paper, or it might be a file on your computer, whatever works best for you. Each entry should include the title of the work you have read and enough information about the authors and so on that you can find the publication again. You should then summarise the work in the paper, including the research question answered by the work, the nature of the answer and the methodology of the research (i.e. what the authors actually did). This will give you a couple of advantages -- you won't forget anything you read because you have a record of it, your literature review can be written from your notes and if you are asked any awkward questions in a viva you can refer back to your log. Secondly, as soon as you feel you have read enough to understand the broad context of the literature, start writing it up formally as your second dissertation chapter. This should really be within two or three months of your starting date. Then, as the project progresses and you read more, you can integrate your new reading into your already drafted chapter. This might seem like a lot of effort early on, but when you come to write up your work, you will be incredibly grateful that your most time consuming chapter has already been written, when it was fresh in your mind, leaving you free to write up the later sections of your work.

An example of good writing

Now you know what a literature review is for, how it fits into your dissertation and how to go about writing your own, it would probably be useful to see an example of (part of) an example review. The paragraphs below give such an example and the text [in italics] is some commentary to explain how each part of the writing contributes towards the start of a good thesis chapter. When you read this, don't worry too much about the subject matter, just try to concentrate on the style of writing and the structure of the text. 

The area of pervasive, or ubiquitous, computing was founded by Wieser (1991) [ referenced] who predicted that computers would one day be integrated into everyday objects and interact with people seamlessly. Although few such products are available today Weiser’s work has led to the creation of a number of research areas, including ambient intelligence (Eli and Epstein 1998), smart dust (Khan et al, 1999) and the Internet of Things (Brickley et al, 2001). [Sets the historical context of the area and defines related areas.]

An early application of pervasive computing was the active badge location system, described by Want et al (1992), in which users and objects were tagged with an "active" badge which could locate and identify them. This system was based on ultrasound locationing, whereas later systems might use RFID technology to achieve the same effect. [describes how the field has changed over time] Uses of the active badge system included routing phone calls, email alerts and so on to the physical location of the receiver. [contextualises the fundamental research]
 
...

 

Tagged academic_writing dissertation literature literature_review thesis writing
​