Skip to content

UMD Research Software Engineering Guide

This document outlines practical guidelines for research software projects. These recommended practices are intended to help researchers increase productivity, improve the reliability and reproducibility of their work, and to make it easier for others (and their future selves) to understand, use, and build upon their work. For more background, see the Best Practices for Scientific Computing: Introduction

1. Data Management

Having a plan for organizing, cleaning, and storing data can make your code easier to write and more understandable. It can also aid in debugging and sharing data.

Resources:

Good enough practices in scientific computing: Data management provides useful guidelines for storing, backing up, and cleaning (pre-processing) data so that it will be ready for analysis and for sharing with others.

The Turing Way: Research Data Management This section goes into detail on topics such as creating a data management plan, FAIR (Findability, Accessibility, Interoperability and Reusability) principles, and sharing and archiving data. The subsection The Turing Way: Data Storage and Organisation has excellent practical tips for data storage, such as directory structures, file naming conventions, and backing up data.

Best Practices for Scientific Computing: Don't Repeat Yourself (or Others) has this useful piece of advice: "Every piece of data must have a single authoritative representation in the system."

2. Project Planning & Design

Spending some time on planning and design before beginning coding will allow the software development work to move ahead more productively and with fewer surprises. It also allows for better tracking of progress toward key goals.

Resources:

Research Software Engineering with Python: Software Project Management describes the design stages in the context of the software lifecycle. It discusses and recommends the agile methodology for software planning, development, and team organization.

The conclusion describes a practical way of incorporating design into a project without sacrificing flexibility:

"Conclusion - Don't ignore design - Do try to write down some user stories - See if there's a known design pattern that will help - Do try to think about how your code will work before you start typing - Do maintain design documents.

"BUT

  • Do change your design as you work, updating the documents if you have them
  • Don't go dark - never do more than a couple of weeks programming without showing what you've done to colleagues
  • Don't get isolated from the reasons for your code's existence, stay involved in the research, don't be a Code Monkey.
  • Do keep a list of all the things your code needs, estimate and prioritise tasks carefully."

The Turing Way: Guide for Project Design provides information about project planning and management. It goes into detail on a variety of relevant topics. It urges researchers, during design, to consider project scope, goals, compliance, timeline, budget, risks, and team and project organization, as well as other important aspects of a project.

3. Working in Teams

In order for a team to work together effectively, it is important to welcome and encourage team members' contributions and to choose technologies to be used for collaboration.

Consider adopting some tenets of the agile methodology for team organization.

You will want to track bugs, enhancement requests, and other types of work.
The work items can be labeled by their category (bug, enhancement, discussion, etc.), by priority (Urgent, High, etc.), status (Not Started, In Progress, Done) and according to other dimensions. Work items can be given an estimate for how long they will take. Team members should sign up for the items they are working on. All this will give a clear picture of the work being done and the progress toward the overall project goals. You can also track them visually on a board.
It may also be useful to create milestones, and assign particular issues to the milestones. That allows for tracking progress toward a particular goal. Organizing work in this way also has the benefit of making it easier to collaborate since it is clear to all who is working on which items. It can also make for more effective communication among the team because it is possible to reference the work items and/or board during discussions.

Resources:

Good enough practices in scientific computing: Collaboration recommends creating a project overview document and a shared to-do list, and deciding on communication strategies, which "includes the location and technology for email lists, chat channels, voice/video conferencing, documentation, and meeting notes, as well as which of these channels will be public or private."

Best Practices for Scientific Computing: Collaborate says, "Once a team grows beyond a certain size, it becomes difficult to keep track of what needs to be reviewed, or of who's doing what. Teams can avoid a lot of duplicated effort and dropped balls if they use an issue tracking tool... to maintain a list of tasks to be performed and bugs to be fixed. This helps avoid duplicated work and makes it easier for tasks to be transferred to different people."

Research Software Engineering with Python (Irving et. al.): Working in Teams recommends including everyone. "In order to leverage a group’s expertise, a project must do more than allow people to contribute: its leaders must communicate that the project wants contributions, and that newcomers are welcome and valued (Sholler et al. 2019)." It also recommends adopting a code of conduct and points out that this is a good one: https://www.contributor-covenant.org/.

"Key Points:

  • Welcome and nurture community members proactively.
  • Create an explicit Code of Conduct for your project modeled on the Contributor Covenant.
  • Include a license in your project so that it’s clear who can do what with the material.
  • Create issues for bugs, enhancement requests, and discussions.
  • Label issues to identify their purpose.
  • Triage issues regularly and group them into milestones to track progress.
  • Include contribution guidelines in your project that specify its workflow and its expectations of participants.
  • Make rules about governance explicit.
  • Use common-sense rules to make project meetings fair and productive.
  • Manage conflict between participants rather than hoping it will take care of itself."

"Whether we are working by ourselves or with a group of people, the best way to manage this is to use an issue tracking system to keep track of tasks we need to complete or problems we need to fix. Issues are sometimes called tickets, so issue tracking systems are sometimes called ticketing systems. They are also often called bug trackers, but they can be used to manage any kind of work, and are often a convenient way to manage discussions as well."

"[A] project team will usually create a milestone, which is a set of issues and pull requests in a single project repository. ... [M]ilestones can have a due date and display aggregate progress toward completion, so the team can easily see when work is due and how much is left to be done."

This section also discusses agile teams and organizing work into sprints.

Another resource is The Turing Way: Guide for Collaboration

4. Project Organization

Good enough practices in scientific computing: Project organization says, "Organizing the files that make up a project in a logical and consistent directory structure will help you and others keep track of them." It also gives a logical structure for oganizing projects. The resources below also give recommendations for project organization and descriptive files that can be populated for each project. There are templates that can be used for a new project. For example: Project Template.

More resources:

Research Software Engineering with Python (Irving et. al.): Project Structure gives a structure for organizing projects and "a few standard files that should be present in every research software project."

Research Software Engineering with Python (Irving et. al.): Key Points

"Key Points

Make tidiness a habit, rather than cleaning up your project files later. Include a few standard files in all your projects, such as README, LICENSE, CONTRIBUTING, CONDUCT and CITATION. Put runnable code in a bin/ directory. Put raw/original data in a data/ directory and never modify it. Put results in a results/ directory. This includes cleaned-up data and figures (i.e., everything created using what’s in bin and data). Put documentation and manuscripts in a docs/ directory."

5. Code Quality

The practices outlined here are designed to increase understandability of your code and to avoid common pitfalls that might lead to bugs or make bugs harder to find.

Using a consistent, clear style will make your code easier to understand, debug, and maintain. Here are some respected style guides:

Resources:

Good enough practices in scientific computing: Software recommends the following set of practices:

  • "Place a brief explanatory comment at the start of every program
  • "Decompose programs into functions... that are no more than 1 page (about 60 lines) long
  • "Be ruthless about eliminating duplication
  • "Give functions and variables meaningful names..., both to document their purpose and to make the program easier to read
  • "Make dependencies and requirements explicit
  • "Do not comment and uncomment sections of code to control a program's behavior
  • "Provide a simple example or test data set
  • "Submit code to a reputable DOI-issuing repository"

Best Practices for Scientific Computing: Write Programs for People, Not Computers

  • "[A] program should not require its readers to hold more than a handful of facts in memory at once
  • "[M]ake names consistent, distinctive, and meaningful
  • "[M]ake code style and formatting consistent"

Best Practices for Scientific Computing: Don't Repeat Yourself (or Others)

  • "[M]odularize code rather than copying and pasting
  • "[R]e-use code instead of rewriting it... Tens of millions of lines of high-quality open source software are freely available on the web, and at least as much is available commercially. It is typically better to find an established library or package that solves a problem than to attempt to write one's own routines for well established problems (e.g., numerical integration, matrix inversions, etc.)."

Best Practices for Scientific Computing: Plan for Mistakes recommends using assertions, automated testing, and symbolic debuggers.

Research Software Engineering with Python (Irving et. al.): Writing Readable Code gives Python style based on PEP-8 and Code Smells and Feels

The Turing Way: Code Quality recommends using static analysis tools and automatic code formatting, which are advanced ways to enhance code quality.

6. Documentation

Good documentation makes code more understandable. This helps to ensure that others can take over the code when the people who wrote it or worked on it have moved on. It also makes it much easier to remember how the code works if you have been away from it for some time.

A README file should be written. It should include:

  • The purpose of the code
  • Technical details that play an important role in how the code works
  • How to set up your environment and work on the code as a developer
  • How to run the program or use the code, including setting up dependencies Make a README gives detailed information about what to include in a README, as well as templates that can be used.

More resources:

Best Practices for Scientific Computing: Write Programs for People, Not Computers has these useful recommendations:

  • "[B]reak programs up into easily understood functions, each of which conducts a single, easily understood, task. This serves to make each piece of the program easier to understand..."
  • "[M]ake names consistent, distinctive, and meaningful"
  • "[M]ake code style and formatting consistent" (Also see the "Code Quality" section of this guide.)

From Best Practices for Scientific Computing: Document Design and Purpose, Not Mechanics:

  • "[D]ocument interfaces and reasons, not implementations... [A] clear description ... at the beginning of a function that describes what it does and its inputs and outputs is useful."
  • "[E]mbed the documentation for a piece of software in that software... Doing this increases the probability that when programmers change the code, they will update the documentation at the same time."

Research Software Engineering with Python (Irving et. al.): Documenting Programs recommends documenting:

  • "The name and purpose of every public class, function, and constant in our code.
  • "The name, purpose, and default value (if any) of every parameter to every function.
  • "Any side effects the function has.
  • "The type of value returned by every function.
  • "What exceptions those functions can raise and when. "The word 'public' in the first rule is important. We don’t have to write full documentation for helper functions that are only used inside our package and aren’t meant to be called by users, but these should still have at least a comment explaining their purpose."

The Turing Way: Code documentation discusses different types of documentation, what should be included, and other topics.

7. Licensing & Citation

Resources: Good enough practices in scientific computing: Collaboration recommends that you "[m]ake the license explicit... Have a LICENSE file in the project's home directory that clearly states what license(s) apply to the project's software, data, and manuscripts." This resource has recommendations for specific licenses to use for different types of information.

The Turing Way: Software Citation with CITATION.cff "The Citation File Format lets you provide citation metadata, for software or datasets, in plaintext files that are easy to read by both humans and machines." This resource gives details about how to create the citation file.

8. Keeping Track of Changes

The Turing Way: Version Control nicely summarizes what version control is and why it is useful: "Version control is an approach to record changes made in a file or set of files over time so that you and your collaborators can track their history, review any changes, and revert or go back to earlier versions. Management of changes or revisions to any types of information made in a file or project is called versioning."

More resources:

Good enough practices in scientific computing: Keeping track of changes recommends using a version control system such as Git, Mercurial, or Subversion, if possible. But they say it is also possible to keep track of changes by using a systematic manual approach. They go into the basics of using version control systems.

Best Practices for Scientific Computing: Make Incremental Changes "The standard solution in both industry and open source is to use a version control system... A VCS stores snapshots of a project's files in a repository (or a set of repositories). Programmers can modify their working copy of the project at will, then commit changes to the repository when they are satisfied with the results to share them with colleagues."

"Reproducibility is maximized when scientists put everything that has been created manually in version control, including programs, original field observations, and the source files for papers. Automated output and intermediate files can be regenerated as needed. "

9. Testing

Writing and regularly running good tests helps to ensure that the code is doing what you expect it to do, both initially and as you continue to add to and make changes to the code. Tests can be run manually or as part of an automated framework.

Testing Guidelines

  • Write system tests for important paths through the system
  • Document the tests and how to run them.
  • Run the tests regularly.
  • If possible, automate the testing
  • If possible, write and run unit tests, which are covered in Section 11 below.

Resources:

The Turing Way: Code Testing The authors talk about why testing is important: "It is very, very easy to make mistakes when coding. A single misplaced character can cause a program’s output to be entirely wrong. One of the examples [of real-world problems due to faulty software] was caused by a plus sign which should have been a minus. Another was caused by one piece of code working in meters while a piece of code written by another researcher worked in feet. Everyone makes mistakes, and in research the results can be catastrophic. Careers can be damaged/ended, vast sums of research funds can be wasted, and valuable time may be lost to exploring incorrect avenues. This is why tests are vital."

They also discuss the different types of software testing.

10. Code Review [Advanced Topic]

This is listed as an advanced topic because small teams of researchers may not have the time to do code reviews, but benefits and best practices can be found in the resources listed here for those who are interested in learning about this topic and possibly incorporating code reviews into their work.

Best Practices for Scientific Computing: Collaborate notes that "a large body of research has shown that code reviews are the most cost-effective way of finding bugs in code. They are also a good way to spread knowledge and good practices around a team. In projects with shifting membership, such as most academic labs, code reviews help ensure that critical knowledge isn't lost when a student or postdoc leaves the lab."

More resources:

11. Unit Testing and Continuous Integration [Advanced Topic]

This is another advanced topic that can lead to a significant increase in the reliability of your code and is especially helpful in ensuring that code continues to function as expected as changes and additions are made.

From Good enough practices in scientific computing: Unit tests: "A unit test is a small test of 1 particular feature of a piece of software. Projects rely on unit tests to prevent regression, i.e., to ensure that a change to 1 part of the software doesn't break other parts."

More resources:

12. Automation & Reproducibility [Advanced Topic]

Automation allows code to be compiled and/or results to be generated when code or input data changes. There is upfront work to set up automation, but it can save time and increase reproducibility in the long run. It may also make it easier to share your work with others since they can make changes and generate results on their own machines.

Resources:

13. Reproducibility & Containers [Advanced Topic]

  • Use containers (e.g., Docker) for consistent environments.
  • Archive releases and workflows for reproducibility.

14. Example Projects

15. Training & Resources

References

  • (https://doi.org/10.1371/journal.pcbi.1005510) Wilson G, Bryan J, Cranston K, Kitzes J, Nederbragt L, Teal TK (2017) Good enough practices in scientific computing. PLoS Comput Biol 13(6): e1005510.
  • (https://doi.org/10.1371/journal.pbio.1001745) Wilson G, Aruliah DA, Brown CT, Chue Hong NP, Davis M, Guy RT, et al. (2014) Best Practices for Scientific Computing. PLoS Biol 12(1): e1001745.
  • (https://carpentries-lab.github.io/good-enough-practices/06-track_changes.html) Carpentries Lab: Good Enough Practices
  • (https://third-bit.com/py-rse/index.html) Irving, et. al., Research Software Engineering with Python
  • (https://book.the-turing-way.org/) The Turing Way
  • (https://alan-turing-institute.github.io/rse-course/html/index.html) Research Software Engineering with Python