Reproducibility is fundamental to scientific research. If you develop a new algorithm and use simulations/experiments to claim its superiority over prior algorithms, your claims are only credible if other researchers can reproduce and confirm them.
The first step towards reproducibility is to describe the simulation procedure in such detail that another researcher can repeat the simulation, but a major effort is typically needed to reimplement everything. The second step is to make the simulation code publicly available, so that any scientist can review it and easily reproduce the results. While the first step is mandatory for publishing a scientific study, there is a movement towards open science that would make also the second step a common practice.
I understand that some researchers are skeptical towards sharing their simulation code, in fear of losing their competitive advantage towards other research groups. My personal principle is to not share any code until the research study is finished and the results have been accepted for publication in a full-length journal. After that, I think that the society benefits the most if other researcher can focus on improving my and others’ research, instead of spending excessive amount of time on reimplementing known algorithms. I also believe that the primary competitive advantage in research is the know-how and technical insights, while the simulation code is of secondary importance.
On my GitHub page, I have published Matlab code packages that reproduces the simulation results in one book, one book chapter, and more than 15 peer-reviewed articles. Most of these publications are related to MIMO or Massive MIMO. I see many benefits from doing this:
1) It increases the credibility of my research group’s work;
2) I write better code when I know that other people will read it;
3) Other researchers can dedicate their time into developing new improved algorithms and compare them with my baseline implementations;
4) Young scientists may learn how to implement a basic simulation environment by reading the code.
I hope that other Massive MIMO researchers will also make their simulation code publicly available. Maybe you have already done that? In that case, please feel free to write a comment to this post with a link to your code.
That’s a praiseworthy decision, thanks for sharing your work!
I remember reading about initiatives that provide a toolchain that makes it possible to automate the entire simulation workflow, a bit like what npm-based project do.
A commendable practice – we should all push for this (insofar the research contracts permit). Actually I have long argued that every paper should come with a Makefile that compiles all relevant programs from source, re-creates all the simulation results, generates all figures and compiles the paper itself.
When will we see the availability of open datasets (e.g. channel or traffic measurements) in wireless communications? In other fields, there are several datasets (SNAP in network science, and lots of datasets, Netflix, Yelp challenges, and more in machine learning) publicly available under generous license conditions. Several of them in fact seem to be of commercial origin. Publicly available datasets could offer great value to the community, and potentially large impact for the data owners.
Absolutely great. I wish all wireless communication researchers, specially the popular ones in the field, should make their implementation codes public as a means of reproduce able research.
It is very generous to share extra research materials like simulation codes and data sets and etc with others. Young researchers specifically know how helpful this could be. I myself am very grateful to all the assistance that I have received from Prof. Bjornson with provided matlab codes in his Github and also his patience in replying E-Mails which is very generous. I wanted to represent my gratitude here.
Thanks a lot.
This is my Github (https://github.com/smajidian), although it’s on signal processing not MIMO!
That’s a step forward in communication and signal processing community which make a huge improvement. Most of the time of grad students are spent in coding other works for benchmarking.
In bioinformatic community, the reviewers and editors oblige the authors to provide their data and code in some repository like Github in order to make the paper reproducible by independent researchers. There are some issues and differences :
1. In contrast to your community which MATLAB is dominant, in bioinformatics, there are various language programming (Python, C, Perl, Rust, GO, ..). Therefore running other codes are very time consuming.
2. Bash scripting is so popular which make it easy to run others code. This also mentioned by Dr. Larsson. Some researcher provide Docker as a container of all prerequisites that is needed.
3. A problem which arises there sometimes is that the codes are based on some library/packages namely, samtools,GATK, .. which are updated yearly may not back-compatible.
4. Some links in papers are not working. After graduating students or moving postdocs their homepage/ftps are removed! So it’s better to provide data in permanent repositories rather than personal ones.
I’m also trying to read your ideas in “Reproducible Research: Best Practices and Potential Misuse”. But in contrast to its abstract which emphasis in open access publication, it is not open access itself. It will be appreciated if you provide it here in your blog or arXiv.
Thanks for the input!
My paper “Reproducible Research: Best Practices and Potential Misuse” is now available on arXiv: https://arxiv.org/pdf/1905.00645.pdf
Researchers can you help me? I want the code for Optimization of Spectral Efficiency in Massive-MIMO TDD Systems with Linear Precoding. I want as a reference?
My book “Massive MIMO networks” is delivered with simulation code that covers all these things. You can find all of it on http://massivemimobook.com