Class Journal Week 8

Mahrad Saeedi

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • Baggerly discusses numerous issues regarding the data and analysis they looked over. There were many discrepancies in the data that was analyzed. Formatting was inconsistent and resulted in the gene data being offset by one, which ultimately offset the respective p values. Gene labels and IDs were mixed up and the data wasn't accessible through every application.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • His recommendations for reproducible research are in line with DataONE on the notions of consistent formatting, compatibility in file types, and, in general, familiarizing oneself with the database and how it operates.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • From Dr. Baggerly's talk it is made apparent that this type of data manipulation can occur around us, even by the most respected and prestigious members in the scientific community. It is thus important that these data and research be made public, not only to enable trust between individuals conducting research, but to also allow for collaboration and further understanding of that specific experiment and it's resulting data.
4. Go back to the Merrell et al. (2002) paper and look at your "sanity check" where you compared the fold changes and p values for certain genes between your work and the paper. Did the values match? Why do you think that is? Do you think there is sufficient information there for you to reproduce their data analysis? Why or why not?
   • The values are not all exactly the same between my work and the paper. This is due to the fact that we conducted our reformatting and analysis of the data using GenMAPP and microsoft excel, while the researchers used a software known as SAM. I am unsure if I would be able to reproduce their data analysis because I am unfamiliar with the SAM software. It could be done using GenMAPP and Excel, but it would definitely take some time.

Msaeedi23 (talk) 23:38, 26 October 2015 (PDT)

Jake Woodlee

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • From what I saw, Baggerly and Coombs identified a lack of transparency in procedure, improper labeling, and an eventual discrepancy in the statistical analysis of the paper. The best practice violated in DataONE were not labeling data properly and an inadequate procedure section. Baggerly said it was a common problem for the statistical analysis to be inadequately written out in a step by step format with incomplete documentation. And while the review board mentioned this they still failed to act.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • He recommends the origin of the paper, the raw data included, any code used to process the data, descriptions of non scriptable steps, and descriptions of planned design are all necessary for clinical trials and should be included in a reproducible scientific paper. He notes that if there is a more standard procedure reproduction would be easier. In the DataONE powerpoint there is mention of consistent data labels and formats that would contribute to reproducible research.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • I am glad that he was so persistent on pushing his review of the paper through all the criticism. I also think it is a good thing he is such a proponent for reproducible research because it is the only think science can really rely on to verify a claim. Also, I thought it was interesting how he mentioned that more complex technologies call for more complex research reproduction tools.
4. Go back to the Merrell et al. (2002) paper and look at your "sanity check" where you compared the fold changes and p values for certain genes between your work and the paper. Did the values match? Why do you think that is? Do you think there is sufficient information there for you to reproduce their data analysis? Why or why not?
   • The values are not exactly the same. This is definitely due to how we processed the data and performed our analysis. They used the program "SAM" while we used GenMAPP and to some extent, Excel. These differences in procedure caused the difference in conclusion, a further and deeper analysis of our procedures will get to the bottom of this. I am not confident I could reproduce their data completely. I might be able to completely figure it out given enough time, I'm just not sure how I would react to the software they used.

Jwoodlee (talk) 15:35, 22 October 2015 (PDT)

Emily Simso

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • The most common issues are that, first, documentation is often poor in clinical trials, thus not explaining what researchers did in their work. There are also then problems with intuition, since people assume things about the data. Overall, Baggerly and Coombs stress that lack of thoroughness with data and research leads to complications further on.
   • The violated best practices according to DataONE were: consistency, being descriptive, and lacking data or information.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • Dr. Baggerly recommends that groups use the same standards for reports, templates are reused, there is a report structure for approval, and use executive summaries for complete documentation.
   • This connects to DataONE because they stress reproducible research through appropriate file types, consistent formatting, clear definitions, and understanding how databases are set up before using them, amongst other points.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • I think watching Dr. Baggerly's talk helped explain how something like the Duke case could happen, because there are so many details in data and research. It seems that mistakes could easily be covered up simply because there is a culture of not documenting every aspect of your work. I think that this needs to change so that future clinical trials are more closely regulated.
4. Go back to the Merrell et al. (2002) paper and look at your "sanity check" where you compared the fold changes and p values for certain genes between your work and the paper. Did the values match? Why do you think that is? Do you think there is sufficient information there for you to reproduce their data analysis? Why or why not?
   • The values did not match between the Merrell et al. analysis and my analysis. This is probably because they are much more trained in this field and have more resources at their disposal. They were also able to perform more tests on the data, whereas we only had the given values. While I think we are able to get fairly accurate results, we are not close enough to the experiment to get the exact same results.

Emilysimso (talk) 20:49, 25 October 2015 (PDT)

Veronica Pacheco

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • Alot of the data being analyzed had discrepancies. They noticed problems in the data and they predict that they switched their input giving a much different resistance number than the actual resistance number. They figured this out by using a different study.They also made the point that there other genes that wasn't making sense in their data. 14/19 genes were accounted for by cross referencing them with another paper but that still leaves the other 5 genes. At this point, the contacted the magazine to report these findings. They said the most common mistakes are simple. Some of these mistakes include experimental design, mixing up sample labels, gene labels and group labels. DataOne explains that organization, consistency, and description is key to practicing good data preservation skills.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • The ask for labeling the columns to tell which sample is which and DataOne emphasizes this point as well. They also ask to provide the code so that it is clear when trying to reproduce the results in a given experiment.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • I like how at the end, he tells the audience what Coombs and him do as part of their protocol.For example, the use literate programming like Sweave. Overall, I liked how this assignment was structured. Reading the case first, then hearing this talk on how they went about figuring out the issues was a neat experience.
4. Go back to the Merrell et al. (2002) paper and look at your "sanity check" where you compared the fold changes and p values for certain genes between your work and the paper. Did the values match? Why do you think that is? Do you think there is sufficient information there for you to reproduce their data analysis? Why or why not?
   • The values did not match. I think it has to do with the accuracy of their method.They were able to use SAM which is Statistical Analysis for Microarrays and it is probably more advanced and accurate than using the pvalues in Excel. Although Excel is a great tool.

Vpachec3 (talk) 21:59, 25 October 2015 (PDT)

Kevin Wyllie

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • Baggerly points out confounding in experimental design (mixing up interpretation of results: sensitive versus resistant), mixing up sample/gene/group labels, and incomplete documentation (which hides the previously-mentioned mistakes). DataONE mentions that data should be stored in a format which allows it to be used by any application. This was violated when the researchers added a column name to the gene ID’s, which tricked their code into offsetting each gene by one.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • Baggerly recommends appropriately labeling data, and providing code (provenance) so that it can be tested by third parties. Baggerly thinks these things should be required before beginning clinical trials. DataONE recommends using “descriptive column names” as well as dataset provenance.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • Dr. Baggerly speaks quite fast and uses some terminology that is unfamiliar to me, so after watching the video twice, I’m still not entirely sure which “errors” he is implying Dr. Potti committed intentionally (if any). To me, this contrasts with the 60 Minutes segment which seemed to mention solely the deliberate manipulation of the data. The off-by-one index error, for example, seems like it could have been an honest mistake (not to say this would relieve Potti of culpability), as I can’t imagine how that would actually add to the (false) significance of the results.
4. Go back to the Merrell et al. (2002) paper and look at your "sanity check" where you compared the fold changes and p values for certain genes between your work and the paper. Did the values match? Why do you think that is? Do you think there is sufficient information there for you to reproduce their data analysis? Why or why not?
   • The values did not match, likely because different statistical methods were used. After reading the paper, I can't say I think the authors provided sufficient information to carry out an identical analysis. In fact, the paper itself mentions very little statistical analysis at all. Baggerly and DataONE would not approve.

Brandon Klein

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • Baggerly and Coombes identified a concerning amount of issues with the data and analysis present in Dr. Potti's paper. Much of the data used for the analysis had consistent off-by-one index errors due to inappropriately omitting column headings, which in turn mismatched data and offset p-values. Further, some data sets appeared in replicate and were inconsistent with themselves, while others were integrated to explain biology but not used in the actual analysis. If this was not concerning enough, some critical labels such as resistant and sensitive were swapped in the data analysis. When challenged, the team at Duke even released a validation data set that contained entirely incorrect and misleading samples.
   • The issues that Dr. Baggerly addressed in his lecture violated various best practices enumerated by DataONE. The Duke team did not maintain dataset provenance (data transformations and duplications were common), ensure data compatibility (off-by-one index error), or use reproducible workflows (many steps were poorly documented or omitted). In addition to this, there were issues with the consistency of the data (e.g. switched labels) and insufficient documentation, particularly when the Duke team was attempting to validate their research in the face of concerns (much of the validation work was incorrect and not released).
   • Dr. Baggerly claimed issues such as off-by-one index errors, confounding experimental design, mixing up labels, and incomplete documentation are unfortunately more common than we would like to admit.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • Dr. Baggerly introduced a series of recommendations for producing reproducible research that he and Coombs had been using to the audience. To begin, he suggested that all papers should include data, code, descriptions of nonscriptable steps, descriptions of planned design if used, and should maintain the provenance of data. This both largely automates and elucidates the data analysis methods, enabling it to be reproducible. Further, steps such as including literature programming (Dr. Baggerly mentioned using Sweave so that anyone could run his code through R and get the same results), reusing templates, reporting structure, providing summaries, and including appendices are all ways to further ensure that your research is reproducible. These suggestions are consistent with the DataONE best practices, such as mantaining dataset provenance, documenting manipulations/assumptions, and particularly using reproducible workflows (DataONE explicitly suggests automating the integration process as much as possible, which Dr. Baggerly stressed).
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • Although I had initially been appalled by the 60 Minutes exposé on this case, watching Dr. Baggerly's lecture still managed to floor me. I had not realized how dramatic and blatant some of the data manipulations were. The fact that entirely incorrect datasets were used, treatment groups were inverted, and data was even replicated/mislabeled to generate a specific result demonstrates a clear intent to fake results. And this was no minor faking of data either; the Duke team had to actively and aggressively make these alterations to the data. This kind of behavior, particularly in the realm of medicine and clinical trials, is extremely concerning to me. Further, I had not realized how difficult it was for Baggelry and Coombes to draw attention to the manipulations made by the Duke team. There should be a system in place to make it easier to expose research fraud. Research publications should be focused on simply presenting discoveries as opposed to finding and supporting sensational stories while turning a blind eye to potential manipulation.
4. Go back to the Merrell et al. (2002) paper and look at your "sanity check" where you compared the fold changes and p values for certain genes between your work and the paper. Did the values match? Why do you think that is? Do you think there is sufficient information there for you to reproduce their data analysis? Why or why not?
   • The "sanity check" that I performed comparing my microarray analysis to the results published by Merrell et al. (2002) did not entirely match up. Slightly less than one-third of the genes they reported as having exhibited significant changes in expression did not exhibit statistically significant expression changes in my analysis. However, this likely occurred because I used a different method to determine significance than they did, assessing p-values instead of running a two-class Statistical Analysis for Microarrays (SAM) analysis. Because the criteria used for judging significance was different, it makes why not all of my results matched theirs. Although the paper is slightly vague when discussing the specific statistical methods they used, I believe they included enough information regarding their methods and software tools to reproduce their results (although minor forensic bioinformatics may be necessary to bridge slight gaps in the documentation).

-- Bklein7 (talk) 18:22, 26 October 2015 (PDT)

Josh Kuroda

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • One of the main issues that Baggerly and Coombs brought up had do to with proper documentation, since the absence of this would require "forensic bioinformatics" so that one may infer what was done to obtain the results. One recurring practice that was violated was the practice to correctly label data points. In one case, 43 samples were mislabled, and 16 samples were not matched at all. Complete documentation is also a practice that was violated, and Baggerly claims that both incomplete documentation and labeling issues are the most common. Mixing up sample, gene and group labels are the most simple mistakes, but happen too often.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • Dr. Baggerly recommends literate programming, reusing templates, report structure, executive summaries, and appendices as elements to the bigger picture of reproducible research. These mostly overlap with what DataONE recommends, since DataONE says that consistently formatted data, documentation of assumptions, reproducible workflows, and compatibility ensuring is vital to good data management.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • In my opinion, many of his points are valid, but I feel like the Duke case is more of a lesson for bioethics than reinforcing reproducible research. In my opinion, the Duke case shows why researchers cannot simply wish for success, since the venture for truth and cures is a journey that is littered with failures and lessons learned.
4. Go back to the Merrell et al. (2002) paper and look at your "sanity check" where you compared the fold changes and p values for certain genes between your work and the paper. Did the values match? Why do you think that is? Do you think there is sufficient information there for you to reproduce their data analysis? Why or why not?
   • No, the values did not match. This is probably because of the differences in data analysis techniques between our assignment and their research. I am not completely sure, but I think we would be able to reproduce their results to a certain degree of accuracy, mostly because of the fact that we would most likely be able to fill in any gaps in information or instruction now that we have a good understanding of how to analyze microarray data.

--Jkuroda (talk) 21:37, 26 October 2015 (PDT)

Anu Varshneya

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • Baggerly and Coombs brought attention to several different issues regarding issues with indexing, headings, mismatched data, and a lack of documentation which makes it harder to reveal the issues with the previous steps. DataONE stressed the importance of storing data in a consistent format that is usable in several programs and applications, and being descriptive and explicit with all data. This best practice was clearly violated in Dr. Potti's data by adding an extra column to his data that caused everything to be misread, which prevented the data from providing accurate data.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • Baggerly recommends descriptive labels on data, and providing code or any other testing procedure to ensure that all data can be tested for reproducibility and accuracy. These recommendations are similar to the recommendations made by DataONE which include reproducible research through easily accessed data, consistency in format, and descriptive labels and definitions regarding data and databases used.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • Dr. Baggerly's talk helped clarify how the mistakes made in Dr. Potti's research could have happened, and how they could be prevented in the future. His talk was insightful in regards to proper practices in data analysis, but it doesn't answer questions regarding how intentional Dr. Potti's data discrepancies were.
4. Go back to the Merrell et al. (2002) paper and look at your "sanity check" where you compared the fold changes and p values for certain genes between your work and the paper. Did the values match? Why do you think that is? Do you think there is sufficient information there for you to reproduce their data analysis? Why or why not?
   • Not all of the genes that Merrell et al. (2002) presented were significant in my data. This is most likely due to differences in data analysis tools - we used Excel to compute t-tests and p values while they used a Statistical Analysis of Microarrays (SAM) analysis. Because the SAM analysis analyzed the data and had different requirements for statistical difference, it is understandable that we had different results. Though the paper was pretty vague regarding the exact methods used, I think it is possible to partially reproduce their findings, but only after some guessing regarding the exact steps they too.

-- Anuvarsh (talk) 22:50, 26 October 2015 (PDT)

Mary Alverson

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • Baggerly and Coombs identified many main issues. Some extremely egregious ones were actually simple fixes, but had huge implications. These main issues included having the data be off by one, because of a software/excel sheet formatting error. Another main issue that stemmed from either software or formatting was that many results got switched completely, ie the data points that were supposed to show a low sensitivity showed the opposite: a high sensitivity. Other main issues were the repetition of data points (especially even after the researchers were asked to fix the mistakes) as well as mislabeled/inconsistent labeling of the data. This last issue is one that DataONE also enumerated.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • He recommends that the data tables be labeled accurately so that others can easily know what values correspond to what. He also recommends that lots of descriptions of steps be published, including both nonscriptable steps as well as planned design. This is so that if someone wanted to reproduce the exact results, they would be able to. He also says it is important to make available any code that is used in the data analysis process. These are somewhat covered by what DataOne recommends. I am confident that DataONE would not disagree with what Dr.Baggerly said, and also vice versa. DataOne's best practices just go into much more detail regarding the data entry.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • After this talk, I am definitely more disappointed in the various articles for not taking Baggerly and Coombs complaints seriously. I feel that the journals carry some blame for this debacle for not seeing the errors that were presented before them. I am also seeing why fact checking or sanity checking is important, as well as why it is important to write down every single little thing I do while doing my homework assignments.
4. Go back to the Merrell et al. (2002) paper and look at your "sanity check" where you compared the fold changes and p values for certain genes between your work and the paper. Did the values match? Why do you think that is? Do you think there is sufficient information there for you to reproduce their data analysis? Why or why not?
   • The majority of my tests showed that the genes Merrell et al.(2002) found to be significant were also significant using my analysis, but not every single one. I am not extremely surprised by that, however, because since they explained how they came to their results it was clear to me that I simply just used a different means to get to the result. Since the majority matched, I was still fairly confident in my analysis techniques. I could try and replicate the analysis in Merrell et al. (2002) but I am not positive that I would be able to successfully because they did not list out what they did step by step, but instead just gave a brief summary on how they came about their findings.

--Malverso (talk) 23:28, 26 October 2015 (PDT)

Trixie Roque

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • Baggerly and Coombs found various issues with the data, which produced major consequences in the long run. Some problems include incorrect labels are data that are slightly off. Additionally, some of the calculations were swapped. DataONE actually pointed out the mislabled data samples. Additionally, documentation of their procedure was not properly created.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • Baggerly recommends that the columns and samples be correctly labeled so there aren't any confusion as to which data corresponds to which. Most of the advice he gave was for reproducing the steps that they originally made. Some of the things he touched upon actually were supposed to be best practices that they should have done in the first place and which DataONE actually also recommends.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • After viewing the talk, I became fairly certain that there needs to be more data regulation. Even though there were some minor mistakes that were made in the experiment, some were just too great that they completely impacted the results. It made me think that these results were really intentional since there were just too many mistakes for anyone to not notice.
4. Go back to the Merrell et al. (2002) paper and look at your "sanity check" where you compared the fold changes and p values for certain genes between your work and the paper. Did the values match? Why do you think that is? Do you think there is sufficient information there for you to reproduce their data analysis? Why or why not?
   • The values that I ended up with did not match up with what the paper presented. I think this is the case since we used a basic tool for calculating our statistical values such as our p values using Microsoft Excel whereas the authors used a more specialized program called " Statistical Analysis of Microarrays" (SAM). I think that the paper does not present enough information for reproducibility since the methods were not elaborated enough so I'm not entirely sure that I could reproduce the results completely.

--- Troque (talk) 23:46, 26 October 2015 (PDT)

Kristin Zebrowski

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • Dr. Baggerly claims that the main issues are a lack of good documentation, incorrect or switched sample/gene labels or duplicates, column and row transpositions and offsets such as the off-by-one indexing error in the example, and even complete misinterpretations or mix-ups of the results. Drr. Baggerly said that the simplest errors were the most commonly made ones and that they are easy to fix but only if documentation makes them easy to find. DataONE emphasizes descriptive column names without spaces or special characters, which was clearly not practiced by the example dataset. Furthermore, data needs to be stored in a consistent file type that renders it reproducible in the future, which the data and analysis that Dr. Baggerly and Dr. Coombs identified did not fulfill.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • To make research reproducible, Dr. Baggerly recommends labeling columns to indicate samples and to provide code. He recommends using Sweave so that someone else can run the program and get the same numbers, whereas switching applications could potentially mess up the data presentation. Similarly, DataONE recommends reproducible workflows that are compatible. DataONE also recommends providing provenance or code to automate the integration.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • After viewing Dr. Baggerly’s talk I was just further convinced that data regulation needs to be more enforced and upheld, but it made me positive about the way things are going. Dr. Baggerly and his team are clearly very committed to passing more stringent data checking measures. I was a bit shocked, however, to hear about how often Dr. Baggerly sees errors that are so simple and yet change the data so drastically—the manipulation was immediately clear by looking at some of the datasets he showed in his presentation. Things like that seem so easy to fix or to catch and yet they still happen all the time. I don’t understand why adequate documentation that enables research to be reproducible isn’t an absolute requirement but I’m glad that people like Dr. Baggerly are trying to change that.
4. Go back to the Merrell et al. (2002) paper and look at your "sanity check" where you compared the fold changes and p values for certain genes between your work and the paper. Did the values match? Why do you think that is? Do you think there is sufficient information there for you to reproduce their data analysis? Why or why not?
   • No, the values didn’t quite match. The Merrell et al. (2002) analysis used SAM and measured significance in terms of a twofold change in gene expression while we used a less-stringent t-test to calculate p values. A difference in statistical methods could definitely result in a discrepancy between their values and ours. I didn’t see sufficient information to reproduce the analysis in the paper personally. I think that there is an adequate description of what they did to determine statistical significance but not necessarily enough information on how they did it.

Kzebrows (talk) 23:33, 26 October 2015 (PDT)

Brandon Litvak

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • Baggerly and Coombs found that the data contained an off-by-one index error (which often put sets of genes together with completely unrelated data). It was also found that the data contained samples that were not distinct (multiple cases of sample reuse that indicate incorrect labels). Other data and figures also contained mislabeled samples or samples that did not match with any labels. The other major issue that was found with the data was the presence of poor documentation that made it very unclear as to how certain results were obtained (forensic bioinformatics had to be utilized in order to figure out some details about what was done with the data). Dr. Baggerly claimed that the common issues included confounding in the experimental design (which is related to the many inconsistencies/mix ups with the data) and the consistent presence of poor documentation.
   • With respect to the best practices, as illustrated by DataONE, the data failed to be stored in a format that is capable of being accurately read by any current/future applications (data contained an extra header/row that resulted in an off-by-one index error, which resulted in an incorrect output with the program utilized in the research). The use of a reproducible workflow was also absent since the data had very unclear documentation. The maintenance of data set provenance (data about the origin/history of the data) was also avoided. Additionally, the data had columns of data that were not consistent (data was often mislabeled, mismatched, or was frequently missing labels).
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • Baggerly recommends, for papers, the appropriate labeling of columns (which make the samples clear, as well as the nature of the data), the exact code that was used in the analysis/work, the provenance of the work (metadata that relates the start and history of a dataset), and proper documentation that includes detailed descriptions of steps that are not scriptable and any descriptions of the "planned design". He believes that these recommendations should be, ideally, non-negotiable requirements prior to the start of clinical trials.
   • Dr. Baggerly stresses the importance of reproducible research and he chiefly suggests consistent data labels, provenance, and the use of summaries/proper documentation. His recommendations regarding data labels directly corresponds to the dataONE best practices that involve the use of descriptive column/data labels (which make it explicit and clear as to what the data actually represents). His recommendations regarding the use of good documentation and summaries relate fairly well to the dataONE recommendation of using workflows that are reproducible (which should allow another party to understand what was done, along with allowing them to evaluate the process, as a whole). Baggerly's suggestion of using metadata to inform others about how the was dataset changed or manipulated (provenance) also directly relates to the DataONE suggestion of maintaining dataset provenance that takes into account the various transformations that a document/dataset underwent.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • After viewing Dr. Baggerly's talk, I feel much more surprised that this study actually led to two sets of clinical trials (and that it took so long for the authoritative organizations to realize the truth regarding the data). It seems very explicit and clear that the data was deeply flawed, and it feels like Baggerly and Coombs did a good job at communicating their findings to Nature and to those involved with the research; it feels pretty absurd that there was even initial resistance, slight doubt, and an attempt to cover-up faulty data in response to the points made by Baggerly and Coombs.
4. Go back to the Merrell et al. (2002) paper and look at your "sanity check" where you compared the fold changes and p values for certain genes between your work and the paper. Did the values match? Why do you think that is? Do you think there is sufficient information there for you to reproduce their data analysis? Why or why not?
   • The values did not entirely match up (many genes that were shown to have significant changes in expression, in the paper, were found to not have any significant changes in the work related to the assignment). The differences between the paper and the work done for this week could be the result of differences in the methods related to the statistical analysis. Merrell et al. utilized a two-class SAM analysis while we utilized Excel and the calculation of the p-values of the log fold changes that reflected all of the samples. Looking at just the information in the paper, it is not exactly clear as to how somebody would reproduce their results; the methods and the rationale behind the statistics is not very clearly touched upon. However, even if the article is fairly light on some of the statistics, I think that through some additional independent research (and data hunting, since the article contains dead links) one would be able to get a result that is similar to what was found in the paper. It seems that this article is chiefly written with professionals in the scientific community (specifically, those involved with genes/microarrays) as the intended audience, who are likely better versed in the statistics and the programs related to this kind of research.

Blitvak (talk) 02:06, 27 October 2015 (PDT)

Lena Olufson

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • Baggerla and Coombs were able to identify many issues relating to the analyzation of the data. The formatting of the data was conflicting and thus caused a major offset in the data, which caused the genes to be offset by one. This off by one indexing error resulted in the p values being off as well. Furthermore, the been labels and the IDs were mixed up due to this offset, leading to the inaccessibility of the data through each application.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • Dr. Baggerly recommends some very similar things as DataONE does regarding reproducible research. These suggestions consist of first getting yourself comfortable with the layout of the database and how it operates, then using uniform formatting throughout the database, and finally using file types that are compatible with one another and can be easily accessed by all users.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • It is still amazing to me that prestigious members of the scientific community are trying to create and trick people with false data analyses. I do not understand why they would want to continue generating false conclusions and claims based on their false data, since in this particular field the false data can be detected. It is important that scientists continue to check each others' work for validity before data sets are accepted in order to prevent the leaking of false information, as well as to protect the general public who are not knowledgeable in the particular field of data manipulation.
4. Go back to the Merrell et al. (2002) paper and look at your "sanity check" where you compared the fold changes and p values for certain genes between your work and the paper. Did the values match? Why do you think that is? Do you think there is sufficient information there for you to reproduce their data analysis? Why or why not?
   • After looking at my values in comparison to the Merrell et al. paper, I concluded that the values between the two are not exactly the same as one another. The researchers used a different software, called SAM, than we did in class as we used GenMAPP and Microsoft Excel to edit the formatting of the data as well as to create an analysis. I do not think that I would be able to personally reproduce the data analysis that the researchers did because I have never heard of or been exposed to the SAM software that they used. Perhaps if I was trained or given time to try and explore the SAM software, I would be able to try and follow a protocol to reproduce the data that the researchers did, but I am not confident that I would be able to do it on my own.

Lenaolufson (talk) 10:16, 27 October 2015 (PDT)

Nicole Anguiano

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • The main issues with the data and analysis were mislabeling, gene duplication, and off by one errors. The data violated the best practices of maintain dataset provenance, review metadata of context, methods, and meaning, and use reproducible workflows. Dr. Baggerly claimed that the simplest issues were most common, particularly the issues of mixing up labels and incomplete documentation.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • Dr. Baggerly recommends that scientists provide data and code, have a system of labeling that is clear and consistent, keep a thorough description of steps that are not performed in code, and to have descriptions of planned designs. These correspond well to what DataONE recommends.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • After watching the talk, I'm still dismayed that this case managed to get so far before trials were shut down. This talk highlighted how many times Baggerly and Coombs tried to get answers for why the data was incorrect, and how many times they were blown off or ignored. It also is slightly worrying to see that the recommendations for reproducible research are only recommendations and not industry standard following a mistake of that scale. It worries me that is possible for something like this to happen again, but I am glad to see that there are those who will fight for reproducible research to become the standard.
4. Go back to the Merrell et al. (2002) paper and look at your "sanity check" where you compared the fold changes and p values for certain genes between your work and the paper. Did the values match? Why do you think that is? Do you think there is sufficient information there for you to reproduce their data analysis? Why or why not?
   • The values do not match exactly, likely because the software used was different. If we both had used the exact same software, it is likely that we would have obtained the same results. With the information given in the paper, I'm not fully sure I'd be able to reproduce their data analysis even if I had the software. They don't fully explain what they did or why they did it, which would make it difficult to perform their experiment again.

Nanguiano (talk) 16:32, 16 September 2016 (PDT)