education, training interns

     * •

Explicit follow-up at WSSSPE5 on all the activities that were created at
WSSPE4. These should be at least lightning talks, perhaps with their own
session.

     * •

financial models for sustaining software

     * •

Frameworks for building software

     * •

have a short beginning presentation for each working group and a short ending
presentation

     * •

How software development training can be adapted to different disciplines
beyond STEM

     * •

How to low the barrier for non-tech heavy researchers to follow best software
practices.

     * •

I would like to have more discussions about research, if people can share
their research or some of the interesting projects that they’re working on,
this will help in knowing the problems in sustaining software.

     * •

I would like to learn more about sustainability issues in industry.

     * •

I would like to see more grounded advice and less blue-sky thinking. A lot of
stuff in the talks was ‘this would be nice’ but there were not enough talks
that were directly applicable to day-to-day programming.

     * •

Like to see more examples from the real world.

     * •

Mechanisms of improving software reusability

     * •

More concrete projects, concrete possibilities and showcases for collaboration
with results

     * •

Practical ways of achieving some of the goals rather than the discussion of
more attempts on the writing of white papers

     * •

Reproducibility through containerization (I guess there is another list of
best practices needed there)

     * •

RSE award, organizational questions regarding setting up RSE teams within
universities

     * •

Software testing

     * •

Some theory on what makes for sustainable software, which is not the same as
best practice for research software engineering. If this had been talked about
before then consider that 80% of us were new.

     * •

Specific scientific software sustainability factors

     * •

Testing methods that work. Ways of promoting organizational change.

     * •

The progress of our current project.

     * •

The use of scientific software at different stages of research / work

     * •

We discussed scientific software. Should we also discuss software FOR
scientists (like Dan’s google scholar replacement)?

  4. (4)

What could be improved?

     * •

A bit less fly by night scheduling though things worked out just fine.

     * •

A lot of the speakers just seemed to be there either to plug their own
projects (e.g. Hydroshare, BioDynaMo) or spent the majority of their talk
waffling (e.g. Manifesto for Personal Responsibility in the Engineering of
Academic Software). Also, the icebreakers were not very useful.

     * •

A single room for the duration of the meeting. (Particular to this WSSSPE4,
the “collaboratory” room with the monitors, was actually perfect for the full
meeting.)

     * •

Allotting more time for questions & discussions in between talks

     * •

close to perfect workshop - if I would change one thing that give more time
for discussions

     * •

consistency of participants

     * •

felt locked in to the breakout group, not sure if that was intended or not

     * •

Finding rooms a bit confusing at the meeting outset

     * •

Having more time for the demos.

     * •

Having more time to discuss some of the talks

     * •

I was disappointed in the fruit plate because it only had melon.

     * •

I would like to be able to ask questions after talks. By the time we can
discuss the talks, I’ve forgotten what I wanted to say

     * •

Identify a local pub/restaurant/cafe to which people can head afterwards for
casual meetup for dinner/drinks/discussion/whatever

     * •

If feedback to people’s projects and lighting talks etc is posted online, then
we would have also benefits from those thoughts

     * •

It was a great event overall I feel like if we can bring some real life
project integration with best practices that will be helpful

     * •

it would be great to see some specific case studies/success stories

     * •

Keep the online agenda up to date. Better air-conditioning.

     * •

Less talks

     * •

more time for discussion and working groups

     * •

not sure, it looks great to me. I would like to see in plenary the perspective
from closed software and open source software (for cross-fertilization)

     * •

Some longer talks that do deeper dives. Good that the parallel track idea was
abandoned so we could hear all talks.

     * •

The identification of what RSE and RS are, especially for newcomers

     * •

The keynote address could have been better targeted for this community
beforehand.

     * •

The workshop was great!!!

     * •

There is lots of vision here, so we need to put more effort in our project to
move forward.

     * •

Think about a little more planning on the BOF Dinners to make it easier to get
groups together. Maybe a Google doc to encourage it etc.

     * •

To have a few speakers (2-3) that talk about concrete scientific software,
showing the simulations and talking about their scientific results, and why
they chose the software they did (as compared to other available ones)

     * •

Try and add some extra Q&A time after talks (unless that’d mean less talks)

     * •

We could have additional time to work in working groups

     * •

WSSPE vision prior to start would have helped

  5. (5)

What would make you more likely to invest time in future WSSSPE initiatives?

     * •

A centralized place/organization to see and discuss WSSSPE initiatives. At the
moment, WSSSPE organization is scattered across many places. One website with
a list of activities and contributors would help.

     * •

A tutorial session.

     * •

Being able to contribute to something going forward, rather than a talking
shop for the initiatives.

     * •

connect to people

     * •

For meetings: again the availability of travel awards.

     * •

For myself, I anticipate looking for ways to tie WSSPE initiatives with my
current projects such that the outcome benefits a/the broader community.

     * •

Free lunch and double coffee capacity? Also - seriously - less filler talks
and filler activities.

     * •

Funded support resources

     * •

funding; focus on more concrete aspects of sustainability

     * •

Having more time to invest in future WSSSPE initiatives

     * •

Having more time to invest in future WSSSPE initiatives; I’d definitely do
more if I could.

     * •

Honestly, having the time available in my own schedule (which I don’t think is
something WSSSPE organizers can change)

     * •

I already plan to do it :)

     * •

I’m not sure. I’m not very interested in fostering WSSSPE as an thing, but I
am interested in WSSSPE as a means to network to find people to work on these
issues with me.

     * •

If I can see the value of the initiative to me and my value to the initiative

     * •

if I had funding to specifically work on that

     * •

If my daily workload would be reduced

     * •

if the community is and remains active

     * •

Keeping up your generous travel support alone is enough for me to not even
think about missing any further WSSSPE events

     * •

Know that the work started in one edition can be presented on the following
one.

     * •

Likelihood of useful new contributors.

     * •

more clearly defined groups

     * •

More followups between the conferences.

     * •

More funding opportunities to be able to devote appropriate attention to the
topic

     * •

More senior participants with (i) experience, (ii) influence

     * •

More time. Some funding.

     * •

Potential a

     * •

Prospects for collaborations leading to citable artifacts, e.g. papers

     * •

some documented forward momentum between sessions

     * •

The networking is incredible for this meeting and should be encouraged to
continue. Having published deliverables out of the community will help this as
well.

     * •

The travel support really helps

     * •

This topic is related to a large component of what I envision as my
professional career path. In particular, the project designed during the
sessions will be fleshed out into a demo, and if adequate, into some form of
larger piece of software.

     * •

To establish a metrics for research software.

     * •

Understanding the landscape of funding and potential for publication

     * •

Yes, definitely I met with some very interesting people through this platform
and would like to be a part of it. It’ll be an honor to collaborate to this
initiative.

  6. (6)

Would you be interested in joining a professional society focused around the
topics of WSSSPE?

---

Answer | Count
---|---
Yes | 24
No | 3
Maybe | 18
Total | 45

  7. (7)

Will you be part of a working group with colleagues going forward on a topic
addressed at WSSSPE, and if so, which one?

---

Answer | Count
---|---
Yes | 32
No | 4
Maybe | 9
Total | 45

  8. (8)

Working group name

     * •

“Citations / CodeMeta” and “Software Practices White Paper”

     * •

Best Practices White Paper and Best Practices Undergraduate Course

     * •

CodeMeta

     * •

CodeMeta

     * •

CodeMeta

     * •

CodeMeta, but I also supplied information to other working groups – mostly
connections to others – that I hope will be of assistance.

     * •

Continuous CI for research software on uncommon hardware

     * •

Continuous Integration

     * •

Continuous integration for research software; Testing of scientific software

     * •

Curriculum for an undergraduate course on software best practices for domain
scientists

     * •

Domain Specific Undergraduate Courses on Best Practices for Software
Development

     * •

Domain Specific Undergraduate Courses on Best Practices for Software
Development AND International Collaborations between institutions.

     * •

integration of software testing through Debian

     * •

Meaningful Metrics for Scientific Software

     * •

Meaningful Metrics in Software

     * •

metadata

     * •

multiple

     * •

Open Research Index, Metrics, Social Science, and maybe others

     * •

Prototypes

     * •

research software studies

     * •

Scientific Software Prototyping Infrastructure (S2PI), WG-Best-Practices

     * •

Setting up Software Sustainability Alliance

     * •

Software Sustainability Alliance

     * •

Software Sustainability Alliance; Metrics; White Paper

     * •

sustainability metrics; best practices

     * •

Sustainable Software Alliance

     * •

Sustainable Software Alliance

     * •

Tenure Letters

     * •

Testing in Sciences

     * •

Testing in Scientific Software

     * •

Verify Best Practices and Metrics for Sustainable Software

     * •

Verify Best Practices Case Studies

     * •

White Paper WG, Open Research Index WG

  9. (9)

Do you have any other comments or suggestions or ideas?

     * •

As a student, coming to WSSSPE4 has been very rewarding experience.
Sustainable software is not my particularly sub-field of interest, but it was
very interesting to me to apply the skills I had learned in my research in a
different field. It definitely taught me a different perspective and expanded
my original perception of where my skills can be applied.

     * •

As a student, this has helped me explore the different computational research
careers and has made me more aware of scientific software best practices. I
found the plenary talks particularly valuable for learning about different
areas of study, and it was great how willing all of the WSSSPE participants
were to share more and help each other. I also found it valuable that there
were other undergraduates at this WSSSPE, and it created another small
community and encouraged me to participate more.

     * •

buddy me-sheet and workshop planning and moderation was excellent

     * •

Don’t start at 8.45am if you’re asking people to socialize and connect in the
evenings! Work on getting a better gender, geographical and discipline
balance. The use of travel bursaries has helped (particularly in getting early
career participants).

     * •

Great organization, enjoyed being here very much, it was surely a success!

     * •

Great to meet colleagues from the UK and Germany at this event and enjoyed
this format for working, engaging, and producing with the community while at
the workshop that was facilitated by KnowInnovation.

     * •

I enjoyed the brainstorming session on Tuesday morning - well organized

     * •

I had hoped to have some discussion of building the sustainable communities
beyond the software writing stage. The HPC focus, though expected slightly,
was too heavy and I think ignores some of the other long term issues, such as
preservation and addressing first principles for new technologies, or smaller,
individual projects that will also support science. Perhaps it would be good
to have a student session as well? (But I do like the integration).

     * •

I think the mixing of students, staff, faculty etc in the working groups is
very effective and will contribute to outcomes outside of the working group.
It was great to see that we have undergraduates leading working groups here.
There should be more attention on diversity.

     * •

I wish there were a project on training interns to work at academic
environment as programmers

     * •

I would like to thank the committee for selecting me. As an undergraduate
student this was a great experience I got to meet professors from the
universities which I wanted to join. Now I exactly know how to prepare my
application. I learned some really good technical skills and will definitely
apply it to my research. So happy to be a part of this wonderful group. I will
design a logo for WSSSPE next year. Hope I can attend this event next year
also.

     * •

Is there something better than sharpies (few of running around the room with a
permanent marker)? Travel Awardee: Discussions here will and have benefitted
my organization and community.

     * •

It was great to connect with people doing research software engineering in
various positions, contexts, and universities. I felt like I learned a lot.
Moving between the two rooms was a bit confusing.

     * •

Maybe involve more researchers from low-income countries if funds are
available.

     * •

no. I think the moderator and the format are both great.

     * •

Nothing comes to mind

     * •

Receiving a travel award was crucial for me to come to WSSSPE4 this time,
especially because WSSSPE4 was held outside the US. On the other hand, I see
the great benefit this had on the geographic diversity of the participants, so
I do think that meetings not only in the US are of great value. And again, I
would like to emphasize, that travel awards are crucial for this type of
meeting, as most participants are not directly supported for this type of
activity - one reason to have a WSSSP5. (on another note: have a multi-line
text field for comments next time)

     * •

The issue of software sustainability is on par with the challenges of building
scientific instruments. Part of what I am for my doctoral dissertation is
finding better abstractions, which leads to sustainability through better
mathematics. This workshop raised my awareness in several other aspects, which
will be considered for this and all future projects I may be related to.

     * •

The travel award made it possible for me to come to WSSSPE-4 and broaden my
horizons. Many issues pertaining research software and sustainable software
were uncovered, and WSSSPE-4 brought to light the importance of the issues.
I’d encourage other travel awards to people, like me, who do not belong to the
community.

     * •

There aren’t lots of researchers knowing such kind of concept, sustainable
software. We need to let more RSE know it.

     * •

There is a lot of discussion, but it’s hard to see what actually will come of
it. The discussion of the 9 working groups from last year at the beginning of
the workshop clearly showed this: IIRC, not one had produced anything, all
seemed to be stalled somewhere. We ought to find ways to actually make things
*happen* after we talk about it (and admittedly have many good ideas).

     * •

This has been my first WSSSPE, and as someone coming from the humanities with
an interest in RSE it has been an enormously rewarding experience to be able
to interact with and learn from individuals and initiatives that have been
part of the greater discussion around WSSSPE topics more in both quantity and
quality. For me personally, it made me aware that I really need to refactor a
lot of my basic assumptions for a PhD thesis, but that’s a great thing because
it means that I was able to draw on the exchanges and experiences from WSSSPE
already. All in all, the workshop has made me more determined to become more
involved with this community. And last but not least i just wanna say THANK
YOU for the generous travel support I’ve received, which meant that my PI and
dept. didn’t have to have any doubts whether I’d drain their resources which
are meant to be used towards achieving a completely different research goal.

     * •

This was an excellent mix of people, timed/structured activities, and untimed
events.

     * •

This was an excellent mix of people, timed/structured activities, and untimed
events. I had the opportunity to talk to numerous people about projects,
issues, and ways to improve science/scientific software, and leave with many
new ideas and possible collaborations/cooperative efforts with complementary
projects. The working groups activities were excellent, a good mix of
projects, and I saw a lot of progress on several I’m very interested in.
(Please group this survey with my earlier one; I hit “end” too soon on the
other! Thanks!)

     * •

We have activity on slack, github, and twitter. And working groups have their
own stuff. Can we index these in a single place?

     * •

What percentage of the working groups actually end up producing anything? It
seems to me as though most come up with an idea and then find out they don’t
have the resources to implement it - nice as the ideas are. Also, I assume
someone is going to attempt to reinvent metrics or best practices every single
year, I guess that’s unavoidable.

     * •

WSSPE seems to be evolving nicely from year to year. I hope that in WSSSPE5
there are lots of results to report from the outcome of WSSSPE4 working
groups.

     * •

You could try unconferences instead of tracks.

## References

  * [1] Katz DS, Allen G, Chue Hong N, Parashar M, Proctor D.  First Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE): Submission and Peer-Review Process, and Results.  arXiv; 2013. 1311.3523.  Available from: http://arxiv.org/abs/1311.3523.
  * [2] Katz DS, Choi SCT, Lapp H, Maheshwari K, Löffler F, Turk M, et al.  Summary of the First Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE1).  Journal of Open Research Software. 2014;2(1).  Available from: https://doi.org/10.5334/jors.an.
  * [3] Katz DS, Allen G, Chue Hong N, Cranston K, Parashar M, Proctor D, et al.  Second Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE2): Submission, Peer-Review and Sorting Process, and Results.  arXiv; 2014. 1411.3464.  Available from: http://arxiv.org/abs/1411.3464.
  * [4] Katz DS, Choi ST, Wilkins-Diehr N, Chue Hong N, Venters CC, Howison J, et al.  Report on the Second Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE2).  Journal of Open Research Software. 2016;4(1):e7.  Available from: https://doi.org/10.5334/jors.85.
  * [5] Katz DS, Choi ST, Niemeyer KE, Hetherington J, Löffler F, Gunter D, et al.  Report on the Third Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE3).  Journal of Open Research Software. 2016;4(1):e37.  Available from: http://doi.org/10.5334/jors.118.
  * [6] Ilik V, Thaney K, Katz DS, others. FORCE11 Conference Code of Conduct; 2016\.  Accessed: 2016-09-15.  https://www.force11.org/meetings/force2016/code-conduct.
  * [7] Code4Lib. CodeOfConduct4Lib; 2015.  Accessed: 2016-09-15. Git hash 4270594fdbe28eeeb45256ea6304c8bd800e6a86.  https://github.com/code4lib/antiharassment-policy/blob/master/code_of_conduct.md.
  * [8] Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors.  Proceedings of the Fourth Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE4). Aachen; 2016.  Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/.
  * [9] Lago P, Koçak SA, Crnkovic I, Penzenstadler B.  Framing sustainability as a property of software quality.  Communications of the ACM. 2015;58(10):70–78.
  * [10] Huisman M, Bos H, Brinkkemper S, van Deursen A, Groote JF, Lago P, et al.  Software that Meets Its Intent.  In: Leveraging Applications of Formal Methods, Verification and Validation: Discussion, Dissemination, Applications. Springer; 2016. p. 609–625.
  * [11] Procaccianti G, Lago P, Vetro A, Fernández DM, Wieringa R.  The green lab: Experimentation in software energy efficiency.  In: Proceedings of the 37th International Conference on Software Engineering - Volume 2; 2015. p. 941–942.
  * [12] The Energy Efficiency Potential of Cloud-based Software: A U.S. Case Study.  Berkeley, California: Lawrence Berkeley National Lab; 2013.
  * [13] Procaccianti G, Fernández H, Lago P.  Empirical evaluation of two best practices for energy-efficient software development.  The Journal of Systems and Software. 2016;117:185–198.
  * [14] Heroux M.  Sustainable & Productive: Improving incentives for quality software.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_10.pdf.
  * [15] Allen G.  Establishing a professional society for research software.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_24.pdf.
  * [16] Philippe O, Hettrick S, Chue Hong N.  A survey of Research Software Engineers in the UK.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_19.pdf.
  * [17] Gwilliams C.  Using industrial engagement to create and develop research ties within academia.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_1.pdf.
  * [18] Dubey A, Riley K.  Software engineering and community codes track in ATPESC.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_17.pdf.
  * [19] Chue Hong N.  Why do we need to compare research software, and how should we do it?  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_29.pdf.
  * [20] Dubey A, McInnes LC.  The lifecycle of software for scientific simulation.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_16.pdf.
  * [21] Queiroz F, Spitz R.  Collaborative gamification design for scientific software.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_6.pdf.
  * [22] Childers B, Davidson J, Graves W, Rous B, Wilkinson D.  Active curation of artifacts is changing the way digital libraries will operate.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_8.pdf.
  * [23] Ganguly D, Garrison W, Wilkinson D, Childers B, Lee A, Mosse D.  Composing, reproducing, and sharing simulations.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_33.pdf.
  * [24] Shende S, Malony A.  Using TAU for performance evaluation of scientific software.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_12.pdf.
  * [25] Sallai J, Iacovella C, Klein C, Ma T.  Development of a software framework for formalizing forcefield atom-typing for molecular simulation.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_35.pdf.
  * [26] Sufi S.  The Software Sustainability Institute Fellowship programme; supporting the social side of research software.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_27.pdf.
  * [27] Gesing S, Dahan M, Hayden LB, Lawrence K, Marru S, Pierce M, et al.  The Science Gateways Community Institute - Supporting communities to achieve sustainability for their science gateways.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_32.pdf.
  * [28] Ram K, Ross N, Chamberlain S.  A model for peer review and onboarding research software.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_13.pdf.
  * [29] Löffler F, Brandt S.  A vision of computing in 10+ years.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_9.pdf.
  * [30] Katz DS, Niemeyer KE, Smith AM, FORCE11 Software Citation Working Group.  Software citation: Process, principles, and implementation.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_31.pdf.
  * [31] AM AS, Katz D, Niemeyer K, FORCE11 Software Citation Working Group.  Software citation principles.  PeerJ Computer Science. 2016;2:e86.
  * [32] Idaszak R, Tarboton DG, Yi H, Stealey M, Dash P, Couch A, et al.  HydroShare: A case study in software engineering best practices and culture change for developing sustainable community software.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_5.pdf.
  * [33] Hwang L, Bangerth W, Heister T, Kellogg L.  ASPECT: Hackathons as an example of sustaining an open source community.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_7.pdf.
  * [34] Goble C.  A simple profiling framework for software user-producer maturity review.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_2.pdf.
  * [35] Druskat S.  A proposal for the measurement and documentation of research software sustainability in interactive metadata repositories.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_20.pdf.
  * [36] Contrastin M, Danish M, Orchard D, Rice A.  Supporting software sustainability with lightweight specifications.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_23.pdf.
  * [37] Allen A, Aragon C, Becker C, Carver J, Chiş A, Combemale B, et al.  I solemnly pledge: A manifesto for personal responsibility in the engineering of academic software.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_15.pdf.
  * [38] Chue Hong N.  Making it easier to understand research software impact.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_30.pdf.
  * [39] van Hage WR, Maassen J, van Nieuwpoort R.  Software impact measurement at the Netherlands eScience Center.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_4.pdf.
  * [40] Allen G, Chen E, Idaszak R, Katz DS.  Report on software metrics for research software.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_26.pdf.
  * [41] Seidel EL, Allen G.  Bringing techniques from software engineering into scientific software.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_28.pdf.
  * [42] Emsley I, De Roure D.  Sustaining the social: Connecting the lives of Drupal community.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_34.pdf.
  * [43] Dongarra J, Hammarling S, Higham NJ, Relton SD, Valero-Lara P, Zounon M.  Creating a standardised set of batched BLAS routines.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_3.pdf.
  * [44] Breitwieser L, Bauer R, Meglio AD, Johard L, Kaiser M, Manca M, et al.  The BioDynaMo Project.  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_14.pdf.
  * [45] Aldabjan A, Haines R, Jay C.  How should we measure the relationship between code quality and software sustainability?  In: Allen G, Carver J, Choi SCT, Crick T, Crusoe MR, Gesing S, et al., editors. Workshop on Sustainable Software for Science: Practice and Experiences. No. 1686 in CEUR Workshop Proceedings. Aachen; 2016. Urn:nbn:de:0074-1686-8.  Available from: http://ceur-ws.org/Vol-1686/WSSSPE4_paper_22.pdf.
  * [46] Wilson G, Bryan J, Cranston K, Kitzes J, Nederbragt L, Teal TK.  Good Enough Practices in Scientific Computing.  arXiv; 2016. 1609.00037.  Available from: http://arxiv.org/abs/1609.00037.
  * [47] Wilson G, Aruliah DA, Brown CT, Chue Hong NP, Davis M, Guy RT, et al.  Best Practices for Scientific Computing.  PLOS Biology. 2014 01;12(1):1–7.  Available from: http://dx.doi.org/10.1371%2Fjournal.pbio.1001745.
  * [48] Ahmed Z, Zeeshan S, Dandekar T.  Butterfly: A Paradigm towards Stable Bio & Neuro Informatics Tools Development.  Frontiers in Neuroscience;(9).  Available from: http://www.frontiersin.org/neuroscience/10.3389/conf.fnins.2015.91.00009/full.
  * [49] Barstow D, Duffey R, Smoliar S, Vestal S.  An automatic programming system to support an experimental science.  In: Proceedings of the 6th international conference on Software engineering. IEEE Computer Society Press; 1982. p. 360–366.
  * [50] Towns J, Cockerill T, Dahan M, Foster I, Gaither K, Grimshaw A, et al.  XSEDE: accelerating scientific discovery.  Computing in Science & Engineering. 2014;16(5):62–74.
  * [51] Post DE, Votta LG.  Computational science demands a new paradigm.  Physics today. 2005;58(1):35–41.
  * [52] Vinter B, Kristensen MR, Lund SA, Blum T, Skovhede K.  Prototyping for Exascale.  In: Proceedings of the 3rd International Conference on Exascale Applications and Software. University of Edinburgh; 2015. p. 77–81.
  * [53] Hwu WM, Chang LW, Kim HS, Dakkak A, El Hajj I.  Transitioning HPC software to exascale heterogeneous computing.  In: Computational Electromagnetics International Workshop (CEM), 2015\. IEEE; 2015. p. 1–2.
  * [54] Fischbacher T, Franchin M, Bordignon G, Fangohr H.  A systematic approach to multiphysics extensions of finite-element-based micromagnetic simulations: Nmag.  IEEE Transactions on Magnetics. 2007;43(6):2896–2898.
  * [55] Jones MB, Boettiger C, Mayes AC, Smith A, Slaughter P, Niemeyer K, et al.. CodeMeta: an exchange schema for software metadata.  KNB Data Repository; 2016.  Available from: https://doi.org/10.5063/SCHEMA/CODEMETA-1.0.
  * [56] Gent I, Jones C, Matthews B.  Guidelines for persistently identifying software using DataCite.  Science and Technology Facilities Council; 2015. 24058274.  https://epubs.stfc.ac.uk/work/24058274.
  * [57] WSSSPE4 Software Best Practices for Undergraduates Working Group. WSSSPE4 Software Best Practices for Undergraduates Google Group; 2015.  Accessed: 2015-12-11.  https://groups.google.com/forum/#!forum/wssspe4-software-best-practices-for-undergraduates.
  * [58] WSSSPE Organization. WSSSPE Slack; 2016.  Accessed: 2016-12-11.  https://wssspe.signup.team/.
  * [59] WSSSPE4 Software Best Practices for Undergraduates Working Group. WSSSPE4 Software Best Practices for Undergraduates GitHub Organization; 2015.  Accessed: 2015-12-11.  https://github.com/orgs/WSSSPE4-undergraduate-course.
  * [60] Patterson D, Snyder L, Ullman J.  Computing Research Association, Best Practices Memo: Evaluating Computer Scientists and Engineers For Promotion and Tenure.  Computing Research News. 1999 September;p. A–B, special insert.
  * [61] National Science Foundation.  Dear Colleague Letter - Supporting Scientific Discovery through Norms and Practices for Software and Data Citation and Attribution.  NSF; 2014. 14-059.  https://www.nsf.gov/pubs/2014/nsf14059/nsf14059.jsp.
  * [62] National Research Council; Computer Science and Telecommunications Board; Committee on Academic Careers for Experimental Computer Scientists, Snyder L, et al.  Academic Careers for Experimental Computer Scientists and Engineers.  The National Academies Press; 1994.