Create empirica users survey

[JamesPHoughton]

Develop a survey for empirica users. Can reuse some components from survey developed for IC2S2 tutorial.
(Couldn't think of a better repo for hosting this issue)

Questions to ask:

• field of study,
• career stage,
• programming experience,
• familiarity with other tools,
• whether they created the experiment themselves or had help from an academic collaborator, a research assistant, or a developer,
• if they have any published works, and any ongoing unpublished projects

Some questions are appropriate to existing users, some are appropriate to current users - could implement a branching structure.

Structure

• Qualtrics or google forms? (or something else?) Google forms might be easier to share amongst the empirica team.

@amaatouq, @malsobay, @npaton, @shapeseas - any additional questions you'd be interested in having, or thoughts on using one platform vs another?

I'll have a crack at a first draft Wednesday the 2nd.

[npaton]

Maybe some questions about deployment, recruitment and operations. Here are are some question ideas, this is not my forte, please change, delete or add questions.

Deployment

If you have deployed empirica, or a similar open source system.

• How did you deploy?
• Did you deploy frequently? (while developing to test or between experiment runs)
• How long did you run your experiment?
• How long did you keep your deployment running?
• Were you hosted in the cloud or faculty servers?
• What were your hosting costs?
• Did you require/get assistance from faculty IT, operations or a developer?
• Were there particular pain points you remember?

Recruitment

I have little knowledge here.

• Did you use a participant recruitment platform? If yes, which one?
• What was your experience with the platform? Good and bad experiences.
• How did you manage recruitment? (pre-recruitment, on demand, etc.)
• How did you schedule experiment runs?
• How did you communicate with participants?
• How did you manage payment?
• How much did you pay participants?

Operations

I think I put operations questions in the last 2 sections… Maybe this could be better organized.

[JamesPHoughton]

Not sure a better way to share this survey with folks who don't use qualtrics. Here's a PDF with every question:
Empirica User Survey.pdf

There is branching logic, here:
Branching_Logic.pdf

The main controls on branching are this question:

If respondents say they are responsible for implementing the experiment, then we ask them the "developer" questions, if they say they are responsible for deploy, we ask them those questions, etc. That way we should avoid giving people a bunch of irrelevant questions.

Here's a link to try out:
https://upenn.co1.qualtrics.com/jfe/form/SV_ai2PzIUzB7SsWx0

Let me know what feedback you'd give. =)

[malsobay]

The survey looks pretty comprehensive, thanks @JamesPHoughton ! Some thoughts:

• I may have missed it, but I think we should have some open-ended questions (e.g. feedback, wishlist, etc.); the current questions do a good job of understanding the user and their current behavior, but open-ended questions could be very valuable for the Empirica dev roadmap (ideas, prioritization, etc.).
• I know this is expanding the scope, but I think it makes sense to also have a branch for people who are not yet Empirica users. For example, we have many people in the slack who may have attended a tutorial, heard about it from coworkers, etc. but are not yet active users, and it would be good to understand that segment as well. For this segment, I think it's important to understand whether they are not conducting experiments at all, or if they are already doing experiments but something about Empirica is keeping them from using it (and if they're already experimenting, what tools do they use?).