Skip to main content

Test design features

A tour of the design tools available when you build a test, from screening participants through to the analysis you get back, with links to the detailed guides for each one.

BetaTesting is a flexible platform for beta testing and user research. This article is a tour of the design tools available when you build a test, from screening participants through to the analysis you get back, with links to the detailed guides for each one.

Use it when you are planning a new test and want to know what the builder can do before you start, or when you have run simple single-survey tests and want to see what a multi-activity study, an A/B comparison, or an in-home product test would involve.

Most of these features work together. A screening survey decides who gets in, tags and activity groups decide what each participant sees, and the built-in analysis turns the answers into charts and statistics without extra setup. Knowing what is available up front helps you design a test that collects the right data the first time, rather than discovering a gap after the results are in.

Some of the features below depend on your plan. Where that applies, it is noted in the section and summarized under Things to know.

Participant onboarding

These features control who gets into your test and what you know about them before they start.

  • Screening surveys. Add screening questions that applicants answer before they are accepted. Mark each answer as accepted or rejected, then choose Automatic Selection to accept everyone who meets your targeting criteria and passes every screening question, or Manual Review to look at each applicant's demographics and answers and accept them yourself. See Screening surveys.

  • Quotas. Split your participants into quota groups by device platform, age, gender, country, or a profile field, and set a target for each. Applicants are placed in the first group they match that still has room. Quotas never reject anyone: once a group is full, further qualified applicants go to your Pending list, where you can accept them by hand if you want more. Each tester's quota group is shown on the Testers page and in exports, and you can reallocate targets when adding more testers to a live test.

  • Digital agreements. Upload an NDA or other project terms as a PDF and applicants sign electronically as part of applying. You can download the signature details afterwards. See Can we have the users sign NDAs?

  • Device detection. Choose which device platforms testers can use, and optionally collect device information when they apply. Device model and OS version are detected automatically, so you can filter results and issue reports by device later. See Testing on more than one device platform.

Flexible test flows

A test can be a single survey or a sequence of activities released on a schedule.

  • Sequence multiple activities. Each activity has its own distribution setting (at the start of the test, at a specific date and time, a set time after a tester starts, or after a tester completes another activity) and its own deadline. See Understanding task and survey distribution.

  • Skip logic and show/hide logic. Send testers to a different question based on their answer, or show and hide questions depending on earlier answers. See Create branching and show/hide logic.

  • Randomization. Randomize the order of answer options in multiple choice and ranking questions, and randomize the order in which activities in a group are delivered.

  • Different paths for different participants. Use tags as a prerequisite on an activity so only testers with that tag receive it. This lets one test run different processes for different groups. See Using tags.

  • Activity groups. Bundle activities so they share one schedule and deadline, then randomize their order or give each tester a subset. This is how you set up A/B tests, concept comparisons, and multi-day studies. See Using activity groups.

  • One question at a time, or all on one page. Surveys show one question per page by default. Turn on Show all questions in the same page to present the whole survey at once. See How to customize the survey-taking experience.

Question types

Five core question types cover most research needs, and the analysis works out the rest. See What type of survey questions can I add? for each type, and Creating high quality surveys for how to combine them into a survey that gets usable answers.

  • Text, multiple choice, scale, ranking, and matrix. Matrix questions ask several ratings at once and support a semantic differential format with opposing word pairs. See Ask several ratings at once with a matrix question.

  • Multiple choice options. Single or multi-select, a limit on how many options can be selected, an Other free text response, and randomized option order.

  • Ranking. Rank the full list or just the top N, with tap-to-rank on mobile. See Asking testers for their top picks.

  • Question bank. Ready-made items for NPS, customer effort, customer satisfaction, star ratings, and participation verification that you can drop into any survey.

  • Dynamic data. Upload a list of records such as promo codes, login credentials, or test card details, and each participant is given one. Useful whenever every tester needs something unique.

Rich content and media

Surveys can carry as much context as testers need, and collect any kind of evidence back.

  • Formatting. A formatting toolbar on every item supports headers, bold, italic, links, and lists.

  • Attachments. Attach any media to any item, including video, images, and PDFs, from your computer or from cloud storage.

  • Video collection. Collect usability recordings or selfie videos, with multiple videos per question and long recordings supported. Turn on Reject videos without audio to send silent submissions back to the tester automatically; leave it off and the tester gets a heads-up instead.

  • File uploads. Accept any file type from testers.

  • Mobile-first design. Surveys are built to be completed on a phone as easily as on a desktop.

  • Automatic translation. If your plan includes automatic translation, participants see your questions and answer options in their preferred language with no setup on your side, and a Show translation button on each response lets you read their answers in yours. See Automatic language translation of test content and communications. Separately, and on every plan, testers and admins can switch the platform interface to their own language; see Using BetaTesting in your own language.

Issue reporting and communication

Surveys are not the only way to hear from testers.

  • Issue reporting. Testers can submit issue reports alongside their survey answers, with priorities, statuses, and device details attached. The report form can include custom fields, and reports can sync to Jira, which you connect from the test's Advanced Settings page. See Managing bug reports, Integrating with Jira, and Advanced settings for your test.

  • Messaging. Message an individual tester in a thread, or send a broadcast to an audience such as everyone accepted, testers who finished a survey, testers who missed a due date, or a hand-picked list. See Communicating with testers during live tests.

Analysis built in

Simple question types, advanced data analysis. You write the question the way you would naturally ask it, and the results page works out how to present it. See Chart statistics and overlays.

  • Auto-detected data types. Each multiple choice question is classified as binary, nominal, ordinal, Likert, numeric rating, interval, or ratio, and gets the right chart and statistics for that type. A Likert scale gets an average and a confidence interval; a category list gets shares and the most common answer. The detected type is shown on the chart so you can check it.

  • Scale direction detected automatically. If your options run from best to worst instead of worst to best, the analysis notices, so averages are never inverted.

  • Not applicable options recognized. Answers like "Not applicable" keep their bar on the chart but stay out of the average.

  • Confidence intervals and NPS. Every average and share comes with a confidence interval at 90%, 95%, or 99%, and any 0 to 10 rating question gets a Net Promoter Score. Results can also be compared across tests or over time; see Comparing charts.

  • AI insights. A test-wide AI analysis groups tester feedback into themes, and on plans with advanced AI features you can give it your own categories and instructions so the report covers what the test was meant to answer. See Test-wide AI insights and Guiding the AI Insights analysis with your own categories.

Things to know

Manual review of applicants, quotas, matrix questions, activity groups, automatic translation, and custom AI analysis depend on your plan. If you do not see one of them in your builder, contact us about enabling it.

Quota targets must add up to your participant count before launch, and a test can have up to 20 quota groups. Profile-field quotas are available on community tests.

Skip logic is available on multiple choice questions. Randomization applies to answer options and to activities within a group; the order of questions within a survey is not randomized.

Device detection records the platform, device model, and OS version. You can require specific platforms for a test, but not specific OS versions; OS version is available as a filter when reviewing results and issues.

Digital agreements must be uploaded as a PDF, one per Document Signing item. Add another item to your screening survey if you need a second document signed.

Broadcasts are limited to 25,000 recipients per message. Testers can opt out of messages, so delivery is not guaranteed for every recipient.

Did this answer your question?