5/2/2020 Lab 8 | CSCI 1320
https://cs.brown.edu/courses/csci1320/labs/lab8/lab8.html 1/6
CSCI 1320
LAB 8: TESTING
DUE MONDAY, MAY 4 10AM
Testing a web application that will be used by the external world is an integral and necessary step
towards success. For this lab, you will be put your final project under various types of testing. Testing
allows you to answer many questions about your program:
Does it work correctly?
Does it work correctly under all cicumstances?
Is it accessible to a wide range of users?
Does it work across devices and browsers?
Will it scale?
You are not trying to prove that your program has no bugs, but rather vigorously finding them via
thorough analysis. In that way, testing cannot prove that your entire program is right, but simply what
is not wrong. The difference is subtle, but the more rigorous your testing is, the closer you will be to a
flawless program. To be a good tester, you should have a mindset that is concentrated on breaking
things and finding edge cases. This will ensure you will find and patch some of the holes that are
lurking in your application.
This lab provides the resources necessary for you to test the project you have been working on this
semester. This lab is your chance to get started and get help. You should work with your final project
team to test applicable aspects for your project.
INTRODUCTION
For the lab, you will be testing your own final project with your fellow group members. Importantly,
you should never test production data, data that will be served or used externally. This means that if
your program is serving data from a database, you should not run tests that directly modifies that
database. The main reason for this is to prevent unwanted changes and manipulations to data used
publically.
PREPARATION
5/2/2020 Lab 8 | CSCI 1320
https://cs.brown.edu/courses/csci1320/labs/lab8/lab8.html 2/6
Thus, the first task of this lab is to prepare your data from your final project for testing. This will
probably mean making a copy of your database (or a subset of data from your database).
If your program depends on functionality from integrating with various APIs, such as using Paypal or
Stripe for payment purposes, be sure to check the relevant testing documentation provided by each
service. Testing is such an important feature that most reliable APIs will provide documentation and
testing environments for developers to thoroughly ensure their integrations are running smoothly.
We have provided a couple links in next section for some documentation on testing various services
that might be used in some final projects, but this list is not complete. Be sure to find the correct
documentation for any integrations you may have in your project. If you have trouble with this, reach
out to a TA for assistance.
The last preparation task you need to complete is to communicate with your group members about
dividing the different types of testing detailed below. You must thoroughly test all categories of your
program.
For each testing category below:
determine whether it is applicable to your application
If it is not applicable, explain why you think this category is not applicable.
If it is applicable, choose at least one of the resources to test your application. You are welcome to use
resources not listed to test your application but make sure to explain why you think the tool you used is
a good addition.
Make sure every team member is involved and work with at least one category of testing.
In each category, we use blue background to highlight the tool we recommend.
Web
WAVE
Common Errors:
Missing Form Labels : This error happens when a form control does not have a
corresponding label. For example, when you have a element by its own.
You can fix this problem by supplying a label. If you do not want the label to be actually
displayed, you can consider making it screen-reader only .
Empty Button : This error happens when a button is empty or has no value text. For
example, when you only put a icon inside a button.
You can fix this problem by supplying screen-reader only text .
CynthiaSays is an alternative website Accessibility scan tool. It requires your application to be
online (deployed) and usually takes longer than WAVE to generate a report. However, this site
covers some different accessibility categories and can be used as a supplement to WAVE.
AChecker . This tool checks single HTML pages for conformance with accessibility standards.
FAE : Functional Accessibility Evaluator.
toptal : Color blind views of your page.
Mobile
A11y Ally enables inspecting Android applications' accessibility.
ACCESSIBILITY
5/2/2020 Lab 8 | CSCI 1320
https://cs.brown.edu/courses/csci1320/labs/lab8/lab8.html 3/6
XCode Accessibility Inspector: Getting Started Tutorial
Application testing (also known as E2E testing) is about testing the components of the system as a
whole, such as how well the web page interfaces with the web server, the server with the database,
and the web server with the user server.
Web
Chrome Devtools Audit (Lighthouse) is an open-source, automated tool for improving the
quality of web pages.
Selenium : Automates web browser for testing. Selenium is more low-level compared to the
following e2e libraries.
Cypress
Nightwatch.js uses the W3C WebDriver API to drive browsers in order to perform commands
and assertions on DOM elements.
Puppeteer is a Node library which provides a high-level API to control Chrome or Chromium
over the DevTools Protocol.
TestCafe : Write tests in JS or TypeScript, run them and view results
Mobile
Detox supports both native apps and React Native projects.
Appium is an open source test automation framework for use with native, hybrid and mobile
web apps. It drives iOS, Android, and Windows apps using the WebDriver protocol. Appium is
more low-level compared to the following e2e libraries.
Kobiton supports real device testing on both Android and iOS devices.
TestProject is a free end-to-end test automation platform for web, mobile, and API testing
APPLICATION TESTING
Back end functional testing (or often called unit testing) is about checking if requests are handled
correctly and that the proper actions are taken and the proper pages are generated. The specific tools
used depends on the technology used in the back end. Feel free to pick one of the tools below,
depending on what language your back end uses.
Node: Jest , Mocha , Jasmine that also works for Node
Python: PyUnit
PHP: SimpleTest
BACKEND TESTING
Compatibility testing is about testing if the web application is compatible across many types of
browsers, such as IE, Firefox, Safari, Chrome, Opera, Edge, etc. and across mobile platforms and
operating systems.
COMPATIBILITY TESTING
5/2/2020 Lab 8 | CSCI 1320
https://cs.brown.edu/courses/csci1320/labs/lab8/lab8.html 4/6
Web
With a free trial on browserstack.com , you can test your locally hosted site--just choose a browser
to test, click the settings icon, and check the box under Server Testing labeled "Resolve all URLs
through my network."
You can use browsershots.org if your site is already hosted!
You can also manually test out core functionalities of your application by trying it in different
browser (Chrome, Safari, Firefox, Edge), operating system (MacOS, Windows, Linux) combinations.
Mobile
Kobiton supports real device testing on both Android and iOS devices.
TestProject is a free end-to-end test automation platform for web, mobile, and API testing
Front end functional testing is about testing the correctness of the HTML, CSS, and JavaScript, which
includes aspects like checking if links are correct and active, if forms are validated correctly and have
default values, if the site handles cookies correctly (e.g. the application works without cookies, cookies
are encrypted correctly, and sessions expire correctly).
HTML
W3C HTML Validator
CSS
W3C CSS Validator
Links
W3C Link Validator
JavaScript
Jest was built and is constantly maintained by the team at Facebook. It is a zero-configuration
javascript testing framework recommended by React.
Mocha provides great freedom and flexibility.
Jasmine very beneficial for frontend testing.
QUnit originally developed for testing jQuery, jQuery UI and jQuery Mobile, it is a generic
framework for testing any JavaScript code.
Mobile (Overview)
Android: Test apps on Android
IOS: Testing with Xcode
SauceLabs is a live, automated and continuous testing for web mobile apps on cloud-based
test platform (2 week free trial)
Appium is an open source test automation framework for use with native, hybrid and mobile
web apps. It drives iOS, Android, and Windows apps using the WebDriver protocol. Appium is
more low-level compared to the following e2e libraries.
FRONTEND TESTING
If your application involves payment, follow the testing documentation of the third-party payment tool
you used.
PAYMENT TESTING
5/2/2020 Lab 8 | CSCI 1320
https://cs.brown.edu/courses/csci1320/labs/lab8/lab8.html 5/6
Stripe
PayPal
Square
Performance testing is about testing the responsiveness of a web application, the time taken for
common actions. It includes load testing and network performance, as well as stress testing the limits
of the application.
Web
Performance Tab of Chrome Devtools.
GTmetrix : Gives insight on how well your site loads and provides actionable recommendations
on how to optimize it (requires deployment) .
JMeter : Designed to load test functional behavior and measure performance. It works with a
set of test cases where you can specify the quantity and delays. A fairly complicated and
involved system.
Mobile
IOS: Instruments Tutorial
Android: Measure UI Performance
Cross Platform: DynTrace (offers free trial),
PERFORMANCE TESTING
Security testing is about testing the different ways that security can be breached, including URL security, input
checking, and other methods.
Web
Wapiti is a command-line tool that performs "black-box" scans (it does not study the source
code) of the web application by crawling the webpages of the deployed webapp, looking for
scripts and forms where it can inject data.
OWASP Zed Attack Proxy (ZAP) ( Getting Started , Downloads ) is a multi-platform, open
source web application security testing tool useful for penetration test.
Netsparker is a leading web application security scanner (signup needed).
Mobile
OWASP Zed Attack Proxy (ZAP) ( Use ZAP to test mobile app , Downloads ) is a multi-platform,
open source web application security testing tool useful for penetration test.
Mobile Security Framework (MobSF) is an automated, all-in-one mobile application
(Android/iOS/Windows) pen-testing, malware analysis and security assessment framework
capable of performing static and dynamic analysis.
Quick Android Review Kit is designed to look for several security related Android application
vulnerabilities, either in source code or packaged APKs.
SECURITY TESTING
USABILITY
5/2/2020 Lab 8 | CSCI 1320
https://cs.brown.edu/courses/csci1320/labs/lab8/lab8.html 6/6
Usability is about testing the effectiveness of the user interface, including accessibility and
internationalization testing. It's about seeing what is liked or disliked about the interface, as well as the
speed and ease of use, and whether it is understandable, logical, and easy to follow. This can be done
through user studies or log studies.
You should use bdognom-v2.cs.brown.edu:5002 to submit a user test and complete at least one user
test.
Moreover, in addition to complete user testing task for class, you can either ask your friends to test your
application (similar as what you did for user feedback report) or use online user testing platform like
UserTesting .
Here are a few helpful resources for you to get started on using UserTesting:
The quickest step-through of using usertesting.com
Check out this demo video which will cover what the UserTesting platform does and the basics of
setting up a study
Fill out this planning worksheet to plan a study
Complete guide to user testing
Once you have set up an account, further information can be found here:
How to launch a desktop study
How to launch a mobile website study
How to launch an app study
How to launch a prototype study
Once your team are done testing the various components of your project, one member from the group
has to handin a testing report to Gradescope with every team member included in the submission
group.
The testing report should be a PDF document outlining the following:
Basic Information: your final project name, your mentor ta name, your team member name and
CS Login
Collaboration: Which team member is responsible for which testing categories.
Testing Report: for each testing category, describe the following
Category Name
Which tools are used?
Which tests were run as part of the testing?
Why you choose these tests?
Why do you believe these tests are sufficient?
What results you receive and how do you interpret these results?
Steps taken to fix the bugs
Look at Handin Instructions .
HAND IN