CS 349 - User interfaces, LEC 001
6-25-2016
Elvin Yung
-
We can describe interaction in terms of look and feel.
-
Look is how things are presented in the interface - like the layout.
-
Feel is how responsive the interface is, how users express intent and get feedback, etc.
-
We've talked about the execution-evaluation model, but that's at a different level of abstraction.
-
We're more concerned with the actual interaction with the application.
- We're going to think of UIs as things that we physically interact with.
- A domain object is a thing that you're trying to manipulate.
- The interaction instruments are the things that you use to manipulate the domain object.
- We can use these concepts to decompose a UI.
- You can think of an interface as a bunch of tools that you can use to manipulate an object.
TODO: insert diagram from slide 7
-
The key thing when you're talking about instruments is activation, i.e. how do you trigger it?
-
For example, with a scrollbar, how do you activate it? You click on it and drag to cause it to move, or click it in the gutter to get it to move in different directions.
-
Another example: Making a rectangle in Powerpoint is basically a multiple step process: You have to find the widget to open the shape menu, click it, find and click the rectangle, and click-drag it on the canvas.
-
You can think of instruments as being activated spatially or temporally.
-
Spatial activation means that you directly cause it, usually by directly manipulating some part of the interface. e.g. the scrollbar
-
Temporal activation means that there's a sequence of actions to activate it. e.g. Everything inside the shape menu, like making a rectangle.
-
Most things are spatially activated to an extent, but have different costs, i.e. the amount of work it takes to perform that action.
-
Usually, something that involves an object being selected is at least somewhat temporal, since it has the selection as a precondition.
-
Dialogs are, by definition, temporal. You always have to do something to bring up the dialog.
TODO: add slide 10 diagram
- There are cost involved with different activations.
- For spatial activations, the cost is the space used, and the time it takes to activate it.
- The cost of activating something temporally is the action time of everything you have to go through to activate it.
- In general, spatial activation costs less than temporal activation, so (as you might expect) the most frequently used features should be spatially activated.
- If you don't want to incur temporal cost, you can always add more spatially-activated components.
- But that's also bad, because input devices clutter up the UI.
- The best way is to properly make tradeoffs and manage costs appropriately.
- The degree of indirection is how much you have to do to the instrument to perform the action.
- For spatial activation, the cost is higher when
- For temporal activation, the cost is higher
- The degree of integration is the similarity between the degrees of freedom of your input device and the instrument on the screen.
- For example, a scrollbar has a DOF of 1, and a mouse has a DOF of 2. The scroll wheel, however, also has a DOF of 1, so it is a better fit.
- Another example: Rotating a shape in 3D involves a DOF of 3, but a mouse still only has a DOF of 2. However, we can combine the mouse with a scroll wheel to get a DOF of 3, for a better fit.
- The degree of compatibility is how similar the physical action on the instrument is to the response from the interface.
- For example, dragging has a high degree of compatibility because the feedback directly maps to the motions to move the mouse.
- Scrolling has medium compatibility.
- A dialog box generally has low compatibility. When you change a font, you're doing a bunch of things that don't really feel compatible to what you do with your mouse.
- An effort by a bunch of researchers to promote natural user interfaces.
- We're not going to talk about this at all - they're not really novel since they're basically touch interfaces.
- Up until now, we've been mostly talking about standard mice/keyboard driven GUIs.
- With the rise of multitouch-enabled devices like smartphones and tablets, touch-based interfaces are increasingly important.
- How old is multi-touch?
- We'll go back on a common theme in technology: It takes a long time to go from a vision to a viable product.
- Bill Buxton started doing research with multitouch in the mid 1980s, but it was only in 2007 that the iPhone - the first popular consumer multitouch product - was released.
- Was literally two closely-pressed ("sandwiched" together) conductive layers, when a point is pressed together it registers the point
- Didn't handle multitouch
- Emitters at the 4 corners of the screen
- Indirectively measure change in capacitance to figure out where the finger tapped.
- Can also only handle one point.
- Has an array of sensors, and measure the capacitance change at every place.
- You need some conductive material touching it to actually trigger it.
- Handles multitouch!
- This is the touchscreen technology that's in every device now.
- Generally, modern phones have a ridiculous amount of sensors that we don't usually take advantage of completely.
- Magnetic layer at the back of the screen.
- Magnetized stylus induces an electromagnetic field in the sensing layer.
- We might have active styluses that use bluetooth to talk to the device.
- Can combine with a capacitive touch screen, so that the device can process (and distinguish) touch and pen input.
- Expensive, and fairly rare. The magnetic layer is pretty expensive to produce.
- Literally have a camera watch the interface
- Flood an entire surface with infrared sensors to detect where people are touching it.
- Not super precise, but like mutual capacitance, cheaper at bigger sizes.
- How do you actually get input information into the system?
- Up until now, we've only really talked about indirect input devices, e.g. mice, touchpads, joysticks, where you don't directly control the display.
- Now we'll talk about direct input devices, where we basically unify the input and output devices. A perfect example: a touchscreen!
- Direct touchscreens are generally more common.
- For some very specific use cases, e.g. writing, it makes sense to have a stylus for input.
I would rather draw with my fingers on the dusted windshield of my car than draw with the phone.
-
One more difference between touch and mouse is that there are less input states.
-
With a mouse, you have three states: out of range, tracking, and dragging. In other words, the mouse button is a factor.
-
With touchscreens, you only get two states: touching, and not touching.
-
With pens, maybe you can have the stylus have two different states (active or passive) that makes it kind of like the mouse.
-
Some touch-based inputs incorporate pressure, i.e. how much force the user exerts on the touch input.
-
Apple has force touch, which lets developers do different things based on different pressured touches.
-
On Android, there's no real pressure sensor. Instead, it uses contact area sensing, which uses the size of the contact area to approximate the pressure.
-
Real pressure input is just around the corner!
-
The human finger is massive compared to a pen or mouse cursor!
-
If your finger is bigger than the target with, it makes it hard to see what you're doing.
-
In some touchscreen interfaces (e.g. text select/copy/paste on most modern phones), it usually shows a preview of what you're doing above the touch area.
-
Another interesting thing: LucidTouch - sense touches from the back of the device, and indicate contact on the screen. Pseudo-transparency!
-
It also becomes hard to give users meaningful feedback when the finger is covering things.
-
A standard industry solution: make everything huge. Apple recommends 15mm as the minimum width for a button.
-
Ideally, you want to have some part of the widget visible under your finger.
- Normally, with a mouse, users feel a "click" when they do something.
- With touchscreen, this sort of haptic feedback is harder.
- When something isn't working with a touchscreen interface, it's hard for the user to tell what happened if there isn't obvious feedback.
- Usually phones provide some sort of audio or haptic feedback when you tap.
- With a mouse, when the mouse button isn't pressed down, you get a "hover" state, which gives you things like tooltips.
- With a touchscreen, you don't get this.
- With a mouse, if you mousedown and mouseup with the cursor in the same widget, you register it as a click.
- What do you when you have multiple points of contact, not all of which might be on the widget?
- Various ways:
- Microsoft Surface: Only generate an event when the last contact point is lifted from the screen.
- DiamondSpin: Register each contact independently. But then you have to be careful with your touches.
- When you're working with a touchscreen, you're generally supposed to be interacting with the object on screen directly.
- In other words, the touchscreen relies on direct manipulation.
- Google and Apple realized that these aren't enough - in a lot of cases, feedback is lacking.
- They solve this in a few ways.
- For example, when you scroll to the end and try to scroll further, the screen keeps scrolling, and snaps back elastically when you release your touch.
- We've talked about WIMP (windows, icons, menus and pointing), which is the traditional interaction model for GUI interfaces.
- We don't do that with touchscreens.
- Instead, we use something called direct manipulate (DM).
- Not exclusive to touchscreens - for example, drag and drop, which is common in GUIs, is a DM interaction.
- Direct manipulation relies on the concept of affordances (which we've already talked about), which is the idea that you should be able to figure out what you can do to something based on how it looks like.
- e.g. drag and drop, changing the size of a shape by dragging the corners, etc.
(A bit of nomenclature here: direct touch is short for direct manipulation on a touch interface.)
- It's a bit nebulous when you can apply DM and when you can't - it's easy to apply to things like painting, but not so much things like word processing. You need the right task for it to work well.
- Basically, WYSIWYG.
// TODO: copy points from slides
- Interactions should be natural and learnable.
- Accessibility is hard
- Visually impaired users can't see the graphics
- People who use screen readers find it hard to have a linear flow
- Gestures are hard for some physically impaired users
- etc.
- DM assumes that there's an obvious suggested way to interact with the system, but this isn't easy to do.
- Not all interactions are valid! What does it mean to resize the trashcan?
-
Touchscreens support 10+ points of contact
-
Android supports only touch gestures like touch, long press, pinch open/close, etc.
-
This is actually a pretty limited set of interactions.
-
This is mostly because it's hard to remember lots of gestures, and these are basically the most intuitive gestures.
-
Apple patented many more, but they're all pretty complicated.
-
Another question? Are gestures really natural and intuitive?
-
We think that we do things in the real world that are easily mappable on a touchscreen.
-
For things like copy-and-paste and filling a shape with color, it's hard to figure out the right way to do it on a touchscreen.
-
Unless you train someone on how to use a touchscreen, it's not explorable.
-
So what do we do?
-
Right now, we basically copy whatever Apple and Google do.
-
A guessability study is a way to help design gestures. Basically, a researcher gives you an application, ask them to perform something, and see how they think the interaction should be like.
-
A famous study was one that Wobbrock et al. did.
-
Overwhelmingly, people prefer single-finger gestures.
-
One hand gestures got much more consensus than two hand gestures.
-
Interestingly, we're looking for gestures that are similar to what we did with the mouse.
(not related to D&D...)
- Fat body part problem
- Content orientation
- Hard to reach some parts of the display
- Obviously, you can't just take a desktop application and just scale it down for mobile.
- So usually what happens is that you provide a more minimal UI.
- The rest of the features, generally you support it but hide it so that it doesn't block other things in the UI.
- You generally only run a single application at a time on a phone
- The display fills up the entire screen.
- On a computer, you can usually assume that the user is at a desk, with good lighting, paying full attention, etc.
- With a phone, you can't assume that - the user can be doing lots more things that you need to account for.
- e.g. can't give tooltips, since you have no space.
- Tutorials are bad - Your UI needs to be intuitive.
Here are some tradeoffs that mobile developers make:
- Show different keyboards based on different types of input, e.g. numpad for phone numbers, etc.
- Predict input - autocomplete, prepopulated lists, etc.
- Make frequent operations easy to access
- Make actions obvious - have few large buttons, rather than many small buttons.
- Affordances - Collapse content and controls together.
- Make some controls expandable/hideable, etc.
- Hide things you don't need, like message metadata, etc. delete buttons