Touch behavior is strange when only mouse interaction is described

[inaridarkfox4231]

Most appropriate sub-area of p5.js?

• Accessibility
• Color
• Core/Environment/Rendering
• Data
• DOM
• Events
• Image
• IO
• Math
• Typography
• Utilities
• WebGL
• Build process
• Unit testing
• Internationalization
• Friendly errors
• Other (specify if possible)

p5.js version

1.9.1

Web browser and version

Chrome

Operating system

Windows

Steps to reproduce this

Steps:

1. Describe the interaction using only mousePressed and mouseReleased.
2. Inside the draw loop, use mouseIsPressed to perform console output.
3. In the case of touch, mousePressed and mouseReleased are executed in this order at the timing of release.

Snippet:

function setup() {
  createCanvas(400, 400);
}

function draw() {
  background(220);
  if(mouseIsPressed){
    console.log("is");
  }
}

function mousePressed(){
  console.log("mousePressed");
}

function mouseReleased(){
  console.log("mouseReleased");
}

2024-09-10.23-30-17.mp4

Note that I use a stylus pen for touch because to check on a desktop, but the results are the same on a mobile phone.

In the above code, in the case of touch, only the processing at mouseIsPressed is executed, and at the timing of release, the contents of mousePressed and mouseReleased are executed in this order.

However, the following code achieves the intended behavior even on touch operations. In other words, first mousePressed is executed, then the process for mouseIsPressed is executed in the draw loop, and finally mouseReleased is executed.

function setup() {
  createCanvas(400, 400);
}

function draw() {
  background(220);
  if(mouseIsPressed){
    console.log("is");
  }
}

function touchStarted(){
  console.log("mousePressed");
}

function touchEnded(){
  console.log("mouseReleased");
}

Regarding the mouse, either code results in the intended behavior.

In other words, if you only describe the interaction when using the mouse, mousePressed() will not be executed when touch is started.

Furthermore, in the case of the first code above, if the start and end positions of the touch are different, neither mousePressed nor mouseReleased seems to be executed.

2024-09-10.23-56-55.mp4

It's hard to see in the video, but if you touch it, move it, and then release it, none of the processes will be executed. Any second code will be executed.

OpenProcessing: Interaction bug

[inaridarkfox4231]

The reason I brought up this issue is because a sketch I wrote in the past broke due to a version update.

example sketch (it will broke after 1.9.1 on mobile phone)

[msudipta888]

hey i understand it can you please assign me this issue

[inaridarkfox4231]

thank you!

[inaridarkfox4231]

I'll explain why I consider this a bug.

According to the current specifications, if you only write mousePressed() and do not write touchStarted(), the contents of mousePressed() will be executed for a mouse down operation, but not for a touch start operation.
However, this does not mean that it is not executed at all. For touch start operations, the contents of mousePressed() are executed after the touch end operation. If the content of mouseReleased() is also prepared, it will be executed after that.
However, this is not the intended timing.

What kind of trouble does this cause? The following sketch is second example of reference site, but it doesn't work on my Android. Of course, a stylus pen won't work either. The color of the circle does not change even if I touch it.

mousePressed

function setup() {
  createCanvas(100, 100);

  // Style the circle.
  fill('orange');
  stroke('royalblue');
  strokeWeight(10);

  describe(
    'An orange circle with a thick, blue border drawn on a gray background. When the user presses and holds the mouse, the border becomes thin and pink. When the user releases the mouse, the border becomes thicker and changes color to blue.'
  );
}

function draw() {
  background(220);

  // Draw the circle.
  circle(50, 50, 20);
}

// Set the stroke color and weight as soon as the user clicks.
function mousePressed() {
  stroke('deeppink');
  strokeWeight(3);
}

// Set the stroke and fill colors as soon as the user releases
// the mouse.
function mouseReleased() {
  stroke('royalblue');

  // This is never visible because fill() is called
  // in mouseClicked() which runs immediately after
  // mouseReleased();
  fill('limegreen');
}

// Set the fill color and stroke weight after
// mousePressed() and mouseReleased() are called.
function mouseClicked() {
  fill('orange');
  strokeWeight(10);
}

Of course, if you expect it to work with touch operations, it is better to prepare functions for touch operations, touchStarted(), etc.
However, the reference says that mousePressed() acts as a replacement for touchStarted(), if touchStarted() is not declared, so this behavior feels contrary to that description. That's the reason.
If it is to truly function as a replacement, the timing of its activation must be appropriate.

However, as shown in the example above, you can avoid this problem by using touchStarted() and touchEnded() instead of mousePressed() and mouseReleased(). In fact, if you write it like this, you will get the intended behavior whether you use the mouse or touch operation.

In other words, I mean we should give up on expecting mousePressed() and mouseReleased() function as replacement of functions for touch operations.

This has caused some sketches I've written in the past to no longer work with touch. But it's not difficult to rewrite. It may not be necessary to fix this bug.

However, since there is definitely a behavior problem, we have decided to treat this as a bug.

I leave it up to the contributors to decide what to do about this situation.

[inaridarkfox4231]

I have a library for Interaction myself, and if I use that, I don't have any unnatural behavior like firing mouse down event after a touch end event, so I don't have any particular problems in that sense. I raised this issue for users who cannot help themselves.

interaction test

In other words, I'm not in trouble, so I don't really have a problem if this issue continues to go unnoticed.
If the cause is environmental and this phenomenon only occurs on my Android mobile phone or my stylus pen, then it's even less of a problem. But I can't judge that, so I've raised an issue.

[davepagurek]

Probably relevant, this PR is I think the last one that changed code related to this: #6738

For anyone attempting to write a fix, we should also double check that we aren't accidentally reintroducing the issue this was addressing with events double-firing.

[inaridarkfox4231]

One of the problems is that the reference for mousePressed() is lying.
mousePressed

On touchscreen devices, mousePressed() will run when a user’s touch begins
if touchStarted() isn’t declared.

In reality, the contents of mousePressed() are not executed at the moment of touch.
I think one solution would be to clearly state that in the reference. If you want it to work with touch operations, it is a reasonable decision to use touchStarted().

Another solution is to let it run (i.e. run the mousePressed() with touch start operation) by fix the current logic. Additionally, we need to make sure that previously resolved bugs don't come back, but I don't know what to do.

[inaridarkfox4231]

Here's an idea.
First, set the base to 1.9.0 and make the following changes:

p5js/src/core/main

    this._events = {
      // keep track of user-events for unregistering later
      mousemove: null,
      mousedown: null,
      mouseup: null,
      dragend: null,
      dragover: null,
      click: null,
      dblclick: null,
      mouseover: null,
      mouseout: null,
      keydown: null,
      keyup: null,
      keypress: null,
      touchstart: null,
      touchmove: null,
      touchend: null,
      resize: null,
      blur: null
    };
    this._millisStart = -1;
    this._recording = false;

    this._touchFlag = false; // flag for touch

Set this._touchFlag value to true at touchstart().

touchstart

p5.prototype._ontouchstart = function(e) {
    /* ~~~~~~ (Omit intermediate code) ~~~~~~ */

    this._touchFlag = true;
  }
};

mousedown() and mouseup() events will not be executed if this flag is set.
Also, set this._touchFlag to false at the end of the mouseup() event.

mousedown

p5.prototype._onmousedown = function(e) {
  const context = this._isGlobal ? window : this;
  let executeDefault;
  this._setProperty('mouseIsPressed', true);
  this._setMouseButton(e);
  this._updateNextMouseCoords(e);

  if (this._touchFlag) {
    return;
  }

  /* ~~~~~~ (Omit intermediate code) ~~~~~~ */

};

mouseup

p5.prototype._onmouseup = function(e) {
  const context = this._isGlobal ? window : this;
  let executeDefault;
  this._setProperty('mouseIsPressed', false);

  if (this._touchFlag) {
    this._touchFlag = false;
    return;
  }

  /* ~~~~~~ (Omit intermediate code) ~~~~~~ */

};

In addition to this, apparently the mouse event on end of touch is not fired when touchmove() is executed. Therefore, set this._touchFlag to false in the touchmove event.
touchmove

p5.prototype._ontouchmove = function(e) {
    this._touchFlag = false;

    /* ~~~~~~ (Omit intermediate code) ~~~~~~ */
};

I have now confirmed that there is no problem with the behavior with the mouse, stylus pen, and touch.
DEMO

I don't know if this will work or not. Bugs that are beyond our imagination may occur. Please consider this as a starting point only.

[inaridarkfox4231]

There is a Safari condition in the if statement, but it seems that because of this condition, the intended behavior is not achieved in Firefox. However, I don't know if this is correct.

before:

} else if (navigator.userAgent.toLowerCase().includes('safari') && typeof context.touchStarted === 'function') {

/* ~~~~~ */

} else if (navigator.userAgent.toLowerCase().includes('safari') && typeof context.mousePressed === 'function') {

after:

} else if (typeof context.touchStarted === 'function') {

/* ~~~~~ */

} else if (typeof context.mousePressed === 'function') {

As For Firefox, when the Safari condition was specified, the mouse operation worked as intended in the case of mouse only, and the touch operation worked as intended in the case of touch only. When both were specified, it worked without any problems.

[inaridarkfox4231]

It looks like there will be no problem if we describe both mouse and touch behavior like this, without having to change the current logic, but...
If you want to accept operations from both a mouse and a stylus, the current logic will cause problems.

2024-09-20.09-35-23.mp4

function setup() {
  createCanvas(400, 400);
}

function draw() {
  background(220);
  if(mouseIsPressed){
    console.log("is");
    circle(mouseX, mouseY, 20);
  }
}

function mousePressed(){
  console.log("mousePressed_"+frameCount);
}

function mouseReleased(){
  console.log("mouseReleased_"+frameCount);
}

function touchStarted(){
  console.log("touchStarted_"+frameCount);
}

function touchEnded(){
  console.log("touchEnded_"+frameCount);
}

If you use the mouse, then the stylus, then the mouse again, mousePressed() and mouseReleased() no longer work.
Of course, the previous version of 1.9.0 was not without problems. When using a stylus, mousePressed() and mouseReleased() are called.
The flag that rejects mouse operations cannot be turned off by the user, so there is no way to deal with this situation by writing external code.

In the demo I prepared, all of these issues are resolved.
DEMO

[inaridarkfox4231]

That's all for my suggestion. Since the content seems to overlap with #7195, so I will close this issue.

[diyaayay]

I tried handling this whole situation with pointer event onpointerdown, and onpointerup. Pointer events work for all types of input (mouse, touch, pen, etc.), and best of all they include a few additional features and methods that make doing complex interactions incredibly easy.

2024-11-14.18-12-56.online-video-cutter.com.1.mp4

I feel pointer event could be a possible solution but that might mean revamping the whole mouse and touch system to pointers in p5.js
This depends on how the maintainers would like to approach this issue.
@davepagurek @limzykenneth @inaridarkfox4231

[limzykenneth]

I think pointer event is a good starting point. One thing to consider is that if p5.js function such as touchStarted() and touchEnded(), which is analogous to mousePressed() and mouseReleased(), let's keep only mousePressed() and mouseReleased() in 2.0.

The only touch specific interface to keep around will probably just be touches for multitouch interface.