Support for reMarkable Paper Pro #117
Comments
|
Hello; I would if I had a Rmpp, but to honnest, I don't plan buying one. Maybe if a friend buy one, and I can put my hands on it I could evaluate the work to do to make it compatible. |
|
Hi @owulveryck, my RMPP is arriving next week (Sep 30). I'd like to help you access the tablet -- remotely through SSH, or by giving you a filesystem/memory dump -- to update goMarkableStream. We have overlapping work, since I will need to get RMPP screenshots working in RCU, and both of our projects would benefit by our working together. Please send me an email if you'd like to set something up. |
|
I will get mine in a few days and I was literally planning to build something like this (in golang as well), will be happy to work / contribute on the rmPP support if necessary. |
|
Hi @davisremmel @owulveryck had you have a chance to sync on that and see if this would work ? |
|
Hi, just got the Paper Pro. There is no curl: But there is wget, but: :-( Can I help you in some way to make this project works on Pro? |
|
Addendum, copying binary 0.8.5 via SCP, then running, I get: |
I got the same issue (downloading the binary and the scp it to RMPP). I think it may be solved recompiling the application for the specific SoC. However the application is designed for the form factor of RM2 (resolution is hard coded in the first 2 rows here: https://github.com/owulveryck/goMarkableStream/blob/main/client/main.js). I think may be solved at build time extracting this in a configuration and then creating a build specific for each platform, or at runtime, if possible, reading the model type and modifying configuration according to this. I don't know Go Language but the server side main function is easily readable (https://github.com/owulveryck/goMarkableStream/blob/main/http.go): there is a middleware that streams the screen of the Remarkable via (I think) Websockets. This code should be reusable on the RMPP but I suppose that you need to check if the low libraries are present and usable also in the RMPP. I don't have an idea of what are the steps to rebuild the app, it should be useful a tutorial or similar documentation, a little more than this one: https://remarkable.guide/devel/toolchains.html#official-toolchain |
|
The application is in Go, therefore you can cross compile it on your laptop (one of the reason I choose Go). If you want to play, I hope that the articles on my blog can help you get a picture first and then expand it to the application. Good luck to you. |
|
Hi, how crosscompile for Pro? I'm a Java guy, I don't know how to get binary platform and how to compile. |
Thank you for your reply. I waited to answer because I needed to create a Linux VM on my Macbook. The command I used is the one you write in the Readme: Then I changed the architecture to arm64 And this time it starts (so remember Paper Pro is a 64 bit architecture), but of course if an app starts it does not automatically mean that it works. :) "open ./testdata/full_memory_region.raw: no such file or directory" So i "touched" this file. Now the terminal output does not give an error, but still showing nothing in the browser window.
So I am stalled at this point. I don't understand where is the error, maybe it si related to the full_memory_region.raw (I still didn't analyzed where is located in your code and why it does not create automatically this file) or maybe there is an issue in the /stream endpoint (how can I debug it? Which tool did you used?) or there is something else that not works. |
|
Nice job do far. Maybe we could arrange a Zoom/Meet call for 30 minutes. What is your Time zone? |
Sorry for the late answer but I had to work much last week. First, after analysing code I understood that the app was trying to access the file "./testdata/full_memory_region.raw" because the source fb_arm.go is not valid for architecture arm64 and the compiler defaulted to fb.go (which have a dummy method that I suppose you used this for testing on your PC). Because I don't know what is the right filename (not fb_arm64.go) I simply copied the code of the function "GetFileAndPointer" in fb.go (a dirty solution I know :) ) and now it works. No, it's not really working. :)
gives nothing. After some looking around I tried /dev/dri/card0, I think it's probably wrong. I tried via command line to generate an image raw with the command (I am using you guide, the size is 2160x1620): but for now I don't find a way to convert the raw to an image.
I hope you can help me, also with some suggestions. PS: I can share all the result of xochitl memory segments from the command:
|
|
@cristiannobili the framebuffer has some padding at the end of each row of the memory, the amount of bytes usable for each raw is called stride, I guess the amount of "padding" is different between the RM2 and RMPP If you find the fb device under Also because now there are colors involved you probably need to check the |
|
@danielealbano |
|
Hi. i am also interested to get this running on a rmPP. Thanks for all your effort so far... @cristiannobili when you did dd, what memory address did you actually use? one of the card0 ? regardless which address-range i try, dd results immediately in I/O error with 0 bytes read. However. thanks to your image.raw. import sys
import cv2
import numpy as np
if sys.stdin.isatty():
print ("i need data to convert")
sys.exit(0)
buf=sys.stdin.buffer.read()
array=np.frombuffer(buf,dtype=np.uint8)
buf_len=len(buf)
skip=0
skip_line=0
width=2154
lines=(buf_len-skip)/(width+skip_line)
height=int(lines)
out=[]
array=np.empty((width,height),dtype=np.uint8)
pos=skip
for y in range(height):
for x in range(width):
pos+=1
pix=buf[pos]
out.append(pix)
pos+=skip_line
out=bytearray(out)
array=np.frombuffer(out,dtype=np.uint8)
array=np.reshape(array,(width,height))
cv2.imwrite("out.png",array)When running it cat image.raw | python3 raw_to_png.pyi get the following output |
|
So the image is there. Great job. :)
and you get the raw file. The next steps to do are: a) modify pointer.go in order to obtain the right address of memory area. Something like this: the "magic number" is obtained subtracting the last address minus the first (the memory size of framebuffer is constant). Tested it, it's the right value. b) convert the image inside the application using go language. |
|
Many thanks.. it worked for me as well. i think we have 3 challenges left: Challenge #1:the go issue you mentioned (I never worked in go so far, hopefully i find some time drilling me in) Challenge #2:we are missing the color information. below you see an image that i grabbed from my rmpro. I wrote all the colors (sorry for my handwriting :-)) in their respective colors. In the memory area i have the following value counts:
i would have expected somehow different values to indicate some color infos.. Challenge #3:above the "grey" i wrote "black". (see photo) but this is seemingly missing in the memory-dump... instead we see white space
|
|
Agree. But I think that first we have to get goMarkable working even if only in bw. I isolated the component that does the job: I don't studied yet how to resolve this ( I work on this project only at late evening) but playing with parameters I suspect that there is another issue: when I change page the webpage does not refresh, or better, when I create a new page it does not refresh. Maybe it's a gesture not recognized. I observed also another phenomenon. When you approach the pen without touching the screen, the app receives some event and start refreshing the screen. I think this behavior could be specific of Paper Pro, because here there is an active stylus with a battery (maybe in future Remarkable could activate new functions, like laser pointer or similar). |
|
Wohhh congratulations to all of you. I say again that I am willing to offer help via a zoom/meet call of one hour or so. (@cristiannobili no worries, with your english and my italian, we may find a way to understand each others). Regarding the last comment, I have implemented a mechanism of idleness to spare bandwidth and battery. The stream stops after 2 seconds or so, and start again when there is an activity with the pen. This must be the reason. |
|
The RLE is a compression mechanism, maybe for a first attempt, you can try to bypass it. To do so, you can change this line: goMarkableStream/internal/stream/handler.go Line 107 in 087878d w instead of the rlewriter (there are both implementation of io.Writer that should do the trick).
Then, you must change the web component as well to write the code directly into the image without the RLE decoding process (you need to modify this: ) |
|
Has anyone got anywhere with this? I would love to help If possible :) |
|
I had one hour, so I played with This go code generates a picture: package main
import (
"encoding/binary"
"image"
"image/png"
"log"
"os"
)
func main() {
palette := make(map[uint8]int64)
spectre := make(map[uint8]int64)
testdata := "../testdata/image.raw"
// testdata := "../testdata/multi.raw"
stats, _ := os.Stat(testdata)
f, err := os.Open(testdata)
if err != nil {
log.Fatal(err)
}
defer f.Close()
backend := make([]uint8, stats.Size())
log.Printf("backend is %v bytes", len(backend))
n, err := f.ReadAt(backend, 0)
if err != nil {
log.Fatal(err)
}
log.Printf("read %v bytes from the file", n)
picture := backend[:2160*1620]
boundaries := image.Rectangle{
Min: image.Point{
X: 0,
Y: 0,
},
Max: image.Point{
Y: 2154,
X: 1624,
},
}
img := image.NewGray(boundaries)
for i := 0; i < len(img.Pix); i++ {
if i*2+2 > len(picture) {
break // Avoid out-of-bounds access
}
img.Pix[i] = uint8(binary.LittleEndian.Uint16(picture[i*2 : i*2+2]))
}
for i := 0; i < len(picture); i += 2 {
spectre[picture[i]]++
}
for _, v := range img.Pix {
palette[v]++
}
log.Println(spectre)
log.Println(palette)
png.Encode(os.Stdout, img)
}BUT:
I will have a closer look if I have time. Thank you all for your work |
|
Can one of you provide an Thank you |
|
sorry for my radio-silence.. with the help from @cristiannobili i think i found the memory area: please find attached the complete 22 MB raw dump of the colorful image attached which i posted earlier. I have shared an PNG of the the first 3499200 bytes of that memory area. However the black writing was missing. I believe this is somehow a color-mask used internally to show all non-black colors. Unfortunately i have not the time to scan through the complete 22 MB to see if we somehow find also the color-coded image. I am deeply under water at the moment, sorry that i cannot do more analysis right now. |
|
at the position 8605705 into the colorful.raw from above i have found really interesting stuff... |
|
ok, now we are really close... i think the only missing piece is if the skipping of 8605705 is always the same number.. import sys
import cv2
import numpy as np
import pprint as pp
if sys.stdin.isatty():
print ("i need data to convert")
sys.exit(0)
buf=sys.stdin.buffer.read()
skip=0
width=2154
height=1624
skip=8605705
array=np.empty((width,height,3),dtype=np.uint8)
pos=skip
for x in range(width):
for y in range(height):
array[x,y,0]=buf[pos+3]
array[x,y,1]=buf[pos]
array[x,y,2]=buf[pos+1]
pos+=4
cv2.imwrite("out.png",array)when i run the colorful.raw through the above python code i get the results in this png |
|
I think, i have decoded the memory footprint. The With the following python script we should be able to calculate the correct offset of the screenbuffer and grab the screen import sys
import cv2
import numpy as np
if sys.stdin.isatty():
print ("i need data to convert")
sys.exit(0)
buf=sys.stdin.buffer.read()
offset=0
width=2154
height=1624
length=2
# we seek for the chunk that has room for 2154*1624 4-byte pixels
while length<width*height*4:
# for whatever reason we need to substract 2 bytes from the length
offset+=length-2
# at pos 8-16 it contains the chunk (header + payload length)
header=buf[offset+8:offset+16]
length=int.from_bytes(header,"little")
# we found the screen
# dump it to png
offset+=16 # skip header
array=np.empty((width,height,3),dtype=np.uint8)
for x in range(width):
for y in range(height):
array[x,y,0]=buf[offset+2]
array[x,y,1]=buf[offset]
array[x,y,2]=buf[offset+1]
offset+=4
cv2.imwrite("screen.png",array) |
The length is most likely is a int16 or uint16 which doesn't include itself, what I find curious though is that the actual data in memory is in the BGR format instead of RGB556 or similar, which is fairly common for these kind of displays. Although it might not be there visible, can you check the content of /sys/class/drm/card0/{output} (or similar)? The {output} is the output name used internally, not sure what's going to be called, perhaps card0-DSI-1 or perhaps something else instead of DSI as the screen most likely will be using a different kind of connector. To "protect" the access, the boot sequence might (a) espose the device within the initrd, let X / wayland start, and then switch to root or (b) let X / wayland start and then remove the physical inode, which wouldn't impact the running applications as they have the fd already opened ... or perhaps they are using a patched kernel to make it invisible unless you are a special user. |
|
Here is the corresponding Go code to extract a picture from the package main
import (
"bytes"
"encoding/binary"
"fmt"
"image"
"image/color"
"image/png"
"io"
"log"
"os"
)
func main() {
// Read input data
buf, err := io.ReadAll(os.Stdin)
if err != nil {
fmt.Println("Error reading input:", err)
os.Exit(1)
}
width, height := 2154, 1624
offset := 0
length := 2
// Find the chunk with enough room for pixels
// Recherche d'un morceau de données suffisamment grand pour contenir les pixels
for length < width*height*4 {
// Avancer l'offset pour atteindre le prochain morceau
offset += length - 2
// Extraire l'en-tête de 8 octets à partir de l'offset
header := buf[offset+8 : offset+16]
// Utiliser binary.Read pour lire un uint32 à partir de l'en-tête
var length32 uint32
err := binary.Read(bytes.NewReader(header[:4]), binary.LittleEndian, &length32)
if err != nil {
log.Fatal(err) // Gérer l'erreur si la lecture échoue
}
length = int(length32)
}
// Skip header
offset += 16
// Create image
img := image.NewRGBA(image.Rect(0, 0, width, height))
// Extract and mirror pixel data horizontally
for x := 0; x < width; x++ {
for y := 0; y < height; y++ {
// Calculate the mirrored x-coordinate
mirroredX := width - x - 1
// Get the pixel index from the original image
idx := offset + (x*height+y)*4
// Set the pixel color at the mirrored x-coordinate
img.Set(mirroredX, y, color.RGBA{
R: buf[idx+2], // Red channel (from original image)
G: buf[idx], // Green channel (from original image)
B: buf[idx+1], // Blue channel (from original image)
A: 255, // Alpha channel (fully opaque)
})
}
}
if err := png.Encode(os.Stdout, img); err != nil {
fmt.Println("Error encoding PNG:", err)
os.Exit(1)
}
} |
|
i have posted a bug above... R=buf[offset]
G=buf[offset+1]
B=buf[offset+2]
A=buf[offset+3] # this would be my guess.. did not check
offset+=4@owulveryck why did you mirror the X axis? and isn't it sufficient to do the mirroring calculation in the outer X-loop prior the Y-loop? |
|
Why not mjpeg? With a quality ratio of 80/85, the single images would be fairly small and wouldn't really lose quality and long can be used for screenshot, to preserve more details Both can easily generated in go and potentially accelerated via NEON (although sending data from the neon simd registers to memory is slower than avx and unless the computation makes it worth it's better to avoid it). A alternative might be to generate a video stream going through webrtc (which can also be easily used via go) but I imagine this would be a major change. |
|
I give a short answer for mjpeg and co. That being said, I want to thank and congratulate all of you for the work so far. |
|
Did you get the chance the measure the power consumption for mjpeg? Additionally, for jpeg it's possible to use libjpeg-turbo, which supports NEON, and it's several times faster than libjpeg. |
|
I updated my branch, fixed the streaming, added a note that RLE does not work for Paper Pro, and for now added optional zstd compression in addition to gzip, so streaming is a bit faster. If you build it and then run this on the device, it should work: It's all a bit hacky, and I'm not sure that Remarkable 2 isn't broken since I can't test it. |
|
I will try to test it on RM2 Is there someone willing to handle the maintenance of the code for RMPP ? |
|
Congratulations to all of you. |
Thanks @owulveryck! I opened a PR, feel free to change whatever is needed. Again, not everything is working as expected on Paper Pro (I added at least a few things to the PR), so I'd really appreciate it if someone else could also test it 😄 |
|
Hi... i hacked together a small RLE adaptation for the rmppro screens. the idea:
considerations
examplescount not setwe read the follow values (1 * 2 bytes) count setwe read the follow values (2 * 2 bytes) When writing to output we simply explode the pixel from 15 bit to 32 (or 24) bit as needed. The colorful.raw is compressed by from 13992384 to 426408 (or compression ratio of 97%) python PoC of the RLE encoder and decoder# execute with
cat colorful.raw | python3 extract_screen_rle.py# will follow asap |
|
I merged all the code into the main branch and released Than you again for all your amazing work. Enjoy the end of the year. |
|
Works pretty well on my Paper Pro in some initial testing, with one minor caveat: attempting to run the process with an ePub or PDF file open produces I believe this is caused by the presence of the I've created a PR, #124, with a fix that should only return the correct |
|
Thank you; |
|
@AngleOSaxon had you considered sub-processing out to pgrep rather than walking a number of files in /proc ? |
No; I don't do a lot of Go or *nix work, and was aiming for the shortest path from current state to fixed. |
|
to each their own, but I'd suggest The command we'd like is Not that it matters but it's about twice as fast when checked via |
|
@Lewiscowles1986: Thanks so much for the suggestion! 😊 I’m aiming to make the software fully self-sufficient, which is why I prefer a pure Go implementation. If That said, this code only runs once at startup, so while I always appreciate optimizations, this one probably wouldn’t make much of a difference to the user experience. Thanks again for sharing your thoughts! 🚀 |
|
is there a tick list of to-do list we can attach to issue to see progress / opportunity for contribution, or is this complete? I see:
|
|
I just wanted to point out that the "Setup goMarkableStream as a systemd service" does not work for me on the RMPP. More specifically, the systemd service gets created but it does not "survive" a full reboot, meaning the file /etc/systemd/system/goMarkableStream.service no longer exists after a full reboot. Is there anything specific that needs to be done to render this file persistent over a reboot? Thanks in advance for any answers. |
? |
Hi @2Belette thanks for your feedback. Just a few questions before to make any mistake:
and after checking the volatile part it does not include the folder /etc/systemd/, which according to the mount info is already in rw mode, but the problem is that any modification I make there it does not seem to be persistent. |
|
The overlay volume is a virtual volume, used for example by containers as well, that allows you to write the changes to a different folder retaining the "original" content unchanged. If you notice, the "workdir" and the "upperdir" are both inside a "/var/volatile/..." folder which, as indicated by the name, is not persisted by OS, most likely it's mounted as a tmpfs or such. The umount -R does a recoursive umount on /etc and whatever other folder is mounted under /etc, from your grep the -R will do nothing as there are no volumes mounted under /etc/ apart from /etc itself. The first command will allow you to write on the root fs meanwhile the second command will drop the temporary filesystem behind /etc. Meanwhile I have never run these commands directly on the RMPP, most likely it's safe and you can just re-run the mount with the overlay fs using the same parameters indicated in mount after you finish with the changes (or just restart the RMPP). |
Yes. However, you may want to "undo" it after you've made your changes. The command for this would be
On the rMPP, the
What this means is, programs are able to write to files in the
After this, any files written under |
|
@danielealbano and @kg4zow thanks a lot for your quick and very comprehensive replies. At this point I just wonder if these commands which seem to be necessary to ensure that the service is added in a persistent way across reboots should also be included in the description "Setup goMarkableStream as a systemd service". This would help a lot people who are not familiar yet with the Remarkable ecosystem. |
|
I actually prefer having to SSH into the tablet and start it manually, rather than having it running in the background all the time. I have a shell script in root's home directory of my rM1 and rM2 tablets, which I use to print a list of URLs (iTerm les you click on URLs, easier than copy/pasting) and then run goMarkableStream with my preferred options. When I SSH in, I run The script looks like this: #!/bin/bash
PORT=2000
export RK_HTTPS="false"
########################################
# Print a list of URLs
IPS="$( /sbin/ip -4 a | grep '^ *inet ' | awk '{print $2}' | grep -v ^127 | sort -V )"
cat <<EOF
Visit one of the following URLs in a browser:
EOF
for IP in $IPS
do
echo " http://${IP%/*}:$PORT/"
done
########################################
# Run the command
cat <<EOF
Starting goMarkableStream
EOF
export RK_COMPRESSION="false"
export RK_SERVER_BIND_ADDR="0.0.0.0:$PORT"
./goMarkableStream --unsafeI don't know yet if I'm going to need to update it for the rMPP, if so I hope it'll be fairly simple. |
|
@kg4zow thanks for the info. As a matter of fact, I just tried your script and it woks fine also for the RMPP. I just had to add the export for the compression which is not supported yet on RMPP, that is
On a side note I noticed that sometimes when the RMPP comes back from sleeping (or other random situation) while trying to run the executable goMarkableStream it gets stuck before the binding of the port and there is no way to make it work until a full reboot of the device. Please note that, I did check if some other process was hanging on the port while goMarkableStream was trying to bind to it but this was not the case, in fact even trying to force another port with the environment variable RK_SERVER_BIND_ADDR would not fix this issue. As I said the only way to get it back to work is by forcing a reboot of the device. Another thing, I am not sure if this "normal" or perhaps just a consequence of the software being still experimental for the RMPP but there is a kind of "fixed" 1 sec delay (at least) between drawing a line and being able to visualize it remotely on the PC browser. |
|
@Andrea79 I found the rMpp slow too, but it got a lot faster (still < 10fps) over USB |
@Lewiscowles1986 thanks for your reply. Over USB is slightly faster (but frankly to me it is still lagging a little bit too much). I just wonder if this is only for the rMPP because the support is experimental or if it is like this for any rmX device. On a side note my rMPP seems quite slow in general with the latest firmware (this is how I got it so I have no experience with either previous firmware or previous rmX) and not very responsive to gesture control. I know this is off-topic (even though it could be related to the lag experienced for the goMarkableStream) but I wonder if this is true for anybody else |
|
What environment varibale options have you launched with? I think somewhere I read that it's about 10% of CPU to get one frame using the current method. It's copying from memory, then I'd presume doing some conversion to be suitable for streaming, so you / someone would need to profile that code, to understand where the bottlenecks are. If there are none, then maybe using ARM NEON if available could help; or a totally different approach. I thought I heard it's a quad-core CPU. |
|
Hi @Lewiscowles1986, at current I am running it with the following environment variables: I have tried several combinations, e.g., with/without https, with/without compressions), but I have not seen any particular difference in performance. Indeed, USB works better, but I like working through wireless. On their website, they mention the processor to be a 1.8 GHz quad-core Cortex-A53 But since I got the devices (3 days ago) running on firmware 3.16.2.3 this device has not seemed so responsive (I am not sure whether my expectations were biased, though). Let's see if things get any better with a firmware update in the future. |
I know the rMPP is brand new and most people (including myself) haven't received theirs yet, but I'm kinda surprised nobody has asked this yet.
Are you planning to update goMarkableStream to support the rMPP?
If so, you may want to add something to the repo's
README.mdfile saying this.The text was updated successfully, but these errors were encountered: