Unable to Track Objects with ArUco Tags with sksurgerybard.py and configuration_val.json file

[rohanscryenx]

I was tring to follow the documentation mentioned in the doc folder to replicate the results and was able to complete the first teo steps successfully. In the third step i.e. camera calibration using the chessboard pattern image when executing the following command and pressing 'c' key to capture and 'q' key to quit, the program doesn't responds.

python bardVideoCalibration.py -c config/video_calib_chessboard.json

Moving Forward onto the next step i.e. Tracking Objects with ArUco Tags and running the following command to track the reference marker

python sksurgerybard.py --config config/reference_only.json

The above code gives an error as the reference_only.json file doesn't exists in the config folder. So I tried to use configuration_val.json file instead of reference_only.json from the config folder using the below command

python sksurgerybard.py --config config/configuration_val.json

The above code executes and I point my camera to the image attached below but doesn't show any spherical markers as mentioned in documentation.

Please provide the exact steps to replicate the results of the repo as mentioned in the doc folder. And the documentation is not updated with the recent changes in other folders.

[thompson318]

Hello, sorry for the slow response, I've just got back from holiday. Thank you for raising these issues, I'll go through and try and sort them out.

I've just checked your first issue "python bardVideoCalibration.py" and can't replicate it. My first thought is that maybe the BARD window doesn't have the focus so pressing 'c' and 'q' doesn't get captured by BARD. Can you please check this by mouse clicking on the window showing the image captured by your webcam and trying again.

Can you also please confirm what version of BARD you're using by running "python bardVideoCalibration.py --version" and also what operating system you're using (Linux, Windows, Mac?).

[rohang1411]

Hi, Thanks for replying. I was able to solve the aforementioned issue and successfully replicated the steps mentioned in the documentation by downloading the source code from latest release.

I want to overlay a custom model(ex. a heart model) instead of the prostate model used in the code. I tried replacing the prostate model with a heart model but faced some errors. I'm attaching a screenshot of the same below for your reference. I just replaced the FullPelvis.vtk file with a vtk file of heart in scikit-surgerybard\data\PelvisPhantom folder and kept the name of the file same.

I'm using sksurgerycalibration version 0.1.9, which I checked through the command you provided. And I'm using Windows.

If you could please provide me the steps to use a custom 3D model with the code, it would be immensely helpful. Thanks in advance.

[thompson318]

Hello, that's great that you've got it running. I've made a few small changes to fix the documentation, thanks for pointing it out.

Looking at your output it says "Cannot read data set type: structured points". So I assume your heart data is vtk structured points not polydata, i.e. it doesn't contain any polygons for BARD to draw. Are you able to convert your point data into a surface model?

[rohang1411]

Hey, thanks for replying. I know it has been a long time but I was working on a different project for the past few months. Now I'm back on it. I'm trying to overlay a different model than the pelvis phantom provided. Here's a link to the model I was testing with

Link - http://www.cistib.org/cistib_shf/downloads/PCAmodel_full_phaseOne_90pct_proc05it.rar

But it gives an error when running the below command.

python sksurgerybard.py --config config/reference_with_model.json

Error -

I renamed the model to the model provided in the repo but still there was the error. I think it's different kind of .vtk file.

Actually I'm kind of new to 3D model stuff and don't have a lot of experience so can you please guide me how can I use a custom model or how can I create a similar kind of model on my side to test it. I want to build something similar but with a heart model, and I really think that this repo can help me a lot. It's just that I'm stuck. Thanks in advance.

[thompson318]

Your model worked for me. I unzipped it to PCAmodel....vtp and put it in a new directory. Then I edited the config file "reference_with_model.json" so the the entry "models_dir" points to that directory. Looking at your output it looks like you still have FullPelvis.vtk in your model directory directory.
Did you remane PCAmodel....vtp to FullPelvis.vtk? You shouldn't need to rename the model file. Just put it in the right directory.

[rohang1411]

Yeah I was renaming my model to the same name. But now I followed your steps and it works. Thanks for helping. It's just that ihe model renedered is very small, so I'm trying to find another model. Is there any way to increase model size via the repo code only? For the next few days I'll be working on this repo only, trying to understand the code, if I'll face any error I'll post it here. Thanks for all the help.

[thompson318]

I think the safest thing to do is to rescale your model so that it is at the correct anatomical size. By default everything is in mm. Alternatively, if you want to experiment you could try changing the "reference_to_model.txt" file. This defines the affine transform between the reference markers you're tracking and the displayed model. Most of the examples are simple rigid transforms (3 rotations and 3 translations) however I think it might be possible to include a scaling component in the matrix too.

You could start with a simple 10x scale matrix like:
10.0 0.0 0.0 0.0
0.0 10.0 0.0 0.0
0.0 0.0 10.0 0.0
0.0 0.0 0.0 1.0

and see what happens.

[rohang1411]

Okay Thanks. I'll be changing my models according to my requirement. Now I'm trying to understand the code, exactly how everything is working. as of now I'm trying to work out the code on my own. Is there any specific code walkthrough or flowchart from where I can get some help regarding the code.

[thompson318]

Hello, no I'm afraid there isn't really a code walkthrough at the moment. There's the documentation which includes a tutorial on how to get a system up and running, but the code walk through section is incomplete. For the application itself, everything starts at sksurgerybard/widgets/bard_overlay_app.py and is then delegated to other modules which you can hopefully follow through the import statements at the start of bard_overlay_app.py. Feel free to ask questions.