airo-camera-toolkit/notebooks/rerun_tutorial.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Rerun RGBD camera example\n",
    "This example shows how to use [rerun](https://www.rerun.io/) for logging images, image annotations, point clouds, transforms between elements in the world, time-series in a way that you might want to do for a robotic manipulation setup. Make sure you have a **Realsense or ZED2I camera connected**.\n",
    "\n",
    "\n",
    "Rerun has more features such as logging meshes, logging 3D bboxes, URDFs (in process). Check the docs to learn more. \n",
    "Also note that this tool is still very 'young', it became public only mid february 2023. There are many issues that need to be solved. The team is very responsive on github so shoot if you find issues etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-07-13T13:02:19.850842Z",
     "iopub.status.busy": "2023-07-13T13:02:19.850593Z",
     "iopub.status.idle": "2023-07-13T13:02:20.143617Z",
     "shell.execute_reply": "2023-07-13T13:02:20.142401Z"
    }
   },
   "outputs": [],
   "source": [
    "import rerun as rr\n",
    "#autoreload\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# start rerun. If the UI is already running, it will connect to it. Otherwise it will start a new UI and connect to it.\n",
    "# you can also start rerun using `python -m rerun`.\n",
    "rr.init(\"test\", spawn=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from airo_camera_toolkit.cameras.realsense.realsense import Realsense\n",
    "\n",
    "realsense = Realsense()\n",
    "camera = realsense"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from airo_camera_toolkit.cameras.zed.zed2i import Zed2i\n",
    "\n",
    "zed = Zed2i(depth_mode=Zed2i.NEURAL_DEPTH_MODE, resolution=Zed2i.RESOLUTION_2K)\n",
    "camera = zed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "point_cloud = camera.get_colored_point_cloud()\n",
    "rgb = camera.get_rgb_image()\n",
    "depth = camera.get_depth_image()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# log the colored point_cloud to the UI\n",
    "rr.log(\"world/camera1/point_cloud\", rr.Points3D(positions=point_cloud.points, colors=point_cloud.colors))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "point_cloud.colors[3156]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# log the rgb and depth images to the UI in a new image/camera1 space, this will automatically become a 2D image viewer.\n",
    "rr.log(\"image/camera1/rgb\", rr.Image(rgb))\n",
    "rr.log(\"image/camera1/depth\", rr.Image(depth))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# log a dummy 2D bbox\n",
    "rr.log(\"image/camera1/rect\", rr.Boxes2D(mins=[20, 100], sizes=[300, 500]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# log some dummy keypoints and attach labels\n",
    "rr.log(\n",
    "    \"image/camera1/keypoints\",\n",
    "    rr.Points2D([[600, 500], [400, 500]], keypoint_ids=[0, 1], radii=20, labels=[\"keypoint1\", \"keypoint2\"]),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# log dummy camera extrinsics from the world space.\n",
    "# This specifies the pose of camera in world.\n",
    "translation = [0, 0.5, 0.7]\n",
    "rotation = [-0.707, 0, 0, 0.707]  # scalar-last!\n",
    "# rr.log_rigid3(\"world/camera1\", parent_from_child=(translation, rotation))\n",
    "rr.log(\"world/camera1\", rr.Transform3D(translation=translation, rotation=rotation))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# log the actual camera intrinsics, to create a pinhole camera model in the UI.\n",
    "rr.log(\"world/camera1/rgb\", rr.Pinhole(image_from_camera=camera.intrinsics_matrix(), resolution=camera.resolution))\n",
    "rr.log(\"world/camera1/rgb\", rr.Image(rgb))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set up the 'view' of the 3D world space. This is for convenience so that rerun can sensible starting orientations for the spaces.\n",
    "rr.log(\"world\", rr.ViewCoordinates.RIGHT_HAND_Z_UP, timeless=True)\n",
    "rr.log(\"world/camera1\", rr.ViewCoordinates.RDF, timeless=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# log some more data\n",
    "for _ in range(5):\n",
    "    point_cloud = camera.get_colored_point_cloud()\n",
    "    rr.log(\"world/camera1/point_cloud\", rr.Points3D(positions=point_cloud.points, colors=point_cloud.colors))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# log time-series (e.g. F/T sensor)\n",
    "for i in range(100):\n",
    "    rr.log(\"world/robot/force\", rr.TimeSeriesScalar(i))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "execution": {
   "allow_errors": true
  },
  "kernelspec": {
   "display_name": "airo-mono",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  },
  "vscode": {
   "interpreter": {
    "hash": "61c61de08031e8c1b5d1a9d2c1c853908e3c2df123d111707e8338c45268ee37"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}