Our innovation has two parts in it, one is processing the live video stream into the computer and another one is through controlling the Robotic Arm using microcontroller (we are using Arduino mega 2560). Now these two parts are inter-related to each other. In the first part of our program, (which we did in PROCESSING IDE software) we process the live video feed, streamed by the camera which is fixed at the top of the working area. Our program then analyses the video stream or in other words our program walks through each & every pixel within its proximity and then further compares the RGB value of that pixels with the already set RGB value. Now the question arises, how do we actually give the RGB value of our own preference? That is just by clicking on any pixel of the processed video in the computer for which our program starts searching for the required RGB value of that pixel. When it finds the matching pixel at the first instance, it draws a rectangle on that pixel showing that it is detected. It further keeps on searching for the same pixel within its threshold range, and if it finds more of the similar RGB valued pixel within its range of 50 pixels (because 50 pixels is the distance threshold, it is adjustable if required) adjacent to the first matching pixel then it again repeats the same procedure of drawing a rectangle around the detected area, and this process is called as blob detection. So hence it can find multiple objects with same colour placed at minimum 50 pixels distance (if less than 50 pixels it takes two different but same coloured object as one object).
It not only finds the objects of same colour (colour preference is given by us), but also tracks that coloured object throughout the capturing area which is captured by the camera.
The Colour threshold is another important aspect of our program. In our program red colour is denoted by (255, 0, 0) means (RED, GREEN, BLUE). It means that for denoting red, RED value must be equal to 255 (which is a 8-bit data 11111111) and others should remain zero as it is. But in real life camera sends a 8-bit data for red which may be (255, 10, 15) meaning green and blue value may or may not be equal to zero, so certainly there is a tolerance, hence we have set that tolerance (positive tolerance) value to 25, i.e. if camera sends (255, 30, 35) as a colour information of any pixel then it does not consider that pixel as a red. Also it may happen that for a red pixel RED value may not be exactly 255, it can obviously be something like 250/245 or so , hence for this we had to set this tolerance (negative tolerance) value to 25 too. If camera sends a RED value of a red pixel as 220, then it does not consider that pixel as red.
This entire process is same for any coloured pixel on which we had to click for our desired colour detection.
Now coming to the hardware part, we have used Arduino mega 2560 for controlling the robotic arm abbreviated as “S.A.R.A.” (SEMI-AUTONOMOUS-ROBOTIC-ARM). It has 3 Degrees of Freedom (3 axis) containing base joint (SERVO 1), shoulder joint (SERVO 2 & 3), elbow joint (SERVO 4) and a gripper consisting of two fingers. All the motors are controlled by PWM signal which is sent by Arduino Mega 2560.
We have made S.A.R.A. in such a way so that it can move only a predefined path i.e. point A to point B. Point A is the point from where the robotic arm picks the object and Point B is the point where the robotic arm places the object.
Hence we have used computer vision technology for the implementation, which designates Point A as an action area for computer vision, meaning that our program for computer vision scans the entire captured area which furthermore starts tracking the specific coloured object and when it reaches to Point A position, our program for computer vision sends a signal to the Arduino Mega 2560 for removing the object from there and placing it to the Point B.
We have programmed Arduino Mega in such a way that whenever it receives a signal from our computer vision program, only then it runs the program which is burned in it(pick and place action) & further send PWM signals to the respective servo motors. Any other objects of different colour (other than from our chosen colour) reaches to Point A but “S.A.R.A.” does not react to that since that blob isn’t recognised now as of before.