Charging slow on Werner

[gestom]

Hi,

since yesterday, Werner's charging rate is very low. The charging current is typically below 2A and is fluctuating wildly. The 'charger_status' field internal_error_flag is True and 'const_current_mode' is fluctuating between True and False. Battery level was about 77% when I checked.
We recorded the 'battery_status' for a few hours - the rosbag is available here: https://lcas.lincoln.ac.uk/owncloud/index.php/s/FbRBWMZnZQQjVcQ.

Can we do anything to figure out why is the charging slow ? normally it's about 10A.

Best,

Tom

[marc-hanheide]

@creuther as our valuable MetraLabs contact, could you forward this to the relevant people, please?

[creuther]

@marc-hanheide Of course, thanks for notifying me. Without knowing any details, I think the charger would need repair or even replacement so can you give me a c/o address so we can expedite the whole process a bit and get a new charger sent out? At least that is what I suspect will happen, I can only confirm it tomorrow morning though.

[marc-hanheide]

Thanks for coming back to us so quickly. If there's any diagnosis we can do, please let us know.

[gestom]

Maybe that the reason for slow charging is caused by the additionally installed PCs inside of the robot, which might cause the robot to overheat. We have never observed the charging slowdown with a robot that had it's skirt off. To test it, I let the other robot (LINDA) patrol 'naked' to deplete its batteries to 20%, put the robot skirt on, started the deployment systems, placed a digital thermometer inside of the robot (at the top part of the skirt area, approximately at the level of the PTU controller) and let the robot charge in a room with controlled 30 °C temperature. First, the robot would charge at 10 A, but as the temperature inside of it would increase, the charging current would drop to +-2A. After the battery was charged to 35%, I removed the skirt and set the room air condition to 19°C. The charging current started to slowly increase again, see the attached graphs.

[marc-hanheide]

Wow, thanks, guys. This is on Linda, so the question is if the effect is the same on Werner. How did you measure the temperature? Have we got any idea what the temperature profile is on Werner?
Also, have we got a chance to measure the temperature in the battery cells themselves? It would now be interesting to see if we start charging without skirt and at 19℃ and obtain a very different profile...

[creuther]

Great analysis, thanks for that! I have created a service ticket and have some diagnostics instructions for you.

First of all, we suggest flashing the latest firmware of the charger as it does improve the stability of the charging "arc", but obviously it won't solve the temperature dependency if that turns out to be the cause. We don't have the best track record with firmwares and usually believe in not changing running systems, but I personally feel very confident about our latest firmware bunch. It includes a couple of fixes, among others it noticeably improves the odometry stability (thanks to the great investigative work by @cburbridge). I will write an email about this to the STRANDS mailing list within the next hour.

As for diagnostics, it would be more helpful for us if you could create a MIRA tape of the /robot/charger/BatteryState channel (topic) while this phenomenon is happening. Doing that is fairly easy, you can just connect to the locally running MIRA instance (it should by default be port 1234) by using
miracenter -k 127.0.0.1:1234
and then see whether there is already a "Recorder" tab in the bottom center workspace area. If not, just add the view part by going "Window -> Show View -> Tape -> Recorder". In the widget, press the green "+" button and select the /robot/charger/BatteryState channel. After that you just need to press the red "Record" button and specify a filename, then it'll record the channel. Exactly the same as ROS bags, but readable by our diagnostics tools ;-) Also you need to press "Stop" when done for the tape to finish writing.

There is also a way to access the persistently stored errors of the charger, by using the RPC /robot/Robot.savePersistentErrors(std::string filename)
This can be called from miracenter using the RPC Console view (if not already added, add in the same way as above). This persistent error file, if there are any stored, would be helpful as well for the diagnostics.

[creuther]

Just in case, here is the file sent to the STRANDS mailing list.
scitos_firmware_upgrade_17032016.tar.gz

[gestom]

How did you measure the temperature?
Using the Temper1F sensor, see the metallic piece in front of the router.

Have we got any idea what the temperature profile is on Werner?
No. But hospitals might have some (contact less) thermometers and people who can use them. I am not sure it these work on robot skin as well.

Also, have we got a chance to measure the temperature in the battery cells themselves?
It would now be interesting to see if we start charging without skirt and at 19℃ and obtain a very different profile...
We might try. However, this might not be the case of the batteries only, but the other charging circuitry as well.

[gestom]

I am about to try the firmware upgrade. However, how do I backup the previous firmware ?

[marc-hanheide]

I recall haven't read somewhere "you can't", but I can't find it...

[gestom]

so we might end up with a broken system ?

[gestom]

I will do some outstanding experiments before then :)

[gestom]

OK, I did the firmware and run the robot around with and without the cover. Navigation was working fine and the charging current fluctuations were reduced. The average charging current was slightly higher than before the update, but one could still see that with the cover on, the charging took much longer time.

[marc-hanheide]

So, would you then recommend we update Henry? We have an easter break at the moment anyway...

[gestom]

Actually, I would not recommend it. Although the charging current is not fluctuating after the update, its average value did not change much and the overheating problem persists. Rather than updating firmware, I recommend upgrading hardware and cut some additional venting hole(s) in the robot shell. I am currently running another batch of tests with the lower skirt removed. This might give us a hint where to cut.

[gestom]

So, removal of the lower skirt also helps, so IMHO we might start trying to drill a venting venting hole in the rear part of the robot 'lower' skirt. The question is if Metralabs can provide us with a few replacement plastic parts of the robot's shell and for how much ?

[creuther]

I can find out for you, what exactly do you need? Which parts? How many of them? In the same colour as Werner or just some generic colour?

Besides, what I find a bit odd is that the problem arose so suddenly. I can see from your measurements that there is some correlation with temperature, but before it was charging fine despite the exact same situation (shell on) right? If so, then I wouldn't rule out a problem with the charger yet. If you could do the diagnostics I posted above, we might be able to find out more.

[gestom]

Maybe the problems are not observed that much because when not used in deployments, the robots were running half-naked. Moreover, I think that we observed the overheating problems last year as well, but this time it might be more severe. I have plenty of rosbags from the experiments, cannot they substitute the MIRA tape ? The topic is the same.

[gestom]

miracenter -k 127.0.0.1:1234 reports that it
'could not connect to the framework'

[creuther]

I had a look at the scitos_mira node, apparently the MIRA framework there doesn't start with a port open for incoming connections unless you specify the server_portvariable. But either way, you can also run a separate MIRA instance when the scitos_mira node is not running (as only one connection to the CAN bus is allowed).

miracenter SCITOSConfigs:etc/SCITOS-mapping.xml --variables robot=SCITOS-A5,canType=MLCAN

The reason we cannot easily use the ROS bags instead is that we have tools that read the tape files (and channels within) for analysis. I guess it would be possible to build a converter but as this can't be done generically, we have never put too much priority into it.

[gestom]

OK, the tape recorded by @Jailander is at labe.felk.cvut.cz/~tkrajnik/miratapebattery.tape. The persistent errors of the charger are at labe.felk.cvut.cz/~tkrajnik/persistent.errors. We will probably need one 'bottom' skirt and one 'back-middle' skirt with Linda-blue color.

[creuther]

Thanks, we are analyzing the tape and error files and will get back to you. As for the case parts / skirts, I talked to my boss about it.
Unfortunately we can only sell the large middle parts as a whole / together as they are commissioned in that way. So the two middle parts and the bottom part including paintwork would cost 750€ net altogether. Let me know whether you want a formal quote or how you want to continue. Otherwise I'll be back in touch about the charging irregularities.

[marc-hanheide]

OK, as this tape was on Linda, we shall create one from Werner as well.

[cdondrup]

I recorded a bit of data:
https://lcas.lincoln.ac.uk/owncloud/index.php/s/VwGbXIiOWIBqhk0

This contains 4 files:

• persistenterrors.txt which is the persistent error log
• a rosbag file which contains recordings of the charging topic while the robot docks and undocks
• charger.tape a mira tape of the robot being on the charger
• drop.tape a mira tape that hopefully shows the drop in current when docking. This was created by sending the robot to the charger and when it docked, immediately killing the scitos node and starting mira centre. It does not have the docking in it but should start with the high current we observed and then drop.

[marc-hanheide]

Here is the plot of the rosbag @cdondrup created:

Sorry, about the x axes... it's samples, so roughly it's second * 10.

I plotted the current (as before), and we see a similar behaviour. indeed we went through a number of docking and undocking cycles, where we always observed pretty much the same behaviour. After docking the charging started at a high rate (so negative current, red line, at about -8A, one such docking starts at 2362 in the plot). At that point we also see the Voltage (green line) to steeply go up, but interestingly slowly, and not directly with the current. At 3537 we see the dropping occurring when the charger seems to go give less power to the robot, until levelling out at about -3A from 4524 onwards. We have then repeated this procedure three times, quickly undocking and re-docking to see if it makes any difference in the speed it declines. One hypothesis was that if we charged before, the unit might still be warm and the decline in charging speed should be faster than at the beginning when the robot was away from the docking station for longer... this seems to be the case as indeed in the last cycle the "good" charging duration was only about half long as in he first cycle.

Note: This time it continued charging because we had a very low load on the robot, not runnng anything but the bare necessary things to run the system. On full load, those 3A would be eaten by the computer running stuff. That's what we currently see: The current is just about enough to power the processing, but not to charge. It would be perfectly enough if the higher charge we see at the beginning would be maintained.

[marc-hanheide]

So, @creuther is this helpful? @gestom, I think this all confirms what we were suspecting, right?

[marc-hanheide]

One last note... the voltage profile is a little surprising to me, which is why I also included it, maybe it help diagnosis.

[creuther]

@marc-hanheide @TobKoer We sent you a new charger today, I'll provide you with instructions on how to change it soon.

The additional information you provided is quite interesting as well, I forwarded it to our hardware guys. I'll keep you posted!

[marc-hanheide]

I forgot to add, that this was all measured when the robot was about 50% charged, so to make sure we don't end up in the kind of tickle charge when it is nearly full...

[cdondrup]

"tickle charge" hihi XD

[Jailander]

Yes the plate gets hot, with and without the skirt I imagine the effect is reduced when the upper skirt is off because of the improved heat transfer. but the plate is always moderately warm

[Jailander]

I'll check in a second, we need a thermal camera ;)

[Jailander]

Ok Linda has a full battery right now so the charger is cold, I'll let her discharge for an hour and plug her and see after a while, however since the lower skirt is not here it won't be as warm as in other cases.

[marc-hanheide]

We have a thermal camera, ask Greg.

[Jailander]

haha I know, just realised we could have used it before I made the comment, but right now it doesn't make sense since we don't have the skirt. But I'll ask Greg for it when we have the part back.

[creuther]

I have been doing some asking around and had a look at Bob but it seems while I was doing this there were quite a few new posts. Anyway, I'll just continue with what I started typing before ;)

@marc-hanheide Unfortunately the life-percentage is not really that reliable, so it would be more helpful to see voltage plotted against current. What kind of services were running during the night, and what kind of power draw do you except the robot to have due to them? If the robot was still drawing around 6A then the numbers seen might make sense.
Also, the charger per se should not need excessive cooling. The only time we have run into a temperature problem was with the HOBBIT project, which had a quite similar setup in terms of complexity and power consumption. The reasons there were heat build-up due to insufficient air circulation as well. So as @cdondrup also confirms that the problems do not happen with the skirt off, we have to improve the circulation / get rid of the heat build-up.

In terms of the derating flag, I have some pleasant news to deliver - there is yet another bug in the firmware at the moment that automatically sets the flag to true when the robot is charging. But heat will still most likely be the issue.

To see whether everything else in terms of charging is working correctly, I'd suggest again letting the robot charge with all computers shut down and then reading the current and voltage from the status LED display. It should read around -10A.

Bear in mind though that as soon the battery is full (voltage around 29V +-0.5V), the current will drop to 0 and then hover around small positive and negative currents.

But all in all it seems most likely that everything is in order and that the heat is the problem at the moment.

[Jailander]

@creuther in one of those charts @gestom made when we recorded the tape you could tell that the problem also happens with the skirt off just in a much lower scale, however, this could be happening on Linda because of the defective cell. Anyway solving the heat build up issue seems to improve charging.

[marc-hanheide]

Thanks @creuther, so given all this, have your MetraLabs guys any advice how to best get rid of the heat build up? What did you do in Hobbit?
Taking the skirt off is no option. I agree that what we see now is mostly consistent, and when it was cool, we saw indeed that 10 A - current_drawn = current_charge.

So, the charger is functioning as it should, the firmware apparently isn't? Can we set the derated to false somehow?

[creuther]

@marc-hanheide I'm waiting for their response on it. The firmware bug is only a "display" issue, the charger isn't necessarily going into derating mode but when it is on the charging, the derating flag is being shown as set regardless of the internal state.

Apparently it is possible to see the actual power draw of the charger when on the charging station, but this data is not yet exposed in the charger status channel / topic (it is certainly the first I've heard of it). It is nevertheless present in one of the CAN bus PDOs and should be viewable with moderate effort. I am testing it on Bob now, I'll report back when I'm done.

[bfalacerda]

can someone point me to the scripts to do the nice plots, so we leave betty charging tonight?

[marc-hanheide]

I don't have a script. I used

1. rosrun topic_tools throle messages /battery_state 1 /battery_state_throttled to limit the data recorded to once per second

2. rostopic echo -p /battery_state_throttled > batt.csv used to record this into a CSV file directly

3. downloaded this file to my laptop

4. grep -v "%" batt.csv | tr "," "\t" > batt.dat to convert it into something gnuplot understands

5. run gnuplot with the following command:

   plot "bat.dat" using (($3-1460397600000000000)/3600000000000-4):5 axes x1y1 title "Current [A]", "bat.dat" using (($3-1460397600000000000)/3600000000000-4):6 axes x1y2 title "lifetime" with lines
   

   where 1460397600000000000 is the UNIX epoch multiplied by 10e9 to get the starting time of the plot... and 3600000000000 is how long an hour is in nano seconds ;-)

Feel free to cast into a script!

[creuther]

So I tested the PDO stuff on Bob and it seems to work, even though it not that user-friendly. You have to get exclusive access to the CAN bus (i.e. every other MIRA instance accessing the CAN bus has to be killed) and then opening miracenter e.g. like this: miracenter SCITOSConfigs:etc/SCITOS-mapping.xml --variables robot=SCITOS-A5,canType=MLCAN

Then open the "CAN Tools View" by going "Window -> Show view -> CAN Tools View". You should see something like this:

PDO 0x183 and 0x184 are the interesting ones. Bytes 3 and 2 will indicate the voltage in mV and bytes 5 and 4 will indicate the current in mA. 0x183 is the battery voltage and current, and 0x184 is the charging station (ACS) voltage and current. So for example for PDO 0x184 we get voltage = 89ef hex = 35.3 dec (least signifcant byte first or whatever, anyway the second one needs to go first).

To get the overall current provided by the charger, you do
I_charger = (V_acs * I_acs / V_battery) * 0.95
And of course it holds that I_charger = I_draw + I_battery so we can see the power draw of the robot and its components as well.

A note on my part: It would be easier for these diagnostics to adapt the server_port parameter here to enable interprocess communication between MIRA frameworks. This way, we wouldn't have to kill anything to access MIRA stuff, we could just connect to the running instance by doing miracenter -k ip:port.

[bfalacerda]

So we got betty to use ~8A by loading all her processors, and then put her to charge with everything closed. Looks good I think:

[bfalacerda]

Just to clarify,battery starts dropping once @ferdianjovan started doing stuff with her

[marc-hanheide]

Wow, this looks nice... So, @creuther what is the difference between this newer robot (betty is the new one!) and the older versions? @bfalacerda can you do the same with bob, just to see if there is a difference?

[bfalacerda]

Bob has a broken charger thingy, so he only charges at 5A tops (we're getting a new one). I left him charging with a cable (we only have 1 charging station in Bham), and with skirt off, and this is what it looks like. It's less stable, but it also looks good I think. I'll try the same with the skirt on, but atm if we use more than 5A he'll be losing battery even when plugged in. That makes it harder to heat up the system... How should I do it?

[creuther]

@marc-hanheide Especially with a new charger, there should be no significant difference between the robots. As mentioned before, can you run another one of these plots with voltage instead of lifetime? Especially after changing the charger but also in general, the lifetime is (unfortunately) not the most reliable metric.
In your last plot it could in theory be that the robot reached full charge at around 8am, though I know that it is unlikely. We would be able to see that by plotting the voltage, which would level out when the battery is fully charged (and then jittering as charging is turned on and off periodically to keep the robot at full charge but not overcharge it).

Also, to exclude any issues we should really do a test where the robot is charging from a non-full state with all computers and external peripherals turned off. The status LED will still show the voltage and current, and there we could see how well the robot charges without any kind of load.

[gestom]

Hi, since Betty had some problems with her charger at TSC, we left Linda's charger there and took Betty's to Lincoln. With Linda, the charging is working quite OK, but we still had the overheating problem. When charging, Linda's charging current initially started about 6A, but it dropped quickly to 3A, which makes charging rather slow. We went for a hardware solution with a 92 mm fan, see
.

We tried three full charging cycles: without the vent, vent only and vent+fan. The results are attached as well.

So for us, the HW solution to prevent the overheat seems to work quite OK.

[marc-hanheide]

<3

[cdondrup]

Why is this assigned to me? Maybe you meant @creuther ?!

[marc-hanheide]

I assume so, yes.

[hawesie]

@gestom @marc-hanheide do you have instructions somewhere on how to do this or a description of what you did? I'd like to get Betty equipped with a fan before she goes back to TSC.

[gestom]

we (our technician) cut a circular vent in the bottom skirt (see the pic) and installed a 92mm 12V PC fan, which we connected to one of the side PCs

[marc-hanheide]

@kirumang this is what needs to be done for henry!

[gestom]

@kirumang I checked the logs from the last two days and Werner is charging only by 2 Amps. There are two differences to our robot: 1) the fan is 80mm instead of 92 2) the firmware was not probably updated. I will get a silent 92mm fan and update the firmware now. It seems that I will need to come here next week as well.

[marc-hanheide]

Boooh... Not again or still...