Interesting… Okay well we’re getting closer to identifying the problem at least!
Could you please, on your microscope, run:
sudo ofm upgrade -u https://build.openflexure.org/openflexure-microscope-server/debug/openflexure-microscope-server-mmgoessens-0.tar.gz
then sudo ofm stop
then sudo ofm serve
Your console will fill up with a TONNE of junk constantly showing the size of the current stream frame. Could you please watch this number (it should be pretty high, hundreds of thousands) then move to a region that causes your stream to cut off, and see if the number drops to 0 or stays high?
When you’re done, you cat Ctrl-C to exit. Please only do this once, as pressing again during the shutdown sequence can cause your settings file to become corrupt (unlikely but I’ve seen it happen)
I appreciate this is a time sink for you, but it would really help debug the issue.
Thanks!
For anyone else on the dev team following this, my logic is that if the frame size stays high, then we can assume it’s an issue with the network. If the frame size drops to zero then it may be something like running out of VRAM when encoding the camera data to JPEGs. If it’s a network limit then there’s not a lot we can do save for allowing reduced stream quality at the expense of autofocus reliability.
I also am having this issue.
At first it was both a local and remote issue so I swapped out the Pi 2b I was using with a 3B. This fixed the issue locally, but was still having the issue remotely so I disabled the wifi and used an ethernet connection. I think, but it’s too early to tell that this may have sorted it. Could that suggest it is a resource and bandwidth issue?
Edit: My camera wasn’t calibrated correctly and the colours were off. After recalibrating it no longer streams over network. Also if I move the slide the Pi grinds to a halt for several seconds and the mouse doesn’t work etc… This has only started happening since most recent update. I’ve used an onion cell slide as a test throughout. As above not using motors and does stream fine when unfocused using manual focus. Has also occurred to me that this may have caused the issues I was having calibrating my platform and the error message that it wasn’t seeing movement after moving the platform as I only tried to calibrate that on the older Pi.
It is a shame this is still happening. I know Joel is working at lightning speed doing a code spring clean and puting in lots of type checking and memory improvements. Hopefully this will solve this issue.
@jc2450 are there any logs that will help you debug this?
I haven’t yet had a response from the previous poster and it would be great if you could check this, as it’d give a lot of useful information.
Thanks!
Edit: your information about WiFi to ethernet helping is super useful! It does seem like this may be bandwidth related. If that’s the case then we may be able to solve this by allowing users to lower the stream quality.
!
I have to admit Pis and linux aren’t my strong point. I didn’t seem to get any stream values with running your suggestion? Have I done something wrong? In the meantime it is at least now consistently working locally, just not over network, wired or wi-fi but it definitely wasn’t working on the Pi 2 at all.
Then from the camera settings lower the stream quality to something really low like 25? I’ve no idea if it’ll help, but it’s worth a go.
Edit: Ignore my previous comment. Contrary to the PiCamera documentation, the MJPEG quality setting seems to do literally nothing.
I’m pushing out a 2.9.1 patch release which replaces the quality option with a bitrate option. Note though that lowering the bitrate can impact fast-autofocus performance! I’ve done some non-extensive tests and it seems like it still works well enough in most cases, but your milage may vary.
Once this is out, try loweing the stream bitrate one level at a time and see if that helps.
I’m also having this problem (or at least one with the same symptoms).
RPI 3B, Camera module v1, good WIFI connection.
I’ve just now updated to server 2.9, and the issue is still there.
Looking at the MJEPG stream in the browser, I noticed that I have a white bar at the bottom of the picture, and that grows the sharper the focus of the image is; screenshots with the two extremes are at the end of the post – I could literally increase/decrease the size of that white bar with the focus knob.
I don’t think it is a RAM or CPU issue, there’s 512MB of RAM free on this freshly booted system, and CPU idle never dropped below 84%. I also don’t think it’s the network, I’m getting a consistent 100Mbps with iperf.
If I reduce the stream bitrate from “Maximum” to “High” the problem goes away, at least with this slide.
@jc2450 BTW: do you still need output from that debug build, or anything else? I’m good with linux and tomorrow is a bank holiday, so feel free to throw anything you need my way especially now that I have a way to reproduce the issue.
Hey, thanks! To be honest the most useful thing to do at this stage would be to upgrade to 2.9.1, then see if the new bitrate setting helps.
It’s in the Camera section of settings. If lowering the bitrate doesn’t help then I’ll make a new build with frame data logging to help debug.
Sorry ignore that! I see lowing the bitrate helped. That’s fantastic! I think we might actually be straining 100mbps with this high framerate stream at maximum bitrate. I’m running a gigabit LAN and don’t see this issue so my money is still on that.
What I’ll do then is add a help section to the GUI with advice on lowing the bitrate if this issue presents itself. If this is indeed the issue (and I’d like to verify it still), then it’s not something we can easily avoid other than suggesting lowering the bitrate.
It might indeed be the bandwidth – I just had time for a short test, so I connected the microscope to my notebook via ethernet, and saw the same phenomenon when setting the stream bitrate to “Maximum”.
I checked the traffic, and it maxed out at about 280 Mbps, which IIRC is the limit of what the PI 3B+ can do even on Gigabit Ethernet. When unfocusing the image, at about 250 Mbips the stream was fine again.
It’s weird because we have people running the microscope with a Pi 3 over gigabit ethernet and not hitting this issue, so I’ll have to investigate what specifically would be causing it. I guess it could just be the sample but I’d like to find out if it’s something else.
While Raspberry Pi 3 Model B+ added Gigabit Ethernet connectivity, throughput on Raspberry Pi 4 is free from the single shared USB 2.0 channel to the SoC. The throughput of all Raspberry Pi models with a built-in Ethernet port is measured using the iperf3 tool, showing the average network throughput (in megabits per second) over several runs.
Following on from this, I’ve tried using Wondershaper to limit my Pi 4 ethernet bandwidth to (REALLY) low values (~10mbps). In these tests, all that happens is that the framerate drops.
The way Flask generators work means this is what I’d expect. I now have a new working theory on this.
I wonder if it’s related to the available VRAM? When you limit the bandwidth, you limit the bitrate coming out of the GPU image encoder. I wonder if running out of RAM or VRAM would mean that, at high bitrates, a whole frame cannot be stored? The server is then sending all the data it has, as expected, it’s just an unfinished frame because of a memory issue.
It may not be that, but it’s an idea at least. Again, for now lowering the bitrate seems to help, but I’m throwing all my ideas out here as it’s a tricky one…
Edit: But we fix the amount of allocated VRAM in our SD image builder, and it’s the same for all models…
The bitrate at which video will be encoded. Defaults to 17000000 (17Mbps) if not specified. The maximum value is 25000000 (25Mbps). Bitrate 0 indicates the encoder should not use bitrate control (the encoder is limited by the quality only).
That should say -1 disables bitrate control. This is our “maximum” bitrate value, and leads to the best fast-autofocus performance (since the JPEG frame quality is constant), and as such is the default.
What I now think may be happening is the following:
I established a while ago that the quality parameter in the video encoder has a negligible, if any, effect on the MJPEG frames. We assume that the MJPEG quality is fixed, and high by default.
Lowering the bitrate actually does cause the encoder to lower the JPEG frame quality.
Disabling bitrate control then means that the encoder will attempt to output high quality frames with no regard for its own bitrate limit. Therefore, in situations where the frame is highly detailed, the required bitrate may exceed the camera limit, resulting in incomplete or missing frames coming out of the encoder
This may be tricky to really solve long-term. I can add an interface to change the stream resolution to see if that helps. Otherwise, for now we may just have to deal with the tradeoff between bitrate and autofocus reliability.
Edit: A nice test for if this issue is occurring is to check if the end of a frame is the correct JPEG end of image bytes (0xFF, 0xD9). If it doesn’t end with that then it’s been truncated by bitrate limits.
That would be helpful, as then you could show a message to the user. As it stands now, in the client there’s just a white screen, while in firefox there’s a white bar at the bottom of the stream.
As I’m leaving the project in basically 2 weeks, I’m focusing on bug fixes rather than new features now, so hopefully the SSE stuff will get included at some point, but for now I suggest lowering the stream resolution in settings, or limiting the bitrate to Highest (under the Advanced section of the camera settings.)
I’d be interested to know how fast autofocus performs with the bitrate limit in place, so if you’re able to test that on a region of the sample that previously broke the stream, that’d be really useful.
