RPi 1B + PiCamera V1.3 - Auto Calibration (SOLVED!)

Greetings from Brazil,


Introduction:

I extend my sincerest congratulations to all those involved in the Openflexure initiative. The concept is truly remarkable, and upon my discovery of it, I was immediately compelled to embark on its construction.

Currently, I am in the process of building a version based on the beta V7 with high-resolution capabilities and epi-illumination. However, I encountered a need for an upright version during the assembly process. Consequently, I am currently experimenting with various designs to accommodate a beamsplitter cube in an inverted orientation, as well as exploring other methods to facilitate seamless transitions between epi-illumination and trans-illumination. These developments will be detailed in a subsequent post, once I have attained more tangible and presentable results to share.

Beyond my personal requirements, I envision a simplified and cost-effective iteration suitable for implementation in underprivileged public schools within my country. My aim is to demonstrate the efficacy of utilizing older Raspberry Pi boards paired with inexpensive cameras. Consequently, I am presently testing a configuration employing an antiquated RPi 1 B+ (later substituted with an RPi 2B due to issues with the SD card connection) in conjunction with the affordably priced PiCamera V1.3, alongside an Arduino Nano. Despite encountering various warnings and challenges reported in the forum, I became committed to this endeavor.

Just to make my point about affordability in developing countries like Brazil:

Affordable Version Price (USD) Recommended by OpenFlexure Price (USD)
Component Local Aliexpress Component Local Aliexpress
Raspberry Pi 1B+ / 2B (second handed) 20.00 * Raspberry Pi 4 Model B (2Gb) 125.00 **
PiCamera V1.3 (new - local market) 11.00 3.30 PiCamera V2 41.60 28.00
Total 33.00 23.30 Total 166.60 153.00

/* Aliexpress - not available
/** Aliexpress $75 + import taxes kicks-off (+$70) = $145, therefore not worthwhile

In summary, by acquiring second-hand RPi 1B / 2B boards (readily available from individuals selling them - usually used as RetroPie video games) and PiCamera V1.3 modules, we can assemble six microscopes at a fraction of the cost compared to building them with the currently recommended components. This approach significantly enhances affordability and accessibility, particularly for initiatives aimed at providing essential scientific tools to underserved communities.

Indeed, it’s crucial to acknowledge that while I am currently utilizing older and outdated components to construct the microscope, I fully support the community’s broader efforts to adapt and advance the project with newer technologies such as the Raspberry Pi 4/5, PiCameras V2/V3, Sangaboard V4, and so forth. These advancements are essential for enhancing performance and functionality. However, it’s important to recognize that newer products often come with higher price tags, posing challenges for widespread adoption, particularly in resource-constrained communities.

By leveraging older components, we can mitigate cost barriers and ensure that the benefits of this technology reach those who need it most. Nonetheless, as the project evolves, finding ways to balance innovation with affordability will remain a key consideration to ensure inclusivity and accessibility for all.



Solving the issues with Raspberry Pi 1B+ / 2B and PiCamera V 1.3

Moving beyond these observations, let’s delve into the practicalities of setting up and operating an OpenFlexure Microscope with a Raspberry Pi 1B+ / 2B and PiCamera V 1.3.

For the Raspberry Pi, the setup is relatively straightforward. One must utilize the headless version of Raspbian OpenFlexure Lite, which lacks desktop support. However, this version allows seamless access to the microscope through its web interface.

However, the real challenge arises with the PiCamera V1.3. Initially, users may encounter difficulties getting the OpenFlexure’s Auto Calibration feature to function correctly, as extensively discussed in various online forums and posts.

To overcome this hurdle, I embarked on an in-depth exploration, delving into various concepts such as Bayer filters, lens shading correction, and binning. I dedicated approximately one week, investing numerous hours in studying resources such as the Openflexure-microscope-server, Picamerax documentation, and code repositories:


My quest also led me to experiment with algorithms for creating lens shading correction tables, exploring resources from contributors like 6by9 and cpiaxip:


Additionally, I ventured into recompiling RaspiStill.c to incorporate a lens shading correction table:
Through this rigorous exploration, I believe I have discovered a solution to enable Auto Calibration and Lens Shading Correction with a PiCamera V1.3 within the openflexure-microscope-server. I eagerly await feedback and reviews from esteemed members of this community.

For those interested in replicating my efforts, I offer detailed instructions below. However, feel free to skip ahead to the end of this post for the concise code adjustments necessary to enable PiCamera V1.3 compatibility within the openflexure-microscope-server.


A. Auto White Balance (AWB)
The Auto White Balance (AWB) functionality with a PiCamera V1.3 presents challenges, often resulting in greenish images that are notably inferior to the original ones. There are several posts about such behavior and the “conclusion” was that the PiCamera V1.3 has been used instead of the supported V2.

To illustrate this discrepancy, I conducted experiments both with and without samples, yielding consistent results:

  • Figures A and B depict the microscope’s view without any sample (a clear field) before and after AWB adjustment.
  • Figures D and E showcase the microscope’s view with onion cells before and after AWB adjustment.

Upon investigation, I discovered that the PiCamera V1.3 (OV5647) and V2 (IMX219) employ different Bayer filter patterns. Specifically:
  • The PiCamera V1.3 utilizes the following Bayer pattern:
    GBGBGBGBGBGBGB…
    RGRGRGRGRGRGRR…
    GBGBGBGBGBGBGB…
    RGRGRGRGRGRGRG…

  • Conversely, the PiCamera V2 employs this Bayer pattern:
    BGBGBGBGBGBGBG…
    GRGRGRGRGRGRG…
    BGBGBGBGBGBGBG…
    GRGRGRGRGRGRG…

Analysing the OpenFlexure Microscope Server code, I discovered that the function “channels_from_bayer_array” in “recalibrate_utils.py” (at /var/openflexure/application/openflexure-microscope-server/openflexure_microscope/api/default_extensions/picamera_autocalibrate/) was designed to map the V2 bayer pattern to BGGR channels, but not to map the V1.3 one.

Therefore, the trick is to change the mapping in line 256:

  • from : bayer_pattern: List[Tuple[int, int]] = [(0, 0), (0, 1), (1, 0), (1, 1)]
  • to: bayer_pattern: List[Tuple[int, int]] = [(0, 1), (1, 1), (0, 0), (1, 0)]

Indeed, the optimal approach to addressing the issue of different Bayer filter patterns between PiCamera V1.3 and V2 would involve implementing a more flexible solution within the "channels_from_bayer_array" function. Rather than hardcoding a new mapping specific to the V1.3 camera, a more robust method would be to introduce an additional parameter that specifies the camera version. This parameter could then be used to dynamically select the appropriate mapping.

Here’s a conceptual outline of how this adjustment could be made:

def channels_from_bayer_array(bayer_array: np.ndarray, cam_version: int) → np.ndarray:
if bayer_array.shape==2:
…# New PiCamera: V2 - IMX219
…bayer_pattern: List[Tuple[int, int]] = [(0, 0), (0, 1), (1, 0), (1, 1)]
else:
#PiCamera V1.3 - OV5647
…bayer_pattern: List[Tuple[int, int]] = [(0, 1), (1, 1), (0, 0), (1, 0)]

We can determine the camera version parameter “cam_version” by using a similar method as described in the Picamerax recipe for Capturing Raw Bayer Data , in preparation for calling “channels_from_bayer_array” as follows:

cam_version = {
… ‘RP_ov5647’: 1,
… ‘RP_imx219’: 2,
}[camera.exif_tags[‘IFD0.Model’]]

With the implementation of this straightforward adjustment, the Auto White Balance (AWB) functionality of the OpenFlexure-Microscope-Server can be rectified, ensuring compatibility with both PiCamera V1.3 and V2 versions. Figures C and F serve as prime examples of the successful AWB procedure, displaying significantly improved image quality compared to the original images (A and D) and versions affected by incorrect AWB mapping (B and E).


B. Auto Flat Field Correction (Lens Shading Correction)
Let´s move onto the issue that took most of my time: the Lens Shading Correction.

After reading the great paper written by RW. Bowman at al. Flat-Field and Colour Correction for the Raspberry Pi Camera Module, I was able to “start” understanding the problems related to LSC, vignetting, etc, to rerun most of their codes, and I became confident that the distorted colors of my microscope images could be corrected (if not directly through the openflexure-microscope-server, it could be done offline).

I also understood that RW.Bowman took the main ideas about LSC from 6by9 and transferred them to pycamerax and later to the openflexure-microscope-server code. Nevertheless, for some unknown reason my images after the “Auto Flat Field Correction” were no better than the original ones, even after the “corrected” AWB correction. My white (clean field) images like Figure G were transformed into color distorted images like Figure H after the Auto Flat Field Correction procedure, no matter how homogeneous my illumination was (that was neither homogeneous nor totally symmetric to be sincere).

After further study and hands-on experimentation, I successfully ran the programs developed by 6by9 (in C) and cpixip (in Python) to generate Lens Shading Correction (LSC) tables. Additionally, I rebuilt RaspiStill.c to integrate these new tables. As a result, I achieved reasonably homogeneous grayish images when using a clean field input, marking a significant milestone in my efforts to improve color accuracy in microscope images (yehhhh! :grinning:)

But why didn’t it happen with the algorithm implemented in the OpenFlexure Microscope Server? Why was the LSC working in RaspiStill but not in the OpenFlexure?

To gain insight into the issue, I decided to employ a non-white image, specifically a tilted grid pattern (Figure J), as the basis for the LSC procedure. This allowed me to visualize the image pattern in the LSC tables.

Upon evaluating the tables generated by the programs from 6by9 (in C) and cpixip (in Python) for the tilted grid pattern, I observed differences in their outputs. Interestingly, both sets of tables yielded perfect and similar results when integrated into the modified RaspiStill.c. However, there were distinct characteristics: the ls_table.h generated by 6by9’s program exhibited weights/gains similar to the input image(fig. J), while the ls_table generated by cpixip’s program was flipped horizontally and vertically (Figures N and O were created with the red channel of the ls_table.h created with 6by9 and cpixip’s programs, respectively).

A crucial parameter, ref_transform, was present at the end of the ls_table.h files generated by both programs. Notably, in 6by9’s program, ref_transform was set to 0, aligning with the similarity between the ls_table and the input image. Conversely, in cpixip’s program, ref_transform was set to 3, reflecting the flipped input image. This parameter was directly passed to the MMAL data structure in RaspiStill.c (mmal.MMAL_PARAMETER_LENS_SHADING_T). In short:

  • 6by9’s program: ls_table.h similar to input image and red_transform=0
  • cpixip’s program: ls_table.h flipped input image and red_transform=3

Additionally, examining the implementation in picamerax (Figure M), I found that the value 3 was hardcoded in the "_upload_lens_shading_table" function of "camera.py". This observation was accompanied by a comment indicating the need for further clarification on the appropriate value for ref_transform:

LINE 2548: ref_transform = 3, # TODO: figure out what this should be properly!!!


In summary, the issue stemmed from picamerax, utilized by the OpenFlexure Microscope Server, expecting a lens shading correction table that needed to be flipped horizontally and vertically in relation to the white clean field input image. To confirm this discovery, I utilized the tilted grid image (Figure J) to execute the "Auto Flat Field Correction'' process. The resulting "lens shading corrected" image (Figure K) clearly indicated that the correction was not applied as expected.

Rectifying this issue was relatively straightforward. By reversing the horizontal and vertical order of the second and third axes (x, y) within the “lst_from_channels” function in “recalibrate_utils.py”, the correction process was corrected:

  • Original Code: return lens_shading_table[::-1, ::, ::].copy()
  • New Code: return lens_shading_table[::-1, ::-1, ::-1].copy()

With this simple adjustment, the Auto Flat Field Correction procedure now functions properly with my PiCamera V1.3. Figure L depicts the tilted grid image as input, while Figure I shows the result with the white clean field image as input.

While this adjustment resolves the issue for PiCamera V1.3, I remain somewhat confused about how the current Auto Flat Field Correction code operates with a PiCamera V2. Given that picamerax expects a flipped lens_shading_table, I wonder how this might affect the correction process for PiCamera V2. Unfortunately, without access to a PiCamera V2 for testing, I am unable to ascertain the exact implications.

I am eager to hear insights from experts like @r.w.bowman, whose expertise could shed light on this matter and provide valuable clarification.



Dirty temporary corrections to make AWB and LSC work for PiCamera V1.3 (hardcoded)

  1. Open the program “recalibrate_utils.py” located in the directory /var/openflexure/application/openflexure-microscope-server/openflexure_microscope/api/default_extensions/picamera_autocalibrate of your Raspberry Pi with your preferred file editor.

  1. Go to line 256 and replace the mapping
  • From: bayer_pattern: List[Tuple[int, int]] = [(0, 0), (0, 1), (1, 0), (1, 1)]
  • To: bayer_pattern: List[Tuple[int, int]] = [(0, 1), (1, 1), (0, 0), (1, 0)]

  1. Go to line 354 and replace the return statement
  • From: return lens_shading_table[::-1, :, :].copy()
  • To : return lens_shading_table[::-1, ::-1, ::-1].copy()

  1. Restart the OFM server with “ofm restart” and you may test the Automatic Calibration of your cheap PiCamera V1.3 within your OpenFlexure Microscope.


I trust that this (too extensive) post has proven to be worthwhile, and I hope that this information may help someone. While I have only just begun this journey, dedicating a week+ to delving deeply into the intricacies of Auto White Balance (AWB) and Lens Shading Correction (LSC) is merely a small fraction of the countless hours that many of you have dedicated to advancing our collective understanding.

Best regards,
PEB


Figures:

5 Likes

Hello, welcome, and well done! You have done a power of work here, and I heartily commend you for sharing everything here. Getting hold of older Pi and Pi Camera modules in the UK is quite hard: it’s not a question of cost so much as they aren’t sold any more, and at the University it’s hard to buy things second hand. I hadn’t realised that the discontinued boards were still readily available.

You are quite correct that I made a few assumptions in my lens shading code. I think the Pi Camera v2 is flipped by default, so the settings that are hard-coded are correct for the default configuration. I always intended to go back and generalise it (e.g. reading the flipping state from the camera) but as those parameters were not properly documented (at least not when I wrote that code) I never found the time to properly reverse-engineer it. I am very impressed you’ve managed to get it working with so few changes.

If you wanted to submit a pull request to picamerax and/or the OpenFlexure Server to make it work properly for the camera module v1.3, I would happily try to get that in. However, I’ll mention at this point that we’re hoping to release a rewritten v3 of the software, which will make most of v2 obsolete in the next few months. One important change is updating the underlying OS and libraries (to Bookworm). That raises an interesting question: would the older Raspberry Pi models you are using support Bookworm? It’s still 32-bit, so no problem there, but it may not cope with the smaller RAM or processor. The good news is that the code for the camera is much more modular in v3, so adapting it to support the older camera module should be possible.

By the way, my GitHub/GitLab notifications often don’t reach me, so feel free to contact me on here if I appear to be ignoring you there!

All the best,

Richard

Hello Richard,

Thank you for your response.

The availability and cost in each country are factors that must be addressed locally, and you and the development team should continue to evolve the microscope as you are doing.

I felt it was my duty to share my findings after all the excellent work the OFM team has done in developing and open sourced the OFM. Additionally, you have done an excellent job with Picamerax and the OFM server - creating software that is versatile enough to accommodate all hardware possibilities is simply not feasible, and assumptions inevitably have to be made.

My question, stemming from my lack of knowledge, pertains to who is responsible for flipping the LSC table: Was it flipped by the camera, by MMAL, or…? I am not certain, but it may not be particularly relevant (I believe).

Suggestions:

  1. Submit a pull request to correct OFM’s AWB procedure.

  2. Submit a pull request to correct the LSC handling (ref_transform) in Picamerax. With this adjustment, Picamerax’s LSC will be applicable for purposes beyond OFM, eliminating the need to alter the logic of the “lst_from_channels” function in OFM. I have already made the change in my local environment, and it worked perfectly:
    PIcameraX

I am not a programmer anymore (30 years ago, I knew something…), so I have no idea how to properly perform a pull request on GitHub/GitLab (total noob). Let me research how to do it. If someone could provide a reference or a step-by-step procedure, I would appreciate it. (It can be sent via direct message to avoid cluttering this post.)

Regards,
PEB

PS.: I will need some help to test the changes of the pull requests as I just own a PiCamera V1.3.

1 Like

Thanks for that - I think if it can be fixed in picamerax that’s likely the most useful place to fix it. I maintain both, so it makes little difference to me.

I think the problem is something like this: the camera can be configured to flip the images in hardware, i.e. change the read-out direction, if I remember correctly. The lens shading table is then flipped by the camera pipeline to match the sensor - so if I calibrate the LST with flipping enabled, and then use the camera with it disabled, this should cause the LST to be flipped to match the image.

I think handling the transform correctly would be the neatest option for sure: probably just grabbing whatever the current value of flipping is, and putting that into the transform, would be enough. I suspect that integer should be 0,1,2,3 depending on the values of hflip and vflip, but it might take a little experimenting to find out…

As for how to make a pull request, the best way is to make a fork of the repository (click the “fork” button on github) then clone that to your microscope (Github has instructions - but you basically run git clone https://github.com/your_username/picamerax.git). Install it by changing into the repo’s directory, and running ofm activate followed by pip install -e .. You can then make any edits you need to the files, and commit them with git commit (which will pop up an editor for the commit message). After that, “push” your changes to your fork with git push.

If you look at the upstream copy of the repository on github, you should then be able to click a button to create a pull request - and once that’s done, I can probably take it from there.

If you can verify it works with picamera v1, I can check it on picamera v2.

Richard, thanks for the explanation about what probably is going on with the different versions of the picameras, and thank you so much for you guidance regarding how to create de pull request.

On March 4th, 2024:
Unfortunately, I might be doing something really wrong because after forking, cloning, activating the python OFM environment, when I try to perform the installation of the forked picamerax (pip install -e . ), I receive a message saying “ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: ‘/var/openflexure/application/openflexure-microscope-server/.venv/bin/pip’ Check the permissions.”.

I tried to change temporarily the permissions of /var/openflexure/application/openflexure-microscope-server/.venv to my user (sudo chown -R mysuser:myuser .venv). After that, I was almost able to perform the installation of the forked picamerax, but I receive another error: “ERROR: pip’s dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. openflexure-microscope-server 2.11.0 requires picamerax~=20.9.1, but you have picamerax 21.9.8 which is incompatible.”

On March 5th, 2024:
Never mind. After some research, I realized that I should had “installed” the development environment of the ofm server beforehand(ofm develop) - no need to change permissions, etc…

Nevertheless, even though the pycamerax installation worked as expected, the message about the version incompatibility remains: “openflexure-microscope-server 2.11.0 has requirement picamerax~=20.9.1, but you’ll have picamerax 21.9.8 which is incompatible.”

Should I look for forking the picamerax 20.9.1 version instead of the current one (how)? Or should change the dependencies/requirements of the openflexure-microscope-server 2.11.0 to picamerax 21.9.8 (probably not)?

Regards,
PEB

Hi @pbrugugnoli,
Your code change looks merge-ready as it is; I think if you click in the banner that says “this repository is 1 commit ahead of labthings/picamerax” you should be able to simply click a button to create a merge request. If you create it, I will check and adjust the settings before clicking merge.

I had not spotted that the server requires an out of date version of picamerax. Don’t worry about that - we should update the version of its requirements, and I can do that.

Richard