media: i2c: imx477: Support for the Sony IMX477 sensor

dt-bindings: media: i2c: Add IMX477 CMOS sensor binding

Add YAML device tree binding for IMX477 CMOS image sensor.

Signed-off-by: Naushir Patuck <[email protected]>

media: i2c: Add driver for Sony IMX477 sensor

Adds a driver for the 12MPix Sony IMX477 CSI2 sensor.
Whilst the sensor supports 2 or 4 CSI2 data lanes, this driver
currently only supports 2 lanes.

The following Bayer modes are currently available:

4056x3040 12-bit @ 10fps
2028x1520 12-bit (binned) @ 40fps
2028x1050 12-bit (cropped/binned) @ 50fps
1012x760 10-bit (scaled) @ 120 fps

Signed-off-by: Naushir Patuck <[email protected]>

media: i2c: imx477: Add support for adaptive frame control

Use V4L2_CID_EXPOSURE_AUTO_PRIORITY to control if the driver should
automatically adjust the sensor frame length based on exposure time,
allowing variable frame rates and longer exposures.

Signed-off-by: Naushir Patuck <[email protected]>

media: i2c: imx477: Return correct result on sensor id verification

The test should return -EIO if the register read id does not match
the expected sensor id.

Signed-off-by: Naushir Patuck <[email protected]>

media: i2c: imx477: Parse and register properties

Parse device properties and register controls for them using the V4L2
fwnode properties helpers.

Signed-off-by: Laurent Pinchart <[email protected]>

media: i2c: imx477: Selection compliance fixes

To comply with the intended usage of the V4L2 selection target when
used to retrieve a sensor image properties, adjust the rectangles
returned by the imx477 driver.

The top/left crop coordinates of the TGT_CROP rectangle were set to
(0, 0) instead of (8, 16) which is the offset from the larger physical
pixel array rectangle. This was also a mismatch with the default values
crop rectangle value, so this is corrected. Found with v4l2-compliance.

While at it, add V4L2_SEL_TGT_CROP_BOUNDS support: CROP_DEFAULT and
CROP_BOUNDS have the same size as the non-active pixels are not readable
using the selection API. Found with v4l2-compliance.

This commit mirrors 543790f done for
the imx219 sensor.

Signed-off-by: Naushir Patuck <[email protected]>

media: i2c: imx477: Remove auto frame length adjusting

The V4L2_CID_EXPOSURE_AUTO_PRIORITY was used to let the sensor control
frame length (effectively framerate) based on the requested exposure
time requested. Remove this feature as it is never used, and goes
against how V4L2 likes to handle exposure and vblank controls.

Signed-off-by: Naushir Patuck <[email protected]>

media: i2c: imx477: Add very long exposure control to the driver

Add support for very long exposures by using the exposure multiplier
register. Userland does not need to pass any additional controls to
enable long exposures, it simply requests a larger vblank to extend the
exposure control range appropriately.

Currently, since hblank is fixed, a maximum of approximately 124 seconds
of exposure time can be used. In a future change, hblank could also be
controlled in userland to give over 200 seconds of exposure time.

Signed-off-by: Naushir Patuck <[email protected]>

media: i2c: imx477: Fix crop height for 2028x1080 mode

The crop height for this mode was set at 2600 lines, it should be 2160
lines instead.

Signed-off-by: Naushir Patuck <[email protected]>

media: i2c: imx477: Replace existing 1012x760 mode

The existing 1012x760 120 fps mode has significant IQ problem using
the internal sensor scaler. Replace this mode with a 1332x990 120 fps
mode instead. This new mode has a smaller field of view, but does not
suffer from the bad IQ of the original mode.

Signed-off-by: Naushir Patuck <[email protected]>

media: i2c: imx477: Remove internal v4l2_mbus_framefmt from the state

The only field in this struct that is used is the format code, so
replace the struct with this single field.

Save the format code in imx477_set_pad_format() when setting up a new
mode so that imx477_get_pad_format() performs the right lookup.
Otherwise, this caused a bug where the mode lookup occurred on the
12-bit table rather than the 10-bit table.

Signed-off-by: Naushir Patuck <[email protected]>

media: i2c: imx477: Remove unused function parameter

The struct imx477 *ctrl parameter is not used in the function
imx477_adjust_exposure_range(), so remove it.

Signed-off-by: Naushir Patuck <[email protected]>

media: i2c: imx477: Fix for long exposure limit calculations

Do not scale IMX477_EXPOSURE_OFFSET with the long exposure factor during
the limit calculations. This allows larger exposure times, and does seem to be
what the sensor is doing internally.

Signed-off-by: Naushir Patuck <[email protected]>

media: i2c: imx477: Extend driver to support imx378 sensor

The imx378 sensor is almost identical to the imx477 and can be
supported as a "compatible" sensor with just a few extra register
writes.

Signed-off-by: David Plowman <[email protected]>

media: i2c: imx477: Fix framerates for 1332x990 mode

The imx477 driver's line length for this mode had not been updated to
the value supplied to us by the sensor manufacturer. With this
correction the sensor delivers the framerates that are expected.

Signed-off-by: David Plowman <[email protected]>

media: i2c: imx477: Allow control of on-sensor DPC

A module parameter "dpc_enable" is added to allow the control of the
sensor's on-board DPC (Defective Pixel Correction) function.

This is a global setting to be configured before using the sensor;
there is no intention that this would ever be changed on-the-fly.

Signed-off-by: David Plowman <[email protected]>

media: i2c: imx477: Sensor should report RAW color space

Tested on Raspberry Pi running libcamera.

Signed-off-by: David Plowman <[email protected]>

media: i2c: imx477: Add vsync trigger_mode parameter

trigger_mode == 0 (default) => no effect / no registers written
trigger_mode == 1           => source
trigger_mode == 2           => sink

This can be set e.g. in /boot/cmdline.txt as imx477.trigger_mode=N

Signed-off-by: Jonas Jacob <[email protected]>

media: i2c: Update imx477 Kconfig entry

Bring the IMX477 Kconfig declaration in line with upstream entries.

Signed-off-by: Phil Elwell <[email protected]>

media: i2c: imx477: Correct minimum exposure lines

The minimum number of exposure lines value (IMX477_EXPOSURE_MIN) was
previously 20 but this is not correct. The datasheet is not completely
explicit, however the new value of 4 has been tested with all the
sensor modes supported by this driver, and matches the lowest exposure
value of 114us that could be achieved wtih Raspberry Pi's legacy
firmware driver.

Signed-off-by: David Plowman <[email protected]>

media: i2c: imx477: Allow dynamic horizontal blanking control

Currently, the V4L2_CID_HBLANK control is marked as read-only. Remove this
restriction and allow userland to modify the control if needed.

Set the maximum limit of the line length to 0xfff0.

Signed-off-by: Naushir Patuck <[email protected]>

media: i2c: imx477: Reset hblank on mode switch

Reset the hblank control to the minimum value on every mode switch. This is to
account for userland instances that do not yet control hblank, otherwise it
gets set to a non-optimal value.

Signed-off-by: Naushir Patuck <[email protected]>

media: i2c: imx477: Do not unconditionally adjust hblank and vblank limits

On a mode change, only call imx477_set_framing_limits() to adjust the hblank
and vblank limits if the new mode is different from the existing mode. This
preserves any manual control values the user might have set.

Signed-off-by: Naushir Patuck <[email protected]>

driver: media: i2c: imx477: Re-enable temperature sensor

The temperature sensor enable register write got lost at some point.
Re-enable it.

Signed-off-by: Naushir Patuck <[email protected]>

media: i2c: imx477: Fix locking in imx477_init_controls()

The driver does not lock the imx477 mutex when calling
imx477_set_framing_limits(), leading to:

WARNING: CPU: 3 PID: 426 at drivers/media/v4l2-core/v4l2-ctrls-api.c:934 __v4l2_ctrl_modify_range+0x1a0/0x210 [
videodev]

Fix this by taking the lock.

Signed-off-by: Tomi Valkeinen <[email protected]>

drivers: media: imx477: Disable the scaler

The horizontal scaler was enabled for the 2028x1520 and 2028x1080 modes,
with a scale factor of 1. It caused a single column of bad pixels on the
right edge of the image. Since scaling is not needed for these modes,
disable it entirely.

Signed-off-by: Naushir Patuck <[email protected]>

drivers: media: imx477: Set horizontal binning when disabling the scaler

The horizontal scaler has been disabled but actually the sensor is not
binning horizontally, resulting in images that are stretched 2x
horizontally (missing the right half of the field of view completely).

Therefore we must additionally set the horizontal binning mode. There
is only marginal change in output quality and noise levels.

Signed-off-by: David Plowman <[email protected]>
Fixes: f075893 ("drivers: media: imx477: Disable the scaler")

drivers: media: imx477: Add V4L2_CID_LINK_FREQ control

Add V4L2_CID_LINK_FREQ as a read-only control with a value of 450 Mhz.
This will be used by the CFE driver to corretly setup the DPHY timing
parameters in the CSI-2 block.

Signed-off-by: Naushir Patuck <[email protected]>

drivers: media: imx477: Correctly set IMX477_PIXEL_RATE as a r/o control

This control is meant to be read-only, mark it as such.

Signed-off-by: Naushir Patuck <[email protected]>

drivers: media: i2c: imx296,imx477: Configure tigger_mode every time

Don't assume the camera has been reset each time we start streaming,
but always write registers relating to trigger_mode, even in mode 0.

IMX477: Stop driving XVS on stop streaming, to avoid spurious pulses.

Signed-off-by: Nick Hollinghurst <[email protected]>

media: i2c: imx477: Squash fixes

imx477: make trigger-mode more configurable

Allow trigger-mode to be overridden using device tree so that it can be
set per camera. Previously the mode could only be changed using a module
parameter, which would then affect all cameras.

Signed-off-by: Erik Botö <[email protected]>

drivers: media: imx477: Add V4L2_CID_LINK_FREQ control

Add V4L2_CID_LINK_FREQ as a read-only control with a value of 450 Mhz.
This will be used by the CFE driver to corretly setup the DPHY timing
parameters in the CSI-2 block.

Signed-off-by: Naushir Patuck <[email protected]>

Author: naushir

Documentation/devicetree/bindings/media/i2c/imx477.yaml

@@ -0,0 +1,113 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/media/i2c/imx477.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Sony 1/2.3-Inch 12Mpixel CMOS Digital Image Sensor
+
+maintainers:
+  - Naushir Patuck <[email protected]>
+
+description: |-
+  The Sony IMX477 is a 1/2.3-inch CMOS active pixel digital image sensor
+  with an active array size of 4056H x 3040V. It is programmable through
+  I2C interface. The I2C address is fixed to 0x1A as per sensor data sheet.
+  Image data is sent through MIPI CSI-2, which is configured as either 2 or
+  4 data lanes.
+
+properties:
+  compatible:
+    const: sony,imx477
+
+  reg:
+    description: I2C device address
+    maxItems: 1
+
+  clocks:
+    maxItems: 1
+
+  VDIG-supply:
+    description:
+      Digital I/O voltage supply, 1.05 volts
+
+  VANA-supply:
+    description:
+      Analog voltage supply, 2.8 volts
+
+  VDDL-supply:
+    description:
+      Digital core voltage supply, 1.8 volts
+
+  reset-gpios:
+    description: |-
+      Reference to the GPIO connected to the xclr pin, if any.
+      Must be released (set high) after all all supplies and INCK are applied.
+
+  # See ../video-interfaces.txt for more details
+  port:
+    type: object
+    properties:
+      endpoint:
+        type: object
+        properties:
+          data-lanes:
+            description: |-
+              The sensor supports either two-lane, or four-lane operation.
+              For two-lane operation the property must be set to <1 2>.
+            items:
+              - const: 1
+              - const: 2
+
+          clock-noncontinuous:
+            type: boolean
+            description: |-
+              MIPI CSI-2 clock is non-continuous if this property is present,
+              otherwise it's continuous.
+
+          link-frequencies:
+            allOf:
+              - $ref: /schemas/types.yaml#/definitions/uint64-array
+            description:
+              Allowed data bus frequencies.
+
+        required:
+          - link-frequencies
+
+required:
+  - compatible
+  - reg
+  - clocks
+  - VANA-supply
+  - VDIG-supply
+  - VDDL-supply
+  - port
+
+additionalProperties: false
+
+examples:
+  - |
+    i2c0 {
+        #address-cells = <1>;
+        #size-cells = <0>;
+
+        imx477: sensor@10 {
+            compatible = "sony,imx477";
+            reg = <0x1a>;
+            clocks = <&imx477_clk>;
+            VANA-supply = <&imx477_vana>;   /* 2.8v */
+            VDIG-supply = <&imx477_vdig>;   /* 1.05v */
+            VDDL-supply = <&imx477_vddl>;   /* 1.8v */
+
+            port {
+                imx477_0: endpoint {
+                    remote-endpoint = <&csi1_ep>;
+                    data-lanes = <1 2>;
+                    clock-noncontinuous;
+                    link-frequencies = /bits/ 64 <450000000>;
+                };
+            };
+        };
+    };
+
+...

MAINTAINERS

@@ -22571,6 +22571,14 @@ T:	git git://linuxtv.org/media.git
 F:	Documentation/devicetree/bindings/media/i2c/sony,imx415.yaml
 F:	drivers/media/i2c/imx415.c
 
+SONY IMX477 SENSOR DRIVER
+M:	Raspberry Pi Kernel Maintenance <[email protected]>
+L:	[email protected]
+S:	Maintained
+T:	git git://linuxtv.org/media_tree.git
+F:	Documentation/devicetree/bindings/media/i2c/imx477.yaml
+F:	drivers/media/i2c/imx477.c
+
 SONY MEMORYSTICK SUBSYSTEM
 M:	Maxim Levitsky <[email protected]>
 M:	Alex Dubov <[email protected]>

drivers/media/i2c/Kconfig

@@ -972,6 +972,17 @@ config VIDEO_UDA1342
 	  To compile this driver as a module, choose M here: the
 	  module will be called uda1342.
 
+config VIDEO_IMX477
+	tristate "Sony IMX477 sensor support"
+	depends on I2C && VIDEO_DEV
+	select VIDEO_V4L2_SUBDEV_API
+	help
+	  This is a Video4Linux2 sensor driver for the Sony
+	  IMX477 camera. Also supports the Sony IMX378.
+
+	  To compile this driver as a module, choose M here: the
+	  module will be called imx477.
+
 config VIDEO_VP27SMPX
 	tristate "Panasonic VP27's internal MPX"
 	depends on VIDEO_DEV && I2C

drivers/media/i2c/Makefile

@@ -59,6 +59,7 @@ obj-$(CONFIG_VIDEO_IMX335) += imx335.o
 obj-$(CONFIG_VIDEO_IMX355) += imx355.o
 obj-$(CONFIG_VIDEO_IMX412) += imx412.o
 obj-$(CONFIG_VIDEO_IMX415) += imx415.o
+obj-$(CONFIG_VIDEO_IMX477) += imx477.o
 obj-$(CONFIG_VIDEO_IR_I2C) += ir-kbd-i2c.o
 obj-$(CONFIG_VIDEO_ISL7998X) += isl7998x.o
 obj-$(CONFIG_VIDEO_KS0127) += ks0127.o