Howdy does not currenty support that, but has a concept called "recorders". Every recorder is responsible for getting the video feed from the configured video device and delivering it to OpenCV. If you're interested i'm sure it's possible to write a recorder for your usecase and add it to v3.0.0

How does the "recorder" concept work - does it depend on capturing video stream into a file and then processing it?
Could you please point me to the documentation/howto for it?
MS Surface Pro tablets will soon get a support for IR sensor so being able to use Howdy on Surface would be a dream come true...
Thanks!

Hi,
I'm also facing this issue.
Did you solve this?

I'm looking into how I can integrate this into /etc/pam.d/common-auth, but are struggling a bit :-)

Dan Scally on Surface Linux camera project was working on libcamera support for Howdy, but that was maybe 6 months ago and then there was silence...
Needless to say, we wont have Howdy on any recent built-in cameras until there is libcamera support in Howdy. And/or gstreamer or pipewire, once they have libcamera support. OK so Im not being very exact but something like that. Lower level lib/HW support > Higher level lib abstraction layer > Applications.
Until this is a reality, we need libcamera support in Howdy. Sigh.

Dan Scally on Surface Linux camera project was working on libcamera support for Howdy, but that was maybe 6 months ago and then there was silence... Needless to say, we wont have Howdy on any recent built-in cameras until there is libcamera support in Howdy. And/or gstreamer or pipewire, once they have libcamera support. OK so Im not being very exact but something like that. Lower level lib/HW support > Higher level lib abstraction layer > Applications. Until this is a reality, we need libcamera support in Howdy. Sigh.

I might have a workaround for this. I have made a systemd service for this command: gst-launch-1.0 libcamerasrc camera-name='\SB.PCI0.I2C2.CAMF' !
video/x-raw,width=1280,height=720,framerate=30/1,format=NV12
! videoconvert ! video/x-raw,format=YUY2 ! queue !
v4l2sink device=/dev/video42

I just need to find a way to start the service through pam, before triggering Howdy.
The plan is to stop the service again after a successful login, so that the webcam isn't on constantly.

Dan Scally on Surface Linux camera project was working on libcamera support for Howdy, but that was maybe 6 months ago and then there was silence...

Yeah sorry; I haven't had much spare time lately (my wife had our third child 6 and a half months ago; from one kid to two was a doddle but it turns out that going from two to three really kills your time). I am actively working on both sides of this problem though; extending libcamera to handle the IR cameras on IPU3 devices and adding a libcamera recorder to Howdy. I recognise you've been waiting a long time @afalout - I'll try to get everything working even if not upstreamable and share it.

I might have a workaround for this. I have made a systemd service for this command: gst-launch-1.0 libcamerasrc camera-name='\SB.PCI0.I2C2.CAMF' ! video/x-raw,width=1280,height=720,framerate=30/1,format=NV12 ! videoconvert ! video/x-raw,format=YUY2 ! queue ! v4l2sink device=/dev/video42

I just need to find a way to start the service through pam, before triggering Howdy. The plan is to stop the service again after a successful login, so that the webcam isn't on constantly.

This kind of thing should work in the interim as long as you don't mind not using the IR camera.

Yeah sorry; I haven't had much spare time lately (my wife had our third child 6 and a half months ago; from one kid to two was a doddle but it turns out that going from two to three really kills your time). I am actively working on both sides of this problem though; extending libcamera to handle the IR cameras on IPU3 devices and adding a libcamera recorder to Howdy. I recognise you've been waiting a long time @afalout - I'll try to get everything working even if not upstreamable and share it.

I totally understand this. No worries. Take care of your kids and wife :-).

This kind of thing should work in the interim as long as you don't mind not using the IR camera.

Thanks. I'll share the results later.

I've figured it out, but this is quick and dirty! Do this on your own risk! It might be smart to have root access while setting this up and testing it, so that you can disable Howdy if it fails.

Feel free to ask if you have any questions.

1. Set up and make sure that libcamera with v4l2loopback works according to the Wiki from this tutorial: https://github.com/linux-surface/linux-surface/wiki/Camera-Support#gstreamer-loopback-device-for-other-applications
   Certainly! Here’s the revised document with the markdown issues fixed:
2. Set up loopback to video device:

   sudo vi /etc/systemd/system/libcamera_loopback.service
   

3. Paste the following:

   [Unit]
   Description=libcamera loopback
   
   [Service]
   Type=simple
   User=
   #Group=
   ExecStart=/usr/bin/gst-launch-1.0 libcamerasrc camera-name='\\\_SB_.PCI0.I2C2.CAMF' ! video/x-raw,width=1280,height=720,framerate=30/1,format=NV12 ! videoconvert ! video/x-raw,format=YUY2 ! v4l2sink device=/dev/video42
   Restart=on-failure
   
   [Install]
   WantedBy=multi-user.target
   

4. Create a policy kit exception so that it can be run by all users in the system:

   1. sudo vi /etc/polkit-1/rules.d/libcamera.rules
      

   2. Paste the following:

      polkit.addRule(function(action, subject) {
          if (action.id == "org.freedesktop.systemd1.manage-units" &&
              action.lookup("unit") == "libcamera_loopback.service" &&
              (action.lookup("verb") == "start" || action.lookup("verb") == "stop" || action.lookup("verb") == "restart")) {
              return polkit.Result.YES;
          }
      });
      

5. Test that the service works and that the webcam light turns on and off when the service starts and stops:

   sudo systemctl daemon-reload
   sudo systemctl restart polkit
   sudo systemctl start libcamera_loopback.service
   sudo systemctl status libcamera_loopback.service
   sudo systemctl stop libcamera_loopback.service
   

6. Install Howdy according to the instructions.
7. Create scripts for starting and stopping:

   1. sudo vi /bin/start-libcamera-loopback.sh
      

   2. Paste the following, close and save:

      #!/bin/bash
      systemctl start libcamera_loopback.service
      

   3. sudo vi /bin/stop-libcamera-loopback.sh
      

   4. Paste the following, close and save:

      #!/bin/bash
      systemctl stop libcamera_loopback.service
      

   5. sudo chmod  x /bin/*-libcamera-loopback.sh
      

8. Set up PAM with the following:
   1. Place this at the top:

      auth    required    pam_exec.so seteuid /bin/start-libcamera-loopback.sh
      

   2. Place this at the bottom:

      auth    optional     pam_exec.so seteuid /bin/stop-libcamera-loopback.sh
      

Sweet - thanks for that. For what it's worth I tried to hack the work I'd done so far into working and I think it should...but I haven't had time to test it on my surface yet and likely won't now for at least another week (I'll be away at a conference all next week and probably busy prepping until then). The actual image capture through libcamera works fine, the bit I've been stuck on is convincing opencv to actually convert the image buffers that you receive. To work around that I've just set it to slice out the Y plane from the NV12 buffers that the IPU3 produces and treat it as a gray buffer...it ought to work but as I say I haven't actually managed to test it yet.

If you wanted to test this before then (bearing in mind that I haven't...) you'd need to have built the python bindings in libcamera (meson configure -D pycamera=enabled, requires libpython3-dev pybind11-dev) and in config.ini set recording_plugin=libcamera and libcamera_camera_id="your camera id here". The camera IDs you can find using cam -l:

$ cam -l
[2:51:00.296914792] [21301]  INFO Camera camera_manager.cpp:284 libcamera v0.2.0 139-66f0c2fe-dirty (2024-05-09T10:11:19 01:00)
Available cameras:
1: 'USB  Live camera: USB  Live cam' (\_SB_.PCI0.GP17.XHC0.RHUB.PRT3-3:1.0-0c45:6366)
2: Internal front camera (\_SB_.PCI0.GP17.XHC1.RHUB.PRT3-3:1.0-13d3:5458)

Those are USB cameras on my PC...the IDs for the Surface cameras aren't quite so horrific. I'll get the conversion working once I'm back and then at least the non-IR camera would be available.

Much appreciated, Dan.
I will wait for the IR camera as plain photo one is not usable for any form of authentication in real world.
Please keep us updated, your work is very important for a lot of people.
Cheers

Those are USB cameras on my PC...the IDs for the Surface cameras aren't quite so horrific. I'll get the conversion working once I'm back and then at least the non-IR camera would be available.

Hi Dan,

just a quick check if you had time to move this forward?

Thanks
Andrej

Those are USB cameras on my PC...the IDs for the Surface cameras aren't quite so horrific. I'll get the conversion working once I'm back and then at least the non-IR camera would be available.

Hi Dan,

just a quick check if you had time to move this forward?

Thanks Andrej

Yes, some progress - I can capture from the IR camera with the unpacking into a normal greyscale format done by libcamera. The other problem is that since we can't drive the IR LEDs the image is too dark; I have libcamera mostly correcting that now but it's not quite right. Once that's working I'll share it so you can try it out - it still won't be in a finished state until I have time to do a bunch of cleanup, but it should work at that point.

Thanks
Dan

👋 Another Microsoft Surface (Go 3) user following this work. Any updates?

👋 Another Microsoft Surface (Go 3) user following this work. Any updates?

None, but having been prompted by lots of people in the last week to work on this again I will try to drive it forwards

👋 Another Microsoft Surface (Go 3) user following this work. Any updates?

None, but having been prompted by lots of people in the last week to work on this again I will try to drive it forwards

Hi Dan @djrscally - just noted this article that might be of interest?

https://jgrulich.cz/2024/08/19/making-pipewire-default-option-for-firefox-camera-handling/

Thank you for you efforts!
Andrej

👋 Another Microsoft Surface (Go 3) user following this work. Any updates?

None, but having been prompted by lots of people in the last week to work on this again I will try to drive it forwards

Hi Dan @djrscally - just noted this article that might be of interest?

https://jgrulich.cz/2024/08/19/making-pipewire-default-option-for-firefox-camera-handling/

Thank you for you efforts! Andrej

Heh good picture of Kieran there. The pipewire/WebRTC support is not of direct help to getting the IR camera working, but the Software ISP that's mentioned there is what's making the IR camera work too. The problem with how it's set up on the IPU3 (and why this has taken so long) is that the IR camera's data comes out raw (not through the hardware ISP) and packed into a format that's not supported anywhere - the Software ISP is relatively new and enables us to drive exposure automatically (without which it's useless) and also unpack it to a commonly supported format.

I think I just have one bug left; currently auto-exposure drives the exposure to max and leaves it there for some reason. Following that the libcamera reader I added in the link above needs to be updated to accept the Y10 format that the IR camera will write out, and then in theory it should work...

EDIT:

The problem with how it's set up on the IPU3 (and why this has taken so long)

Well, no, that's what made it hard. It took so long because I stopped working on it for various reasons.

Hi Dan - any luck with auto-exposure ??

Thanks!