User avatar
schoolpost
Posts: 96
Joined: Sun Feb 19, 2017 10:47 am
Location: Canada
Contact: Website

PyDNG - RPI Camera RAW to DNG

Tue Apr 11, 2017 5:52 am

Hello Everyone!

Excited to share this small project I've been working on:
https://github.com/schoolpost/pydng

(raw left, jpeg right )
Image

Requires: Python 2.7 & Numpy

PyDNG lets you convert your picamera RAW containing .jpgs into Adobe DNG RAW files
for use in a wide variety of image editing applications.
( Photoshop, Lightroom, RawTherapee, etc.. )

Bayer data gives you the most flexibility to really tune your image for the best results.
Paired with Lossless compression when output from Adobe DNG Converter
or 3rd Party application like SlimRAW & enjoy an ~2:1 compression at no cost! ( 4-6MB )

Features:
-10/16-bit encoding options
-Flat field correction ( lens shading )
-Hot pixel correction
-Dark frame subtraction

Some future considerations:
-Copy EXIF Data from .jpg to .dng
-Fix magenta tinted highlights
-Improve 10bit/8bit encoding in DNG_10.py
-Apply calibrated color matrix values
-Automated calibration script to let users generate dark/shade/pxl from input images

While the output is a mostly functional dng, for best compatbillity users should convert their DNG's through AdobeDNGConverter to fix any problems. ( and get some free lossless compression as a bonus )

Hopefully this is of some use to somebody out there.
-Csaba Nagy

User avatar
schoolpost
Posts: 96
Joined: Sun Feb 19, 2017 10:47 am
Location: Canada
Contact: Website

Re: PyDNG - RPI Camera RAW to DNG

Tue Apr 09, 2019 2:52 am

Wanted to share my latest update to this project. Version 2.0!

https://github.com/schoolpost/pydng

Store 10-bit bayer data from the PiCamera V2 in the most widely used RAW image format available.

This version improves on all aspects of the original, with improved pre-processing, file encoding and new lossless compression!

Would love for you all to try it, there are aspects still in need of development for those who wish to contribute:
  • Use camera specific color calibration matrix ( the one currently used looks good but is not by any means accurate) ,
  • Detect flipped / rotated images automatically ( change CFA layout based on rotation )
  • Detecting/Mapping dead/hot pixels
  • Copy EXIF data from jpeg to dng tags
  • Improve tile based LJ92 compression ( for even further file size reduction )
10-bit DNG File Size ( 7,201,848 ) vs. 8-bit JPEG File Size ( 3,805,122 )
Image

Thanks for viewing! :)
-Csaba Nagy

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11247
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: PyDNG - RPI Camera RAW to DNG

Tue Apr 09, 2019 9:46 am

Detect flipped / rotated images automatically ( change CFA layout based on rotation )
I can't see this in code with a quick look, but I do see hard coded offsets at https://github.com/schoolpost/pydng/blo ... ng.py#L338
Captures can be done in any sensor mode, therefore there are multiple valid offsets. See https://github.com/6by9/dcraw/blob/master/dcraw.c#L8162

The useful parts of the raw header are "documented" at https://github.com/6by9/dcraw/blob/master/dcraw.c#L6465, and that includes the absolute Bayer order and resolution for the capture - no need to be guessing it based on rotation or offset.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

seasalt
Posts: 5
Joined: Thu Jan 16, 2020 5:48 pm

Re: PyDNG - RPI Camera RAW to DNG

Thu Jan 16, 2020 6:23 pm

* I havn't had an in depth look at the code but to my knowledge, jpg's are compressed and by converting them to dng, you are not uncompressing them, therefore they can not be true raw files.

Am I wrong in making this assumption?

User avatar
mahjongg
Forum Moderator
Forum Moderator
Posts: 13973
Joined: Sun Mar 11, 2012 12:19 am
Location: South Holland, The Netherlands

Re: PyDNG - RPI Camera RAW to DNG

Thu Jan 16, 2020 6:28 pm

seasalt wrote:
Thu Jan 16, 2020 6:23 pm
* I havn't had an in depth look at the code but to my knowledge, jpg's are compressed and by converting them to dng, you are not uncompressing them, therefore they can not be true raw files.

Am I wrong in making this assumption?
No,
JPG's are not a "lossless" format, meaning there is data-loss when compressing, data that can never be recovered.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11247
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: PyDNG - RPI Camera RAW to DNG

Thu Jan 16, 2020 8:35 pm

raspistill with the -r flag appends the raw image to the end of the JPEG. This was done originally as a way of getting raw images out of Android where the camera HAL only allowed you to capture JPEGs. Any JPEG viewer will ignore the data after the EOI flag in the JPEG.

So, yes, this project can produce a DNG from a JPEG(+RAW) file captured with raspistill.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
schoolpost
Posts: 96
Joined: Sun Feb 19, 2017 10:47 am
Location: Canada
Contact: Website

Re: PyDNG - RPI Camera RAW to DNG

Mon May 11, 2020 3:58 am

New version is now available! With support for the new Raspberry Pi High Quality Camera ( IMX477 )
https://github.com/schoolpost/PyDNG

Lots of fixes and refactoring has been done. Among other things we have:

  • support for automatic bayer format order detection for flipped images
  • attached exif data ( exposure, lens, shutter, etc.. )

Simple example script included to convert input file ( JPEG+RAW ) into a DNG file.

Code: Select all


python examples/utility.py color.jpg


Another example available demonstrates how to integrate into common picamera workflow you would do in python.

Code: Select all


import picamera
from io import BytesIO
from pydng.core import RPICAM2DNG



with picamera.PiCamera() as camera:

    stream = BytesIO()
    camera.capture(stream, 'jpeg', bayer=True)

    d = RPICAM2DNG()
    output = d.convert(stream)

    with open('image.dng', 'wb') as f:
            f.write(output)


-Csaba Nagy

cantsin
Posts: 8
Joined: Tue Jun 23, 2020 8:26 am

Re: PyDNG - RPI Camera RAW to DNG

Tue Jun 23, 2020 8:37 am

Thanks for your work!

On Debian 10, I didn't manage to get PyDNG installed - first, the installer complains that Python is not version 3 (indeed Debian symlinks /usr/bin/python to Python 2.6 even when Python 3 is installed):

Code: Select all

pydng requires Python '>=3.5' but the running Python is 2.7.16
...which I couldn't even fix by relinking /usr/bin/python to /usr/bin/python3.7. So I 'hacked' it by lowering the version requirement in setup.py. However, then I get a gcc error when trying to compile the included jpeg C library:

Code: Select all

pydng/bitunpack.c:99:12: warning: implicit declaration of function ‘PyModul$
_Create’; did you mean ‘PyModule_Check’? [-Wimplicit-function-declaration]      
         return PyModule_Create(&ljpegCompress);                                
                ^~~~~~~~~~~~~~~                                                 
                PyModule_Check                                                  
    pydng/bitunpack.c:99:12: warning: ‘return’ with a value, in function return$
ng void                                                                         
         return PyModule_Create(&ljpegCompress);                                
                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    pydng/bitunpack.c:97:16: note: declared here
     PyMODINIT_FUNC PyInit_ljpegCompress(void)
                    ^~~~~~~~~~~~~~~~~~~~
    pydng/bitunpack.c: At top level:
    pydng/bitunpack.c:88:27: error: storage size of ‘ljpegCompress’ isn’t known
     static struct PyModuleDef ljpegCompress =
                               ^~~~~~~~~~~~~
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Would you know a fix?

User avatar
schoolpost
Posts: 96
Joined: Sun Feb 19, 2017 10:47 am
Location: Canada
Contact: Website

Re: PyDNG - RPI Camera RAW to DNG

Thu Jul 02, 2020 1:32 am

cantsin wrote:
Tue Jun 23, 2020 8:37 am
Thanks for your work!

On Debian 10, I didn't manage to get PyDNG installed - first, the installer complains that Python is not version 3 (indeed Debian symlinks /usr/bin/python to Python 2.6 even when Python 3 is installed):

Code: Select all

pydng requires Python '>=3.5' but the running Python is 2.7.16
...which I couldn't even fix by relinking /usr/bin/python to /usr/bin/python3.7. So I 'hacked' it by lowering the version requirement in setup.py. However, then I get a gcc error when trying to compile the included jpeg C library:

Code: Select all

pydng/bitunpack.c:99:12: warning: implicit declaration of function ‘PyModul$
_Create’; did you mean ‘PyModule_Check’? [-Wimplicit-function-declaration]      
         return PyModule_Create(&ljpegCompress);                                
                ^~~~~~~~~~~~~~~                                                 
                PyModule_Check                                                  
    pydng/bitunpack.c:99:12: warning: ‘return’ with a value, in function return$
ng void                                                                         
         return PyModule_Create(&ljpegCompress);                                
                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    pydng/bitunpack.c:97:16: note: declared here
     PyMODINIT_FUNC PyInit_ljpegCompress(void)
                    ^~~~~~~~~~~~~~~~~~~~
    pydng/bitunpack.c: At top level:
    pydng/bitunpack.c:88:27: error: storage size of ‘ljpegCompress’ isn’t known
     static struct PyModuleDef ljpegCompress =
                               ^~~~~~~~~~~~~
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Would you know a fix?
I will try to look into this, that JPEG library is for JPEG 1992 Lossless compression support which I acutally have off by default for now while I do some more verification and testing with it.

The "hack" way to fix would be to remove the dependancy for the library from the setup.py and any references to it in the src code ( i.e the imports in the python files )
-Csaba Nagy

User avatar
schoolpost
Posts: 96
Joined: Sun Feb 19, 2017 10:47 am
Location: Canada
Contact: Website

Re: PyDNG - RPI Camera RAW to DNG

Thu Jul 02, 2020 1:33 am

New Version Release, 3.1.0 with a bunch of fixes. More info and download here:
https://github.com/schoolpost/PyDNG
-Csaba Nagy

User avatar
schoolpost
Posts: 96
Joined: Sun Feb 19, 2017 10:47 am
Location: Canada
Contact: Website

Re: PyDNG - RPI Camera RAW to DNG

Sun Jul 12, 2020 5:22 am

New Version Release, 3.4.0 with new features and bug fixes. https://github.com/schoolpost/PyDNG
  • Raspberry Pi HQ Camera Calibrated Color Matrices. ( much better color! )
  • Processing Pipeline to manipulate the bayer data before writing to file ( for custom application like hot pixel correction, dark frame subtraction, etc..)
  • RAW2DNG Class, write any Bayer RAW data from any camera into DNG file. ( examples included )
In my own testing DNG certainly opened up a lot of potential for the Raspberry Pi High Quality Camera. Below are some of the highlights that make both DNG and the native JPEG formats strong.

DNG:
  • Widely supported RAW format
  • 12-bit precision
  • White balance control
  • Better Highlight Detail/Recovery
  • More Detail/Image Control
  • Lossless compression!
JPEG:
  • Smaller file size
  • Does not require extra processing/software
  • Built in processing ( noise reduction, lens shading, etc..)
  • Quick!
All in all if you want the most control / highest fidelity image possible you should definitely be using RAW instead of JPEG. Especially for preserving highlight detail and if you wish to perform your own noise reduction ( usually better because it doesn't have to be done so fast like in the ISP )

Some examples below. Top row are DNG's processed in RawTherapee5.8. Bottom row are JPG's straight out of raspistill.

Image
-Csaba Nagy

dabateman
Posts: 34
Joined: Tue Jul 07, 2020 1:51 pm

Re: PyDNG - RPI Camera RAW to DNG

Sun Jul 12, 2020 12:36 pm

Funny to see version 3.4. I was just trying to get Saturdays version 3.3 working and I still have errors.
To install version 3.3 I needed to delete these lines from setup file:

ext_modules = [ljpeg92],
python_requires='>=3.5',

Otherwise it wouldn't install and gave a you have python 2.x.x error every time.
Now when trying to extract a raw from jpeg I get the 1jpegcompress error. Even though I thought compression was off.

Do I need to install the jpeg92 files?
Or do I need to delete line 14 from core.py which indicates to pull from 1jpegcompress?
Is there anywhere else where I would need to delete the compression code?

Did Raw therapee support the DNG files? Where they compressed?

I will edit in either Raw therapee or ART so good to know if its all working.

I would love if raspistill could just output dng files. It would save these extra steps. I am hoping to write a script that will convert the jpeg files collected to dng, similar to the upload to onedrive script. I have dcraw working so thats an alternative, but would need specific scripts depending on my file sets as dcraw processes the images.

Still trying just to get this to work.

CanPi
Posts: 64
Joined: Wed May 20, 2020 12:21 pm

Re: PyDNG - RPI Camera RAW to DNG

Sun Jul 12, 2020 2:50 pm

RT 5.8 could not open compressed DNGs for me, uncompressed worked fine.

User avatar
schoolpost
Posts: 96
Joined: Sun Feb 19, 2017 10:47 am
Location: Canada
Contact: Website

Re: PyDNG - RPI Camera RAW to DNG

Sun Jul 12, 2020 6:40 pm

dabateman wrote:
Sun Jul 12, 2020 12:36 pm
Funny to see version 3.4. I was just trying to get Saturdays version 3.3 working and I still have errors.
To install version 3.3 I needed to delete these lines from setup file:

ext_modules = [ljpeg92],
python_requires='>=3.5',

Otherwise it wouldn't install and gave a you have python 2.x.x error every time.
Now when trying to extract a raw from jpeg I get the 1jpegcompress error. Even though I thought compression was off.

Do I need to install the jpeg92 files?
Or do I need to delete line 14 from core.py which indicates to pull from 1jpegcompress?
Is there anywhere else where I would need to delete the compression code?

Did Raw therapee support the DNG files? Where they compressed?

I will edit in either Raw therapee or ART so good to know if its all working.

I would love if raspistill could just output dng files. It would save these extra steps. I am hoping to write a script that will convert the jpeg files collected to dng, similar to the upload to onedrive script. I have dcraw working so thats an alternative, but would need specific scripts depending on my file sets as dcraw processes the images.

Still trying just to get this to work.

I will update my source to only import the ljpegcompress module if you specify using compression. This should help, but the fact you cannot get this module to compile isn't ideal.

when you do the install try using:

Code: Select all

pip3 install src/.

This will make sure you are installing with Python 3.
-Csaba Nagy

dherrada
Posts: 2
Joined: Tue Jul 21, 2020 8:48 pm

Re: PyDNG - RPI Camera RAW to DNG

Fri Jul 24, 2020 6:26 pm

Just want to thank you for making this. It works incredibly well. I tried so many libraries but none worked with the hq camera except this one.

DrawSprocket
Posts: 1
Joined: Mon Jan 11, 2021 9:20 pm

Re: PyDNG - RPI Camera RAW to DNG

Mon Jan 11, 2021 9:23 pm

Ah, very exciting! I am new to both terminals and even newer to PyDNG. Is there a way to run this script on multiple files at once? Or can it run a whole folder? Either would be fantastic. Thanks for such a great utility!

Theorem 3D
Posts: 1
Joined: Tue Mar 16, 2021 2:10 pm

Re: PyDNG - RPI Camera RAW to DNG

Tue Mar 16, 2021 2:18 pm

Thank you for writing this utility! How can I specify to convert to 10-bit? Currently when opening my DNG in Photoshop it shows my files are 8-bit. Currently I am using the included example "utility.py".

raspberry-photon
Posts: 66
Joined: Fri Jul 31, 2020 5:58 pm

Re: PyDNG - RPI Camera RAW to DNG

Fri Jun 11, 2021 4:21 pm

I want to mention that pydng doesn't work with sensor mode 2.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11247
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: PyDNG - RPI Camera RAW to DNG

Fri Jun 11, 2021 5:15 pm

raspberry-photon wrote:
Fri Jun 11, 2021 4:21 pm
I want to mention that pydng doesn't work with sensor mode 2.
On which sensor?
https://github.com/schoolpost/PyDNG/issues
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

raspberry-photon
Posts: 66
Joined: Fri Jul 31, 2020 5:58 pm

Re: PyDNG - RPI Camera RAW to DNG

Fri Jun 11, 2021 8:15 pm

Imx 477, I haven't tried whether it works with the pixel binning mode of the Imx 219.

dabateman
Posts: 34
Joined: Tue Jul 07, 2020 1:51 pm

Re: PyDNG - RPI Camera RAW to DNG

Sun Jun 13, 2021 1:35 pm

@Schoolpost.
Your program continues to prove to be outstanding.
A question now I have after playing and reading through the code. After version 3.4 does pyDNG apply a color correction to the output dng file from the HQ 477 sensor file?

I now have a monochrome converted HQ camera from MaxMax (LDP). Is it possible to modify the dng output to not offer any correct and provide the raw dng file as pure monochrome?

So that a mono switch can be added to the run script that wouldn't do anything color correction and provide highest resolution?

User avatar
schoolpost
Posts: 96
Joined: Sun Feb 19, 2017 10:47 am
Location: Canada
Contact: Website

Re: PyDNG - RPI Camera RAW to DNG

Sun Jun 13, 2021 7:41 pm

dabateman wrote:
Sun Jun 13, 2021 1:35 pm
@Schoolpost.
Your program continues to prove to be outstanding.
A question now I have after playing and reading through the code. After version 3.4 does pyDNG apply a color correction to the output dng file from the HQ 477 sensor file?

I now have a monochrome converted HQ camera from MaxMax (LDP). Is it possible to modify the dng output to not offer any correct and provide the raw dng file as pure monochrome?

So that a mono switch can be added to the run script that wouldn't do anything color correction and provide highest resolution?

This could potentially be added, but given monochrome HQ camera's aren't very widespread it wouldn't be a top priority at the moment.

As for how you could achieve this, you would need to change the photometricinterpretation tag of the DNG file to something other than CFA ( Color filter array ) Probably want to change that value to 0x01? ( BlackIsZero )

https://www.awaresystems.be/imaging/tif ... ation.html

You can do this without editing the source for PyDNG by doing the DNG encoding yourself as in this example:
https://github.com/schoolpost/PyDNG/blo ... dng.py#L32


Otherwise you'd want to change the source in this file:
https://github.com/schoolpost/PyDNG/blo ... re.py#L387
-Csaba Nagy

dabateman
Posts: 34
Joined: Tue Jul 07, 2020 1:51 pm

Re: PyDNG - RPI Camera RAW to DNG

Mon Jun 14, 2021 5:50 am

Yes thank you.
I didn't mean to imply you should. Sorry I should have been more clear.
I was going to start cutting it up myself and call it pyDNGmono. So that I could add that script to my output program.
Thank you this saves a lot of trial and error. Mostly error on my side.
David

Return to “Camera board”