"ValueError: cannot convert float NaN to integer" within dead pixel correction
EnPT was executed within the GUI on Linux:
QGIS version: 3.28.3-Firenze
Qt version: 5.15.8
Python version: 3.10.8
GDAL version: 3.6.2
GEOS version: 3.11.1-CAPI-1.17.1
PROJ version: Rel. 9.1.1, December 1st, 2022
PDAL version: 2.5.0 (git-version: d98e35)
Algorithm started at: 2023-02-28T23:16:10
Algorithm 'EnPT - EnMAP Processing Tool (v0.7.4)' starting…
Input parameters:
{ 'CPUs' : 32, 'auto_download_ecmwf' : False, 'average_elevation' : 0, 'blocksize' : 100, 'deadpix_P_algorithm' : 1, 'deadpix_P_interp_spatial' : 0, 'deadpix_P_interp_spectral' : 0, 'disable_progress_bars' : True, 'drop_bad_bands' : True, 'enable_ac' : True, 'json_config' : '', 'mode_ac' : 2, 'n_lines_to_append' : None, 'ortho_resampAlg' : 2, 'output_dir' : '/home/gfz-fe/EnPT/Output/20230228__1', 'output_format' : 1, 'output_interleave' : 0, 'path_dem' : '/home/gfz-fe/scheffler/temp/EnPT/EnVAL_ID_134/geo_abs_elevation.tif', 'path_earthSunDist' : '', 'path_l1b_enmap_image' : '/home/gfz-fe/scheffler/temp/EnPT/EnVAL_ID_134/ENMAP01-____L1B-DT0000001867_20220724T104521Z_007_V010111_20230228T092319Z.ZIP', 'path_l1b_enmap_image_gapfill' : '', 'path_reference_image' : '/home/gfz-fe/scheffler/temp/EnPT/EnVAL_ID_134/geo_abs_ref.tif', 'path_solar_irr' : '', 'polymer_root' : '/home/gfz-fe/scheffler/python/polymer', 'run_deadpix_P' : False, 'scale_factor_boa_ref' : 10000, 'scale_factor_toa_ref' : 10000, 'target_epsg' : None, 'target_projection_type' : 0, 'threads' : -1, 'vswir_overlap_algorithm' : 3, 'working_dir' : '' }
Calling EnPT with the following command:
enpt --CPUs 32 --auto_download_ecmwf False --average_elevation 0 --blocksize 100 --deadpix_P_algorithm spatial --deadpix_P_interp_spatial linear --deadpix_P_interp_spectral linear --disable_progress_bars True --drop_bad_bands True --enable_ac True --mode_ac combined --ortho_resampAlg gauss --output_dir /home/gfz-fe/EnPT/Output/20230228__1 --output_format ENVI --output_interleave band --path_dem /home/gfz-fe/scheffler/temp/EnPT/EnVAL_ID_134/geo_abs_elevation.tif --path_l1b_enmap_image /home/gfz-fe/scheffler/temp/EnPT/EnVAL_ID_134/ENMAP01-____L1B-DT0000001867_20220724T104521Z_007_V010111_20230228T092319Z.ZIP --path_reference_image /home/gfz-fe/scheffler/temp/EnPT/EnVAL_ID_134/geo_abs_ref.tif --polymer_root /home/gfz-fe/scheffler/python/polymer --run_deadpix_P False --scale_factor_boa_ref 10000 --scale_factor_toa_ref 10000 --target_projection_type UTM --threads -1 --vswir_overlap_algorithm swir_only
The log messages of the EnMAP processing tool are written to the *.log file in the specified output folder.
2023/02/28 23:16:18: Reading metadata for VNIR detector...
2023/02/28 23:16:19: Reading metadata for SWIR detector...
2023/02/28 23:16:19: Reading image masks in VNIR sensor geometry.
2023/02/28 23:16:22: Converting DN values to radiance [mW/m^2/sr/nm] for VNIR detector...
2023/02/28 23:16:23: Converting DN values to radiance [mW/m^2/sr/nm] for SWIR detector...
2023/02/28 23:16:25: Computing SNR from VNIR TOA radiance.
2023/02/28 23:16:35: Computing SNR from SWIR TOA radiance.
2023/02/28 23:16:37: Computing VNIR geolayer...
2023/02/28 23:16:40: Computing SWIR geolayer...
2023/02/28 23:17:00: Statically using band 40 for co-registration.
2023/02/28 23:17:00: Temporarily transforming EnMAP band 40 to map geometry for co-registration.
2023/02/28 23:17:09: Temporarily transforming EnMAP water mask to map geometry for co-registration.
2023/02/28 23:17:12: Preparing reference image for co-registration.
2023/02/28 23:17:14: Computing tie points between the EnMAP image and the given spatial reference image.
Calculating footprint polygon and actual data corner coordinates for reference image...
Bounding box of calculated footprint for reference image:
(365670.0, 5812395.0, 403785.0, 5849100.0)
Bounding box of calculated footprint for image to be shifted:
(365694.83596754586, 5812421.986557503, 403766.8580051669, 5849070.490111086)
Matching window position (X,Y): 384728.9599848737/5830747.846876609
Initializing tie points grid...
With respect to the provided bad data mask(s) 26 points of initially 2589 have been excluded.
Calculating tie point grid (2563 points) using 32 CPU cores...
Found 2392 matches.
Performing validity checks...
230 tie points flagged by level 1 filtering (reliability).
45 tie points flagged by level 2 filtering (SSIM).
237 tie points flagged by level 3 filtering (RANSAC)
1912 valid tie points remain after filtering.
2023/02/28 23:17:26: Generating misregistration array.
interpolation runtime: 12.19s
interpolation runtime: 12.24s
2023/02/28 23:17:51: Transforming spatial optimization results back to sensor geometry.
/home/gfz-fe/mambaforge/envs/enpt_full/lib/python3.10/site-packages/pyresample/kd_tree.py:384: UserWarning: Possible more than 8 neighbours within 30 m for some data points
warnings.warn(('Possible more than %s neighbours '
/home/gfz-fe/mambaforge/envs/enpt_full/lib/python3.10/site-packages/pyresample/kd_tree.py:384: UserWarning: Possible more than 8 neighbours within 30 m for some data points
warnings.warn(('Possible more than %s neighbours '
2023/02/28 23:18:00: Applying results of spatial optimization to geolayer.
2023/02/28 23:18:03: Pre-processing DEM for VNIR...
2023/02/28 23:18:03: Transforming DEM to VNIR sensor geometry (using first band of VNIR geolayer)...
2023/02/28 23:18:04: Pre-processing DEM for SWIR...
2023/02/28 23:18:04: Transforming DEM to SWIR sensor geometry (using first band of SWIR geolayer)...
2023/02/28 23:18:06: Transforming the 'mask_landwater' attribute from VNIR to SWIR sensor geometry.
2023/02/28 23:18:11: Starting atmospheric correction for VNIR and SWIR detector in 'combined' mode. Source radiometric unit code is 'TOARad'.
2023/02/28 23:18:11: Setting up forward operator...
2023/02/28 23:18:13: SICOR is applied to land pixels only. This may result in edge effects, e.g., at coastlines...
2023/02/28 23:18:18: Transforming VNIR data to SWIR sensor geometry to enable first guess retrievals for liquid water and ice...
2023/02/28 23:18:27: Getting observation metadata...
2023/02/28 23:18:27: Checking if observation metadata values are within LUT value ranges...
2023/02/28 23:18:27: Loading solar irradiance model...
2023/02/28 23:18:27: Loading RT LUT...
2023/02/28 23:18:27: LUT file was properly downloaded and is available for AC!
2023/02/28 23:18:27: Converting LUT to Fontenla solar model...
2023/02/28 23:18:28: Resampling LUT to EnMAP wavelengths...
2023/02/28 23:18:55: Reducing grid dimensionality of LUT to increase interpolation speed...
2023/02/28 23:18:55: Calculating absorption coefficients of liquid water and ice...
2023/02/28 23:18:56: Transforming CWV first guess map to SWIR sensor geometry to enable segmentation and 3 phases of water retrieval...
2023/02/28 23:19:00: Calculating first guess for intercept and slope of absorption feature continuum...
2023/02/28 23:19:00: Segmenting SWIR L1B spectra to enhance processing speed...
2023/02/28 23:19:04: Preparing segmented SWIR L1B data cube...
2023/02/28 23:19:12: Performing 3 phases of water retrieval...
2023/02/28 23:19:12: Setting up multiprocessing...
2023/02/28 23:19:12: Multiprocessing on 32 cpus
2023/02/28 23:19:12: Preparing optimal estimation input...
2023/02/28 23:19:12: Optimization...
2023/02/28 23:19:20: Done!
2023/02/28 23:19:20: Runtime: 8.47 s
2023/02/28 23:19:20: Extrapolating CWV retrieval map to full SWIR data cube...
2023/02/28 23:19:21: Transforming CWV retrieval map and segmentation labels to VNIR sensor geometry to enable VNIR segmentation...
2023/02/28 23:19:29: Segmenting VNIR data cube according to transformed SWIR segmentation labels...
2023/02/28 23:19:31: Starting surface reflectance retrieval...
2023/02/28 23:19:31: Calculating surface reflectance for vnir detector...
2023/02/28 23:19:31: Calculating surface reflectance for swir detector...
2023/02/28 23:19:31: Applying empirical line solution to extrapolate L2A data pixelwise...
2023/02/28 23:19:31: Extrapolating L2A spectra for vnir detector...
2023/02/28 23:19:38: Extrapolating L2A spectra for swir detector...
2023/02/28 23:19:45: Extrapolating liquid water and ice maps...
2023/02/28 23:19:49: VNIR L2A data contain negative values indicating an overcorrection. Please check for errors in the input data, the options file, or the processing code.
2023/02/28 23:19:49: VNIR L2A data contain values exceeding 1 indicating a saturation. Please check for errors in the input data, the options file, or the processing code.
2023/02/28 23:19:51: SWIR L2A data contain negative values indicating an overcorrection. Please check for errors in the input data, the options file, or the processing code.
2023/02/28 23:19:52: SWIR L2A data contain values exceeding 1 indicating a saturation. Please check for errors in the input data, the options file, or the processing code.
2023/02/28 23:19:52: SICOR atmospheric correction for EnMAP in combined mode successfully finished!
vnir
2023/02/28 23:19:52: Running polymer...
2023/02/28 23:19:52: Currently only tested with ERA5 data, i.e. set True for "auto_download_ecmwf"
Initializing ENMAP product of size (1024, 1000)
2023/02/28 23:19:52: Landmask is EnMAP L1B
Download 2022-07-24 10:00:00 -> /home/gfz-fe/scheffler/python/polymer/ANCILLARY/ERA5/2022/07/24/era5_20220724_100000.nc
2023-02-28 23:19:53,012 INFO Welcome to the CDS
2023-02-28 23:19:53,012 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels
2023-02-28 23:19:53,266 INFO Request is queued
2023-02-28 23:19:54,310 INFO Request is running
2023-02-28 23:19:55,854 INFO Request is completed
2023-02-28 23:19:55,855 INFO Downloading https://download-0000-clone.copernicus-climate.eu/cache-compute-0000/cache/data6/adaptor.mars.internal-1677622794.747088-20076-11-6bbabe83-8d5a-4b0e-8963-a6bb1eeb68f9.nc to /home/gfz-fe/scheffler/python/polymer/ANCILLARY/ERA5/2022/07/24/era5_20220724_100000.nc.tmp (9.9M)
0%| | 0.00/9.91M [00:00<?, ?B/s] 1%| | 78.0k/9.91M [00:00<00:14, 726kB/s] 7%|â | 754k/9.91M [00:00<00:02, 3.84MB/s] 32%|ââââ | 3.14M/9.91M [00:00<00:00, 12.9MB/s] 62%|âââââââ | 6.15M/9.91M [00:00<00:00, 19.9MB/s] 87%|âââââââââ | 8.58M/9.91M [00:00<00:00, 21.9MB/s] 2023-02-28 23:19:56,694 INFO Download rate 11.8M/s
Download 2022-07-24 11:00:00 -> /home/gfz-fe/scheffler/python/polymer/ANCILLARY/ERA5/2022/07/24/era5_20220724_110000.nc
2023-02-28 23:19:56,780 INFO Welcome to the CDS
2023-02-28 23:19:56,780 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels
2023-02-28 23:19:56,897 INFO Request is queued
2023-02-28 23:19:57,942 INFO Request is running
2023-02-28 23:19:59,487 INFO Request is completed
2023-02-28 23:19:59,488 INFO Downloading https://download-0001-clone.copernicus-climate.eu/cache-compute-0001/cache/data2/adaptor.mars.internal-1677622797.8498647-11507-3-392d1195-e895-4600-b6db-3147f22fd704.nc to /home/gfz-fe/scheffler/python/polymer/ANCILLARY/ERA5/2022/07/24/era5_20220724_110000.nc.tmp (9.9M)
0%| | 0.00/9.91M [00:00<?, ?B/s] 1%| | 78.0k/9.91M [00:00<00:13, 740kB/s] 7%|â | 751k/9.91M [00:00<00:02, 4.04MB/s] 35%|ââââ | 3.48M/9.91M [00:00<00:00, 14.8MB/s] 70%|âââââââ | 6.95M/9.91M [00:00<00:00, 23.1MB/s] 2023-02-28 23:20:00,244 INFO Download rate 13.1M/s
Starting processing at 2023-02-28 23:20:00.527176
Processing block: size (100, 1000), offset (0, 0)
Processing block: size (100, 1000), offset (100, 0)
Processing block: size (100, 1000), offset (200, 0)
Processing block: size (100, 1000), offset (300, 0)
Processing block: size (100, 1000), offset (400, 0)
Processing block: size (100, 1000), offset (500, 0)
Processing block: size (100, 1000), offset (600, 0)
Processing block: size (100, 1000), offset (700, 0)
Processing block: size (100, 1000), offset (800, 0)
Processing block: size (100, 1000), offset (900, 0)
Processing block: size (24, 1000), offset (1000, 0)
Done in 0:00:53.595420
2023/02/28 23:20:59: Joining results of atmospheric correction.
2023/02/28 23:21:01: Re-applying dead pixel correction to correct for spectral spikes due to fringe effect.
2023/02/28 23:21:01: Correcting dead pixels of VNIR detector...
Used algorithm: linear interpolation in the linear domain
2023/02/28 23:21:04: Percentage of defective pixels: 0.18
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/gfz-fe/mambaforge/envs/enpt_full/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/gfz-fe/mambaforge/envs/enpt_full/lib/python3.10/multiprocessing/pool.py", line 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
File "/home/gfz-fe/scheffler/python/EnPT/enpt/processors/dead_pixel_correction/dead_pixel_correction.py", line 349, in interp_nodata_spatially_2d
data_2d[np.isnan(data_2d)] = fill_value
ValueError: cannot convert float NaN to integer
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/gfz-fe/mambaforge/envs/enpt_full/bin/enpt", line 33, in <module>
sys.exit(load_entry_point('enpt', 'console_scripts', 'enpt')())
File "/home/gfz-fe/scheffler/python/EnPT/enpt/cli.py", line 227, in main
parsed_args.func(get_config(parsed_args))
File "/home/gfz-fe/scheffler/python/EnPT/enpt/cli.py", line 195, in run_job
CTR.run_all_processors()
File "/home/gfz-fe/scheffler/python/EnPT/enpt/execution/controller.py", line 189, in run_all_processors
self.L1_obj.correct_dead_pixels()
File "/home/gfz-fe/scheffler/python/EnPT/enpt/model/images/images_sensorgeo.py", line 753, in correct_dead_pixels
self.vnir.correct_dead_pixels()
File "/home/gfz-fe/scheffler/python/EnPT/enpt/model/images/images_sensorgeo.py", line 132, in correct_dead_pixels
.correct(self.data, self.deadpixelmap)
File "/home/gfz-fe/scheffler/python/EnPT/enpt/processors/dead_pixel_correction/dead_pixel_correction.py", line 182, in correct
return self._interpolate_nodata_spatially(image2correct, deadpixel_map)
File "/home/gfz-fe/scheffler/python/EnPT/enpt/processors/dead_pixel_correction/dead_pixel_correction.py", line 127, in _interpolate_nodata_spatially
image_sub_interp = interp_nodata_spatially_3d(image_sub, axis=1, nodata=deadpixel_map_sub, **kw)
File "/home/gfz-fe/scheffler/python/EnPT/enpt/processors/dead_pixel_correction/dead_pixel_correction.py", line 387, in interp_nodata_spatially_3d
results = pool.starmap(interp_nodata_spatially_2d, args)
File "/home/gfz-fe/mambaforge/envs/enpt_full/lib/python3.10/multiprocessing/pool.py", line 375, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
File "/home/gfz-fe/mambaforge/envs/enpt_full/lib/python3.10/multiprocessing/pool.py", line 774, in get
raise self._value
ValueError: cannot convert float NaN to integer
============================================================
An exception occurred. Processing failed.
The output folder '/home/gfz-fe/EnPT/Output/20230228__1' contains:
ENMAP01-____L1B-DT0000001867_20220724T104521Z_007_V010111_20230228T092319Z.log
No output raster was written.
Execution completed in 300.17 seconds (5 minutes 0 seconds)
Results:
{'success': False}Edited by Daniel Scheffler