Demo [SOLVED]

[matteani]

Hi All,

When running the demo script on a (randomly chosen) file from the Early Data Release (https://erosita.mpe.mpg.de/edr/eROSITAO ... ns/EDRFWC/) I always get the same error (see below).

Code: Select all

idies@acb9cac72113:~/demo$ ./demoscript_4EDR.py FWCv1.0_120_EventList.fits 
Results will be saved in the current folder :
	/home/idies/demo
This script will uses eSASS installed in /home/idies/sw/eSASS4EDR
and the executables in:
	/home/idies/sw/eSASS4EDR/bin
 /home/idies/sw/eSASS4EDR/bin/evt: eSASS4EDR Sep 02 09:43:31 2021
 /home/idies/sw/eSASS4EDR/bin/evt/evtool_main: memset==.false., so memory setting for large files is used.
 /home/idies/sw/eSASS4EDR/bin/evt/merge_extensions: Merging 1 files.
 /home/idies/sw/eSASS4EDR/bin/evt/merge_extensions: Infiles will be read 1 extension at a time.
 /home/idies/sw/eSASS4EDR/bin/evt/evtool_main: Applying filters.
 /home/idies/sw/eSASS4EDR/bin/evt/apply_filter_eventfile: Creating image.
 /home/idies/sw/eSASS4EDR/bin/evt/evtool_main: Writing outfile.
 /home/idies/sw/eSASS4EDR/bin/evt/write_tEventFile: Writing events.
 /home/idies/sw/eSASS4EDR/bin/evt/write_tEventFile: Writing events-extension extras.
 /home/idies/sw/eSASS4EDR/bin/evt: DONE
 /home/idies/sw/eSASS4EDR/bin/evt: eSASS4EDR Sep 02 09:43:31 2021
 /home/idies/sw/eSASS4EDR/bin/evt/evtool_main: memset==.false., so memory setting for large files is used.
 /home/idies/sw/eSASS4EDR/bin/evt/merge_extensions: Merging 1 files.
 /home/idies/sw/eSASS4EDR/bin/evt/merge_extensions: Infiles will be read 1 extension at a time.
 /home/idies/sw/eSASS4EDR/bin/evt/evtool_main: Applying filters.
 /home/idies/sw/eSASS4EDR/bin/evt/apply_filter_eventfile: Creating image.
 /home/idies/sw/eSASS4EDR/bin/evt/evtool_main: Writing outfile.
 /home/idies/sw/eSASS4EDR/bin/evt/write_tEventFile: Writing events.
 /home/idies/sw/eSASS4EDR/bin/evt/write_tEventFile: Writing events-extension extras.
 /home/idies/sw/eSASS4EDR/bin/evt: DONE
 /home/idies/sw/eSASS4EDR/bin/evt: eSASS4EDR Sep 02 09:43:31 2021
 /home/idies/sw/eSASS4EDR/bin/evt/evtool_main: memset==.false., so memory setting for large files is used.
 /home/idies/sw/eSASS4EDR/bin/evt/merge_extensions: Merging 1 files.
 /home/idies/sw/eSASS4EDR/bin/evt/merge_extensions: Infiles will be read 1 extension at a time.
 /home/idies/sw/eSASS4EDR/bin/evt/evtool_main: Applying filters.
 /home/idies/sw/eSASS4EDR/bin/evt/apply_filter_eventfile: Creating image.
 /home/idies/sw/eSASS4EDR/bin/evt/evtool_main: Writing outfile.
 /home/idies/sw/eSASS4EDR/bin/evt/write_tEventFile: Writing events.
 /home/idies/sw/eSASS4EDR/bin/evt/write_tEventFile: Writing events-extension extras.
 /home/idies/sw/eSASS4EDR/bin/evt: DONE
 /home/idies/sw/eSASS4EDR/bin/evt: eSASS4EDR Sep 02 09:43:31 2021
 /home/idies/sw/eSASS4EDR/bin/evt/evtool_main: memset==.false., so memory setting for large files is used.
 /home/idies/sw/eSASS4EDR/bin/evt/merge_extensions: Merging 1 files.
 /home/idies/sw/eSASS4EDR/bin/evt/merge_extensions: Infiles will be read 1 extension at a time.
 /home/idies/sw/eSASS4EDR/bin/evt/evtool_main: Applying filters.
 /home/idies/sw/eSASS4EDR/bin/evt/apply_filter_eventfile: Creating image.
 /home/idies/sw/eSASS4EDR/bin/evt/evtool_main: Writing outfile.
 /home/idies/sw/eSASS4EDR/bin/evt/write_tEventFile: Writing events.
 /home/idies/sw/eSASS4EDR/bin/evt/write_tEventFile: Writing events-extension extras.
 /home/idies/sw/eSASS4EDR/bin/evt: DONE
['expmap', 'inputdatasets=FWCv1.0_120_EventList.fits', 'emin=0.200000 0.500000 1.000000 2.200000', 'emax=0.500000 1.000000 2.200000 10.000000', 'templateimage=./EventList_021.fits.gz', 'withvignetting=yes', 'withmergedmaps=yes', 'withdetmaps=yes', 'withfilebadpix=no', 'withcalbadpix=no', 'mergedmaps=./ExposureMap_021.fits.gz ./ExposureMap_022.fits.gz ./ExposureMap_023.fits.gz ./ExposureMap_024.fits.gz']
 /home/idies/sw/eSASS4EDR/bin/exp: eSASS4EDR Sep 02 09:43:31 2021
 /home/idies/sw/eSASS4EDR/bin/exp/get_evt_keywords: **ERROR2** Unrecognized value '' of FILTER keyword. The CALDB will return you files for all filters, which may not be what you want.
 /home/idies/sw/eSASS4EDR/bin/exp/ero_cifsl: Accessing cal file tm1_detmap_100602v02.fits
 /home/idies/sw/eSASS4EDR/bin/exp/ero_cifsl: Accessing cal file tm1_instpar_200212v09.fits
 /home/idies/sw/eSASS4EDR/bin/exp/ero_cifsl: Accessing cal file tm1_instpar_200331v10.fits
 /home/idies/sw/eSASS4EDR/bin/exp/ero_cifsl: Accessing cal file tm1_instpar_190701v15.fits
 /home/idies/sw/eSASS4EDR/bin/exp/get_detmaps: **WARNING1** More than one GEOM found; using first.
 /home/idies/sw/eSASS4EDR/bin/exp/ero_cifsl: **ERROR3** tm1_arf_filter_000101v02.fits and tm1_arf_open_000101v02.fits have identical REF_TIME
 /home/idies/sw/eSASS4EDR/bin/exp/get_area: **ERROR3** Error finding cal component SPECRESP for telescope 1.
 /home/idies/sw/eSASS4EDR/bin/exp/main: **STOP** Obtaining area failed
 /home/idies/sw/eSASS4EDR/bin/exp: FAILED
Traceback (most recent call last):
  File "./demoscript_4EDR.py", line 490, in <module>
    main(sys.argv)
  File "./demoscript_4EDR.py", line 481, in main
    RunTasks()
  File "./demoscript_4EDR.py", line 128, in RunTasks
    subprocess.check_call(cmd)
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['expmap', 'inputdatasets=FWCv1.0_120_EventList.fits', 'emin=0.200000 0.500000 1.000000 2.200000', 'emax=0.500000 1.000000 2.200000 10.000000', 'templateimage=./EventList_021.fits.gz', 'withvignetting=yes', 'withmergedmaps=yes', 'withdetmaps=yes', 'withfilebadpix=no', 'withcalbadpix=no', 'mergedmaps=./ExposureMap_021.fits.gz ./ExposureMap_022.fits.gz ./ExposureMap_023.fits.gz ./ExposureMap_024.fits.gz']' returned non-zero exit status 1.

Any idea of what is going on? More importantly, which data files is the demo supposed to work on?
Thanks a lot.

Best,
Matteo

[MiriamRamos]

Dear Matteo,

You are trying to process filter wheel closed (FWC) data, which is a collection of FWC data taken at different sky positions. Obtaining a meaningful image might be possible but it will require a lot of work on separating the data, re-projecting it and combining it again. Moreover, you obtain that error because the 'FILTER' keyword in the main header is empty, and the task 'expmap' is looking for a value there.

The demo will work on sky observations, specifically on event files. You can download them from here: https://erosita.mpe.mpg.de/edr/eROSITAObservations/

Any observation there should work without problem.

Best wishes,

Miriam

[matteani]

Dear Miriam,

Thank you very much for your reply.
Unfortunately, the code was not running on the observations already (and for this reason I decided to run it on the FWC data, which seemed to be able to run a little bit more down the pipeline). The following is the stacktrace for the eFEDs data

Code: Select all

idies@c8c25ff3492a:~/demo$ ./demoscript_4EDR.py ~/WorkingFolder/eFEDS/fm00_300007_020_EventList_c001.fits 
Results will be saved in the current folder :
	/home/idies/demo
This script will uses eSASS installed in /home/idies/sw/eSASS4EDR
and the executables in:
	/home/idies/sw/eSASS4EDR/bin
 /home/idies/sw/eSASS4EDR/bin/evt: eSASS4EDR Sep 02 09:43:31 2021
 /home/idies/sw/eSASS4EDR/bin/evt/evtool_main: memset==.false., so memory setting for large files is used.
 /home/idies/sw/eSASS4EDR/bin/evt/merge_extensions: Merging 1 files.
 /home/idies/sw/eSASS4EDR/bin/evt/merge_extensions: Infiles will be read 1 extension at a time.
 /home/idies/sw/eSASS4EDR/bin/evt/evtool_main: Applying filters.
 /home/idies/sw/eSASS4EDR/bin/evt/apply_filter_eventfile: Creating image.
 /home/idies/sw/eSASS4EDR/bin/evt/create_from_events_tImage2D_i4: **WARNING1** 184414 out of 294823 possible events fall outside the image bounds.
 /home/idies/sw/eSASS4EDR/bin/evt/evtool_main: Writing outfile.
 /home/idies/sw/eSASS4EDR/bin/evt/write_tEventFile: Writing events.
 /home/idies/sw/eSASS4EDR/bin/evt/write_tEventFile: Writing events-extension extras.
Traceback (most recent call last):
  File "./demoscript_4EDR.py", line 490, in <module>
    main(sys.argv)
  File "./demoscript_4EDR.py", line 481, in main
    RunTasks()
  File "./demoscript_4EDR.py", line 105, in RunTasks
    subprocess.check_call(cmd)
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['evtool', 'eventfiles=/home/idies/WorkingFolder/eFEDS/fm00_300007_020_EventList_c001.fits', 'outfile=./EventList_021.fits.gz', 'emin=0.200000', 'emax=0.500000', 'image=yes', 'rebin=80', 'size=3240', 'pattern=15']' died with <Signals.SIGKILL: 9>.

What is going wrong?
Thank you very much in advance.

All the best,
Matteo

[MiriamRamos]

Dear Matteo,

The eFEDS data set is quite heavy and requires more memory than the default you are using in your Docker container. Please have a look here:

https://erosita.mpe.mpg.de/edr/DataAnal ... cker_setup

Under the subsection "Memory setting". It describes the problem you are facing and gives you some advice on how to solve it.

Let us know if it works.

Best wishes,

Miriam

[matteani]

Dear Miriam,

Thanks again for the support.
Unfortunately I had already tried that (I allow up to 10Gb, that should be enough right?)

Code: Select all

idies@c8c25ff3492a:~/demo$ ./demoscript_4EDR.py ~/WorkingFolder/LMC_SN1987A/fm00_700204_020_EventList_c001.fits 
Results will be saved in the current folder :
	/home/idies/demo
This script will uses eSASS installed in /home/idies/sw/eSASS4EDR
and the executables in:
	/home/idies/sw/eSASS4EDR/bin
 /home/idies/sw/eSASS4EDR/bin/evt: eSASS4EDR Sep 02 09:43:31 2021
 /home/idies/sw/eSASS4EDR/bin/evt/evtool_main: memset==.false., so memory setting for large files is used.
 /home/idies/sw/eSASS4EDR/bin/evt/merge_extensions: Merging 1 files.
 /home/idies/sw/eSASS4EDR/bin/evt/merge_extensions: Infiles will be read 1 extension at a time.
 /home/idies/sw/eSASS4EDR/bin/evt/evtool_main: Applying filters.
 /home/idies/sw/eSASS4EDR/bin/evt/apply_filter_eventfile: Creating image.
 /home/idies/sw/eSASS4EDR/bin/evt/evtool_main: Writing outfile.
 /home/idies/sw/eSASS4EDR/bin/evt/write_tEventFile: Writing events.
 /home/idies/sw/eSASS4EDR/bin/evt/write_tEventFile: Writing events-extension extras.
 /home/idies/sw/eSASS4EDR/bin/evt: DONE
 /home/idies/sw/eSASS4EDR/bin/evt: eSASS4EDR Sep 02 09:43:31 2021
 /home/idies/sw/eSASS4EDR/bin/evt/evtool_main: memset==.false., so memory setting for large files is used.
 /home/idies/sw/eSASS4EDR/bin/evt/merge_extensions: Merging 1 files.
 /home/idies/sw/eSASS4EDR/bin/evt/merge_extensions: Infiles will be read 1 extension at a time.
 /home/idies/sw/eSASS4EDR/bin/evt/evtool_main: Applying filters.
 /home/idies/sw/eSASS4EDR/bin/evt/apply_filter_eventfile: Creating image.
 /home/idies/sw/eSASS4EDR/bin/evt/evtool_main: Writing outfile.
 /home/idies/sw/eSASS4EDR/bin/evt/write_tEventFile: Writing events.
Traceback (most recent call last):
  File "./demoscript_4EDR.py", line 490, in <module>
    main(sys.argv)
  File "./demoscript_4EDR.py", line 481, in main
    RunTasks()
  File "./demoscript_4EDR.py", line 105, in RunTasks
    subprocess.check_call(cmd)
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['evtool', 'eventfiles=/home/idies/WorkingFolder/LMC_SN1987A/fm00_700204_020_EventList_c001.fits', 'outfile=./EventList_022.fits.gz', 'emin=0.500000', 'emax=1.000000', 'image=yes', 'rebin=80', 'size=3240', 'pattern=15']' died with <Signals.SIGKILL: 9>.

In the above, the stacktrace for the LMC_SN1987A dataset. Let me know if I can provide additional information to guide you into finding the issue.
Thanks again.

Best,
Matteo

[MiriamRamos]

Dear Matteo,

You might be trying to write a large file inside the container, and therefore you are running out of virtual disk - this may look like a memory error too. We suggest mounting a directory where you can write the output files of the demo.

Let us know if this works.

Best wishes,
Miriam

[matteani]

Dear Miriam,

I was also worried about that being the possible issue, so (already in the previous two examples) I mounted a local folder via

Code: Select all

docker run --volume ~/local_folder:/home/idies/WorkingFolder -it --rm erosita/esass-minimal:latest /bin/bash

Is there anything else I can try?
Thank you very much.

Kind regards,
Matteo

[jhaase]

Dear Matteo

From the snippet further above it looks like you are running the demo-script inside the demo folder, which I presume would write the output file in ~/demo, so inside the container.
Do you get the same behaviour when running the script from ~/WorkingFolder?

I don't know what the expected size of the output is, but you can at seen how much virtual diskspace you have in the container with df

cheers
Jonas

[AlainGueguen]

Dear Matteo
the Demo script ./demoscript_4EDR.py
accept 1 or 2 parameters:
the second ( optional ) is the path of the output folder
if you launch the demo script without any parameters you'll have a short information :

by A.Gueguen M.Ramos & C.Grossberger
current version : April 2021
This script must be called with 1 or 2 parameters
demoscript_4EDR.py --help or demoscript_4EDR.py -H, print this message

* First parameter, Mandatory, is the input file, with absolute or relative path
* Second parameter, Optional, is the result folder, with absolute or relative path
if given, and not existing, the folder is created by the script
EXAMPLES

./demoscript_4EDR.py ./em01_237171_020_EventList_003_c946.fits ./MyResults
This call will run the script on a local file em01_237171_020_EventList_003_c946.fits
and save the results in a subfolder of the current working directory named MyResults

./demoscript_4EDR.py ./em01_237171_020_EventList_003_c946.fits
This call will run the script on a local file em01_237171_020_EventList_003_c946.fits
and save the results the current working directory

this program needs python3

In one of your previous post i saw that you mounted a local folder when launchoing the docker
so i suggest that you try the demoscript this way :

./demoscript_4EDR.py ~/WorkingFolder/LMC_SN1987A/fm00_700204_020_EventList_c001.fits /home/idies/WorkingFolder

Cheers
Alain

[AlainGueguen]

Dear Matteo,

by default the demo script save the output in compressed fits files fits.gz
In the docker this fail.
To solve this the best would be to change the demoscript :
the line 17
suffix=".fits.gz"
should become
suffix=".fits"

The best is to copy the demoscript in your mounted folder, do the modification in the local copy and execute this one .

Let us know if Evtool succeed with this modification .

regards