Welcome to the BEAM Forum!

We encourage you to sign in our forum and participate in the BEAM community. The forum is maintained by the BEAM project team who will most likely answer your questions within 24 hours (except during common holidays) - if not done by other community members. Collaborate, share your knowledge and learn from other users!

If you don't find what you are looking for, please also consider the following external forums:

Categories » BEAM Toolbox Usage » RE: How to export a dataset to CVS format using GPT?
Combination View Flat View Tree View
Threads [ Previous | Next ]
RE: How to export a dataset to CVS format using GPT?
How to export a dataset to CVS format using GPT?
gpt export cvs
6/25/14 11:59 AM
Dear Forum,

I am trying to use the write operator node to export to a CVS format a subset of collocated product using MERIS datasets. But I received the following message:

"Error: No data product writer for the 'CVS' format available".

My parameters list for the xmlrequest looks like this:

1<parameters>
2     <file>'+ outputfilename +'</file>
3     <formatName>CVS</formatName>
4</parameters>


Also the call for the gpt looks like this:

1gpt + " " + xmlrequest  + " -Ssource=" + inputFileName + ' -Smaster=" + masterFilename + " -f CVS -t ' + outputFile


What am I missing?

Cheers,

Joseemoticon
Flag Flag
RE: How to export a dataset to CVS format using GPT?
Answer Answer (Unmark)
6/25/14 12:05 PM as a reply to Jose M. Beltran.
I found my error what it was missing was my lens ;)
I wrote CVS instead of CSV ... another silly dislexia error!
Thanks!emoticonemoticon
Flag Flag
RE: How to export a dataset to CVS format using GPT?
8/15/14 3:27 PM as a reply to Jose M. Beltran.
Hi Jose,

i try to use your example under linux as

file: xmlrequest
--
<parameters>
<file>'+ outputfilename +'</file>
<formatName>CSV</formatName>
</parameters>
--

gpt.sh xmlrequest -Ssource=A2014220105500.L2_LAC_OC -f CVS -t tmp.csv

and get next message
Error: Operator SPI not found for operator [xmlrequest]

where is my error?

thank you,
slava
Flag Flag
RE: How to export a dataset to CVS format using GPT?
8/15/14 4:24 PM as a reply to Vyasheslav Vladimirovich Suslin.
Hi slava,

it seems that the 'xmlrequest' is interpreted as operator name by gpt. 'xmlrequest' needs to be a path to a xml file containing the processing graph.

And also you need to change CVS by CSV in your command line call.

regards
Marco
Flag Flag
RE: How to export a dataset to CVS format using GPT?
8/18/14 8:29 AM as a reply to Marco Peters.
Marco,

thank you for quickly answer.

Can you give me an example of xmlrequest-file or may be an other way of solving of my task?
My task is to do a command line to convert level-2 products (A2014220105500.L2_LAC_OC) to CSV format for all pixels.
This is my first experience to work in BEAM.
In handle I did it under seadas 7.0.1, but the problem to do it in command line.

Can you help me?
slava
Flag Flag
RE: How to export a dataset to CVS format using GPT?
8/18/14 8:52 PM as a reply to Vyasheslav Vladimirovich Suslin.
You can use the Write opertor instead of the xml file. You would only need the xml file if you want to process the data.
The command line could look like this:
gpt Write -Ssource="path/to/source/file" -PformatName=CSV -Pfile="path/to/target/file"

regards
Marco
Flag Flag
RE: How to export a dataset to CVS format using GPT?
8/19/14 8:57 AM as a reply to Marco Peters.
Marco,

thank you.

I did it under ubumtu on my PC.

start command line:

/home/slava/seadas-7.0.1/bin/gpt.sh Write -Ssource="/home/slava/Load/MODISA/A2014220105500.L2_LAC_OC" -PformatName=CSV -Pfile="/home/slava/Load/MODISA/tmp.csv"

I didn't get CSV-file! This file is empty.
result is

slava@ac:~/Load/MODISA$ ls -l /home/slava/Load/MODISA/tmp.csv
-rw-rw-r-- 1 slava slava 0 сер 19 10:38 /home/slava/Load/MODISA/tmp.csv

But i get

drwxrwxr-x 3 slava slava 4096 сер 19 10:39 target.data
-rw-rw-r-- 1 slava slava 253272 сер 19 10:39 target.dim

where all products are separately in target.data directory

I'm somewhere wrong again?

Sorry,
slava
Flag Flag
RE: How to export a dataset to CVS format using GPT?
8/19/14 1:36 PM as a reply to Vyasheslav Vladimirovich Suslin.
Dear Slava,

This example may help to keep you going.

first:
I have a function that builds the xmlrequest for the gpt:

 1
 2def gpt_graph(outputfilename):   
 3    '''
 4    Builds an graph XML to be used with the gpt tool of VISAT-BEAM.
 5    outputfilename:     output path/filename for processing
 6    '''
 7   
 8    requestSkeleton =   '<graph id=\"MyProcess\">\n<version>1.0</version>\n'
 9    requestSkeleton +=  '   <node id=\"writeFile\">\n'
10    requestSkeleton +=  '       <operator>Write</operator>\n'
11    requestSkeleton +=  '       <sources>\n'
12    requestSkeleton +=  '           <source>source</source>\n'
13    requestSkeleton +=  '       </sources>\n'
14    requestSkeleton +=  '       <parameters>\n'
15    requestSkeleton +=  '           <file>'+ outputfilename +'</file>\n'
16    requestSkeleton +=  '           <formatName>CSV</formatName>\n'
17    requestSkeleton +=  '       </parameters>\n'
18    requestSkeleton +=  '   </node>\n'       
19    requestSkeleton +=  '</graph>\n'
20    return requestSkeleton


Then I have
1# xml file to be used as request
2xmlrequest = '/path/to/gpt_graph.xml'


Then I have a for loop for all my MERIS files in a path source list, like this:

 1
 2for merisFile in srcList:
 3   ...
 4   inputFileName = merisFile
 5   ...
 6   # Calling my function with the desired output filename
 7   request = gpt_graph(outputFile)
 8   # Delete previous graph settings, i.e.file should not exist
 9   if os.path.exists(xmlrequest):
10        os.remove(xmlrequest)
11   # New graph is generated with the current settings
12   requestFile = open(xmlrequest, 'a')
13   requestFile.write(request)
14   requestFile.close()
15   ExCommand =  "~/beam-5.0/bin/gpt.sh " + xmlrequest  + " -Ssource=" + inputFileName + ' -f CVS -t ' + outputFile
16   os.system(ExCommand)



My xmlrequest file looks like this:
 1
 2<graph id="SUProcess">
 3<version>1.0</version>
 4   <node id="writeFile">
 5       <operator>Write</operator>
 6       <sources>
 7           <source>source</source>
 8       </sources>
 9       <parameters>
10           <file>/output/dir/MEGS_MER_FSG_2PNMAP20120405_092930_000000393113_00223_52823_0001.csv</file>
11           <formatName>CSV</formatName>
12       </parameters>
13   </node>
14</graph>


At the end the call looks like this, pretty similar to what Marco already suggested:

1/home/jobel/beam-5.0/bin/gpt.sh /path/to/gpt_graph.xml -Ssource=/srcpath/MER_FSG_2PNMAP20120405_092930_000000393113_00223_52823_0001.nc -f CVS -t /output/dir/MEGS_MER_FSG_2PNMAP20120405_092930_000000393113_00223_52823_0001.csv


Hope it will help you.
Cheers

Jose
Flag Flag
RE: How to export a dataset to CVS format using GPT?
8/19/14 1:13 PM as a reply to Jose M. Beltran.
Dear Jose,

thank you.
I'll think and try to do it.

best wishes,
slava
Flag Flag
RE: How to export a dataset to CVS format using GPT?
9/15/14 8:22 AM as a reply to Vyasheslav Vladimirovich Suslin.
Dear Jose, Marco,

thank you.
all work for small files!

Q: however, when I try to work with the files of high spatial resolution, the command does not work. Not enough memory?
Q: how increase memory more 512M? Additional key/option?

For example,

/home/slava/beam-5.0/bin/gpt.sh Write -Ssource=/home/slava/Load/SeaWiFS/Test.MLAC/S2001160101315.L2_MLAC_OC -PformatName=CSV -Pfile="/home/slava/Load/SeaWiFS/Test.MLAC/test.oc.csv"

or

M2010223080516.L2_FRS_OC
M2010223080554.L2_FRS_OC

best regards,
slava
Flag Flag
RE: How to export a dataset to CVS format using GPT?
9/17/14 2:33 PM as a reply to Vyasheslav Vladimirovich Suslin.
Hi Slava,

what do you mean by "the command does not work" -- can you provide an error message?

You can increase the memory by editing gpt.sh: -Xmx<number>M is the line you have to edit, with number the amount of megabytes of memory you want to assign to the Java process.

Cheers,
Thomas
Flag Flag
RE: How to export a dataset to CVS format using GPT?
9/19/14 7:55 AM as a reply to Thomas Storm.
Hi Thomas,

see example below

(1)
-----------------
slava@ac:~/Load/MERIS/Test.FRS$ /home/slava/beam-5.0/bin/gpt.sh Write -Ssource=/home/slava/Load/MERIS/Test.FRS/M2010223080516.L2_FRS_OC -PformatName=CSV -Pfile="/home/slava/Load/MERIS/Test.FRS/M2010223080516.L2_FRS_OC.csv"
[INFO] 2014-09-17T19:26:34.549+0400 - JAI tile scheduler parallelism set to 4
[INFO] 2014-09-17T19:26:35.325+0400 - JAI tile cache size is 512 MB
[INFO] 2014-09-17T19:26:35.326+0400 - JAI tile scheduler parallelism is 4
[INFO] 2014-09-17T19:27:00.500+0400 - All GPF operators will share an instance of com.sun.media.jai.util.SunTileCache with a capacity of 512M
org.esa.beam.framework.gpf.OperatorException: Cannot construct DataBuffer.
at org.esa.beam.framework.gpf.internal.OperatorExecutor$GPFImagingListener.errorOccurred(OperatorExecutor.java:373)
at com.sun.media.jai.util.SunTileScheduler.sendExceptionToListener(SunTileScheduler.java:1646)
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:921)
at javax.media.jai.OpImage.getTile(OpImage.java:1129)
at com.sun.media.jai.util.RequestJob.compute(SunTileScheduler.java:247)
at com.sun.media.jai.util.WorkerThread.run(SunTileScheduler.java:468)
Caused by: org.esa.beam.framework.gpf.OperatorException: Cannot construct DataBuffer.
at org.esa.beam.gpf.operators.standard.WriteOp.computeTile(WriteOp.java:312)
at org.esa.beam.framework.gpf.internal.OperatorImage.computeRect(OperatorImage.java:77)
at javax.media.jai.SourcelessOpImage.computeTile(SourcelessOpImage.java:137)
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
... 3 more
Caused by: java.lang.RuntimeException: Cannot construct DataBuffer.
at com.sun.media.jai.util.DataBufferUtils.constructDataBuffer(DataBufferUtils.java:132)
at com.sun.media.jai.util.DataBufferUtils.createDataBufferFloat(DataBufferUtils.java:214)
at javax.media.jai.ComponentSampleModelJAI.createDataBuffer(ComponentSampleModelJAI.java:271)
at javax.media.jai.RecyclingTileFactory.createTile(RecyclingTileFactory.java:389)
at javax.media.jai.PlanarImage.createWritableRaster(PlanarImage.java:1982)
at javax.media.jai.PlanarImage.getData(PlanarImage.java:2160)
at javax.media.jai.PlanarImage.getData(PlanarImage.java:2016)
at javax.media.jai.RenderedOp.getData(RenderedOp.java:2266)
at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:59)
at org.esa.beam.csv.dataio.writer.CsvProductWriter.writeBandRasterData(CsvProductWriter.java:138)
at org.esa.beam.gpf.operators.standard.WriteOp.writeTileRow(WriteOp.java:360)
at org.esa.beam.gpf.operators.standard.WriteOp.computeTile(WriteOp.java:278)
... 6 more

Error: Cannot construct DataBuffer.
----------------

(2) the same results i got if input is S2001160083424.L2_MLAC_OC
or
(3) Export raster data -> CSV for these files manually (see attach file)

(4) After editing gpt.sh: -Xmx<number>M up to 2M the result was the same.

best regards,
slava
Attachment

Attachments: seadas.7.0.2.jpg (32.6k)
Flag Flag
RE: How to export a dataset to CVS format using GPT?
9/19/14 10:02 AM as a reply to Vyasheslav Vladimirovich Suslin.
Hi Slava,

yes -- the M stands for 'Megabyte', so 2M is pretty low. Please try again with, say, 3000 (or even more, if your machine allows). Note that you have to edit gpt.sh for console processing, and visat-d.sh for the graphical environment (you will have to start the graphical enviroment via visat-d.sh then).

Cheers,
Thomas
Flag Flag
RE: How to export a dataset to CVS format using GPT?
10/6/14 3:44 PM as a reply to Thomas Storm.
Dear Thomas,

thank you.
all work.

best regards,
slava
Flag Flag