Categories » BEAM Toolbox Usage » Mosaic performance (using multiple cores)
Combination View Flat View Tree View
Threads [ Previous | Next ]
Mosaic performance (using multiple cores)
toggle
Mosaic performance (using multiple cores) Sergi Pons Freixes 5/18/12 3:33 PM
RE: Mosaic performance (using multiple cores) Ralf Quast 5/21/12 9:49 AM
RE: Mosaic performance (using multiple cores) Sergi Pons Freixes 5/22/12 9:50 AM
RE: Mosaic performance (using multiple cores) Ralf Quast 5/22/12 10:44 AM
RE: Mosaic performance (using multiple cores) Sergi Pons Freixes 5/22/12 4:19 PM
RE: Mosaic performance (using multiple cores) Ralf Quast 5/22/12 6:23 PM
RE: Mosaic performance (using multiple cores) Sergi Pons Freixes 5/23/12 10:06 AM
RE: Mosaic performance (using multiple cores) Ralf Quast 5/23/12 1:41 PM
RE: Mosaic performance (using multiple cores) Sergi Pons Freixes 5/23/12 2:50 PM
RE: Mosaic performance (using multiple cores) Ralf Quast 5/23/12 6:07 PM
RE: Mosaic performance (using multiple cores) Sergi Pons Freixes 5/24/12 9:29 AM
RE: Mosaic performance (using multiple cores) Ralf Quast 5/24/12 2:54 PM
RE: Mosaic performance (using multiple cores) Sergi Pons Freixes 5/25/12 10:50 AM
RE: Mosaic performance (using multiple cores) Ralf Quast 5/29/12 12:12 PM
RE: Mosaic performance (using multiple cores) Sergi Pons Freixes 5/30/12 3:04 PM
RE: Mosaic performance (using multiple cores) Ralf Quast 5/30/12 4:52 PM
RE: Mosaic performance (using multiple cores) Sergi Pons Freixes 5/31/12 9:58 AM
RE: Mosaic performance (using multiple cores) Ralf Quast 6/1/12 8:47 AM
RE: Mosaic performance (using multiple cores) Sergi Pons Freixes 6/1/12 12:06 PM
RE: Mosaic performance (using multiple cores) Sergi Pons Freixes 6/1/12 12:15 PM
RE: Mosaic performance (using multiple cores) Sergi Pons Freixes 6/11/12 10:13 AM
RE: Mosaic performance (using multiple cores) Ralf Quast 6/15/12 9:42 AM
RE: Mosaic performance (using multiple cores) Sergi Pons Freixes 7/23/12 9:19 AM
Mosaic performance (using multiple cores)
mosaic cores
5/18/12 3:33 PM
I've observed that, when using the mosaic functionality, only one core of the processor is used. Is there any way to tune it in order to use more cores?
Flag Flag
RE: Mosaic performance (using multiple cores)
5/21/12 9:49 AM as a reply to Sergi Pons Freixes.
How many cores are used is automatically determined by the Java Advanced Imaging (JAI) library. You can however specify in the beam.config file

1beam.parallelism = N


to tell the system how many cores you would like JAI to use. A value of N indicates that there will be N worker threads in addition to the primary scheduler thread. The default is N = 2.
Flag Flag
RE: Mosaic performance (using multiple cores)
5/22/12 9:50 AM as a reply to Ralf Quast.
Is this parameter also taken into account by GPT?
Flag Flag
RE: Mosaic performance (using multiple cores)
5/22/12 10:44 AM as a reply to Sergi Pons Freixes.
Yes.
Flag Flag
RE: Mosaic performance (using multiple cores)
5/22/12 4:19 PM as a reply to Ralf Quast.
I'm checking that I have just one process, an just one of the cores at 100% while the rest are idle. Is thre anything else I should take into account? This factor is quite a bottleneck.

Extract of ps (corre for the incomplete lines):

1/bin/sh /home/sergi/beam-4.9.0.1/bin/gpt.sh /home/sergi/code/mosaic_gbr_monthly_l1p.x
215251 15252 15249  2441 pts/0    15249 Sl+   1000 389:44              |   |   |           \_ java -Xmx7168M -Dceres.context=beam -Dbeam.mainClass=org.esa.beam.framework.gpf.main.
Flag Flag
RE: Mosaic performance (using multiple cores)
5/22/12 6:23 PM as a reply to Sergi Pons Freixes.
What is the size of your target product in terms of pixel width x height? And how many cores do you have?
Flag Flag
RE: Mosaic performance (using multiple cores)
5/23/12 10:06 AM as a reply to Ralf Quast.
The target is 3840x4600, and the processor an Intel i5 with 4 cores.
Flag Flag
RE: Mosaic performance (using multiple cores)
5/23/12 1:41 PM as a reply to Sergi Pons Freixes.
What exactly were you doing?
Flag Flag
RE: Mosaic performance (using multiple cores)
5/23/12 2:50 PM as a reply to Ralf Quast.
Mosaicking about 20-30 CoastColour L1P images to obtain monthly averages of a region, so 15 bands per image (radiances), discarding pixels according to some flags (CC_CLOUD, INVALID, etc).
Flag Flag
RE: Mosaic performance (using multiple cores)
5/23/12 6:07 PM as a reply to Sergi Pons Freixes.
In VISAT or with GPT? With which parameter settings?
Flag Flag
RE: Mosaic performance (using multiple cores)
5/24/12 9:29 AM as a reply to Ralf Quast.
With GPT. I attach the corresponding xml.
Attachments: mosaic_gbr_monthly_l1p.xml (2.4k)
Flag Flag
RE: Mosaic performance (using multiple cores)
5/24/12 2:54 PM as a reply to Sergi Pons Freixes.
The mosaic is very rapid and uses all available cores (I have tried it right now). What happens when you use the Mosaic tool in VISAT? When I do it all my cores (8) are used and it's very rapid (some seconds for some ten MERIS L2 product sub-scenes, about 60 MB per sub-scene).
Flag Flag
RE: Mosaic performance (using multiple cores)
5/25/12 10:50 AM as a reply to Ralf Quast.
Yes, with the mosaic tool in VISAT it uses most of the cores, and therefore much more faster.
Flag Flag
RE: Mosaic performance (using multiple cores)
5/29/12 12:12 PM as a reply to Sergi Pons Freixes.
Try to edit the gpt.sh script and insert the line

1    "-Dbeam.parallelism=N" \


where N is e.g. 4 or 8, after

1    "-Dbeam.home=$BEAM4_HOME" \


This should work.
Flag Flag
RE: Mosaic performance (using multiple cores)
5/30/12 3:04 PM as a reply to Ralf Quast.
I keep getting just one core working. I attach an image where you can see that when one core goes to 100% the others stay idle. I have both the beam.config and the gpt.sh additions made, with N=4.

BTW, I can see few posts above that you mentioned JAI. Is it a requirement for parallelizing? Because I think it's not working on my system:

1
2SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
3SLF4J: Defaulting to no-operation (NOP) logger implementation
4SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
5Error: Could not load mediaLib accelerator wrapper classes. Continuing in pure Java mode.
6Occurs in: com.sun.media.jai.mlib.MediaLibAccessor
7com.sun.media.jai.mlib.MediaLibLoadException
Attachment

Attachments: cpu_cycles.png (36.4k)
Flag Flag
RE: Mosaic performance (using multiple cores)
5/30/12 4:52 PM as a reply to Sergi Pons Freixes.
Try to increase the tile cache using the -c option of the gpt tool. Then add an explicit WriteOp node at the end of your graph and set the parameters so that complete rows of tiles are processed before they are written, if this is not done by default.
Flag Flag
RE: Mosaic performance (using multiple cores)
5/31/12 9:58 AM as a reply to Ralf Quast.
I'm not able to add the WriteOp node succesfully, I get a
1Error: [output] is null
message. I attach the used xml, and I'm running it with:

1gpt.sh config.xml -c 2048M -Pfile=$OutputFile$ $ListOfInputFIles$


BTW, I've detected an increase of RAM usage with these options.
Attachments: mosaic_gbr_monthly_l1p.xml (2.8k)
Flag Flag
RE: Mosaic performance (using multiple cores)
6/1/12 8:47 AM as a reply to Sergi Pons Freixes.
You have to insert

1<file>${file}</file>


as a parameter of the WriteOp in your XML file.
Flag Flag
RE: Mosaic performance (using multiple cores)
6/1/12 12:06 PM as a reply to Ralf Quast.
It consumes more RAM now (it even needs to use some SWAP, having 8 GB of RAM), but it still only uses 1 core at 100% while the others are at <10%.

I ran yesterday again the mosaic tool of VISAT for testing purposes, and while it uses all the cores, it didn't look like if it was much faster than GPT with one core, so it left me a bit puzzled. I'll try to process the exact set of data with VISAT mosaicking tool, GPT with "out the box" configuration, and GPT with "tuned" configuration, and compare the times.
Flag Flag
RE: Mosaic performance (using multiple cores)
6/1/12 12:15 PM as a reply to Sergi Pons Freixes.
UPDATE of last post: after a couple of hours of processing, I could see that for a while (couple of minutes?) it was using 2 cores at 100%. Then, after loading/writing something to the swap (the HD working heavy and the CPU idle), it's working again with just one core at 100%.

PD: I know when it loads/writes actual data or just uses the swap because they are on different disks.
Flag Flag
RE: Mosaic performance (using multiple cores)
6/11/12 10:13 AM as a reply to Sergi Pons Freixes.
I finally did some tests. I tried to do a mosaic with the size commented previously, from 21 files, a total of 13.2 GB. The resulting image is 2.0 GB.

With GPT, with the parallel settings enabled and disabled, it takes about 5-6 hours (the difference between both cases is of just 15 minutes), using mostly just one core. With VISAT mosaicking tool, using all the cores, it takes 9 hours!

So, apart from not being able to run GPT in parallel, the loss of performance of VISAT vs GPT it's worth taking into account.
Flag Flag
RE: Mosaic performance (using multiple cores)
6/15/12 9:42 AM as a reply to Sergi Pons Freixes.
Many thanks for doing these tests! We now have an idea of what the problem is and know that we have to resolve this problem.
Flag Flag
RE: Mosaic performance (using multiple cores)
7/23/12 9:19 AM as a reply to Ralf Quast.
I'll would like to update this threat adding that with version 5 alfa, it uses all the cores out of the box, without tuning anything.
Flag Flag

 

Welcome to the BEAM Forum!

We encourage you to sign in our forum and participate in the BEAM community. The forum is maintained by the BEAM project team who will most likely answer your questions within 24 hours (except during common holidays) - if not done by other community members. Collaborate, share your knowledge and learn from other users!

If you don't find what you are looking for, please also consider the following external forums: