Welcome to the BEAM Forum!

We encourage you to sign in our forum and participate in the BEAM community. The forum is maintained by the BEAM project team who will most likely answer your questions within 24 hours (except during common holidays) - if not done by other community members. Collaborate, share your knowledge and learn from other users!

If you don't find what you are looking for, please also consider the following external forums:

Combination View Flat View Tree View
Threads [ Previous | Next ]
RE: Can GPT tool read data products from a URL
toggle
Hello BEAM dev team,

Is it possible for the BEAM/SeaDAS gpt tool to read data products from an http URL?

I'm looking for something like the following:
1
2<node id="Read_from_url">
3    <operator>Read</operator>
4    <sources />
5    <parameters>
6       <file>http://localhost:8080/A2013255.L3m_DAY_GBROC_chlor_a_GBR_1km</file>
7    </parameters>
8</node>


The Read operator description says "Reads a product from disk" and as expected the above doesn't work. Is there perhaps any other operator which is capable of doing this?

thanks,
Kutila
Flag Flag
RE: Can GPT tool read data products from a URL
6/2/14 9:11 AM as a reply to Kutila Gunasekera.
Hello Kutila,

it is not possible to read directly from an URL. You can only read from a local disk or a network device.

regards
Marco
Flag Flag
RE: Can GPT tool read data products from a URL
6/3/14 9:06 AM as a reply to Marco Peters.
Thanks Marco!
-Kutila
Flag Flag
RE: Can GPT tool read data products from a URL
6/30/14 2:20 AM as a reply to Kutila Gunasekera.
Hello Marco and Team,

Do you believe it is practical to create a new Read operator which supports reading satellite data products from a URL?
If so, any hints on where to get started are appreciated.

To provide some background, I'm investigating deploying SeaDAS/BEAM on the azure cloud computing platform (using linux Virtual Machines). To get higher read speeds I'd like to store each data product as a 'blob' on their blob storage. These then need to be accessed using a RESTful API.

While I'm on the topic, is there by any chance support for BEAM graph processing to be distributed across multiple machines?

thanks,
Kutila
Flag Flag
RE: Can GPT tool read data products from a URL
7/1/14 11:12 AM as a reply to Kutila Gunasekera.
Hi Kutila,

I think many eo-data formats are not streamable. Which means you first need to download the whole file and then you can open it.
Maybe you can implement the reader as a delegating reader, which first downloads the file and then redirects to the 'real' reader.

In order to implement a ProductReader you also need to implement a ProductReaderPlugIn.
The ProductReaderPlugIn has the method getInputTypes. Your implementation might look like this.
1public Class[] getInputTypes() {
2    return new Class[]{String.class, URL.class};
3}


It will be also important how you implement the getDecodeQualification. In this method you have to decide if you can handle the input.

Your work is very interesting. We have cluster running and we use BEAM and its graph processing there. But we don't split the graph and let each node compute only a part. The processing graph is executed on each node for one source product.
Here you can find a bit about our Calvalus system. It's a bit outdated. The number of nodes is now more than 70.

regards
Marco
Flag Flag
RE: Can GPT tool read data products from a URL
7/21/14 9:05 AM as a reply to Marco Peters.
Hello Marco,

I ended up downloading the necessary data files prior to invoking the GPF. It was the most convenient approach as I already had a place in the code to do checks after graph XML creation and prior to graph execution. It works fine, allowing me to store data as compressed blobs and gives fairly good speeds.

It is good to learn about the existence of Calvalus, yet to read in depth about it though.
Most of the processing on our system involves reading a large number of source products, sometimes producing a single output. Hence, it is not always possible to break a graph into several parallel paths. However, there are cases where it is possible. So, there is opportunity to improve. emoticon

thanks,
Kutila
Flag Flag