Requirements

In order to successfully compile and use OmniRPC system, you need the following programs and libraries which are available on most platforms as distribution packages and thereby can be installed easily.

Required:

• gcc 2.95.4 (or compatible), available at \|\|

• GNU make (or compatible), available at \|\|

• Remote Shell (or compatible)

• flex 2.5.4,

Optional:

• OpenSSH SSH client (or compatible), avilable at \|\|

• Java SDK 1.4.0 (or higher), available at \|\|

• Globus Toolkit 2.4 (or compatible), avilable at \|\|

OmniRPC system was tested with RedHat Linux 7.3 on Intel Xeon 2.4, 512MB RAM and Debian GNU/Linux 2.0 (and higher version) on Intel compatible PCs.

Compilation and Installation

            % ./configure
            % make
            (as root)
            # make install
          

 
            % ./configure --prefix /path/to/install
          

            % ./configure --enable-globus
          

             % ./configure --enable-globus --with-globusDir=/path/to/globus --with-gtpDir=/path/to/gpt 
          

Install Test

To test the installation, proceed as follows. Before testing, you should move "OmniRPC/test" directory. And, in each test, you should move the directories s which are written by number. To obtain more detailed information of the test program, especially if you encounter trouble with program testing, add the command option "--debug", if some trouble on program testing.

            % make
        

            % local-exec-test
          

            % mkdir $HOME/.omrpc_registry
            % make reg_stub
          

            % make
          

            % remote_exec_test1 <host> <stub_directory>
          

            % remote_exec_test2 <host> <stub_directory>
          

            % cat ~/.omrpc_registry/hosts.xml
            <OmniRpcConfig>
            <Host name="localhost"/> 
            </OmniRpcConfig>
          

            % rcp-call-test1 
          

            % rpc-call-test2 <host>
          

            % rpc-call-test3
          

            % rpc-async-test1
          

Simple example

Let's begin programming with OmniRPC using a simple example. In this case, we consider a program to calculate from 1 to 10 using sine function.

#include <stdio.h>
#include <math.h>
     
int main(){
  int i;
  double x, res[10];
  x = 0.0;
  for(i = 0; i < 10; i++){
    res[i] = sin(x);
    x += 1.0;
  }
  for(i = 0; i < 10; i++) 
    printf("sin(%d)=%g\n",i,res[i]);
}
      

On this program, we calculate the sine in a remote node using OmniRPC. We label the computer node on which the program makes the Remote Procedure Call (RPC) as the client host, and the computer node on which procedures are executed by the RPC call as the remote host.

Execution environment

We assume the environment below as a simple explanation.

• Client host's name is jones.

• Remote host's name is dennis.

• From jones to dennis, we can execute "rsh" with no password authentication. In other words, jones is registered to "/etc/host.equiv" or the user's ".rhost" on the remote host.

• On both hosts, OmniRPC is installed to the default path("/usr/local/omrpc/").

In addition, OmniRPC does not need to share the file system in both hosts.

OmniRPC agent and remote executable programs

In OmniRPC, to activate a remote executable program on a remote node which is described in the hostfile, the omrpc-agent is first activated when OmniRpcInit, which is an initialization API is called.

This agent is activated for each program, and runs during program execution. This agent is invoked by an authentication method such as rsh, Globus GRAM, or ssh. And agent provides access to the module information which was registered on the remote node and interface of the job scheduler, communication multiplexing, and so on. If you want to know more details about these features, please see the section, 12. Hostfile Description.

In the upper figure, agent stands for OmniRPC-agent, rex stands for remote executable program. In this example, the agent and rex are executed on the same host (dennis).

Write remote executable program

Here, we'll write a program which calculates the sine on a remote host. We will define the interface of sine function. We name the file which defines interfaces IDL(Interface Description Language) file. The IDL is discussed in detail later section. For example, we can write the calc_sin.idl file as follows.

Module calc_sin;

Globals {
#include <math.h>;
}

Define calc_sin(double IN x, double OUT result[]){
  *result = sin(x);
}
      

Module: We define the module name in the IDL file. In this example, we set this module name as "calc_sin".

Define: Interfaces are defined by the "Define" directive. In this example, we use sin. Arguments are the specified data type and whether the data is input or output. In OmniRPC, we cannot get the return value of the function as a value of the function's value like a original sin function. So, we get the value by specifying arguments as "OUT", as shown in the example. In the part surrounded with {...}, we can write a procedure which is executed on the remote host in C Language. This program calls the sine function and "result", which is an "OUT" argument, returns the since value.

Globals: In the part assigned to "Globals", we can describe any C program which is necessary for functions that are defined in the modules. In this example, the IDL includes files which are required to call the "sin" function.

We can generate a remote executable program from the IDL file by using omrpc-cc, which is OmniRPC's remote executable module generator. So, let's convert using this command. ( "-lm" option is to link math library.)

         % omrpc-cc calc_sin.idl -lm
      

"calc_sin.rex" is generated by executing this program. This is the remote executable program.

By the way, the IDL can define multiple functions in a remote executable module. For Further details, see 13. IDL file Description.

Registration of remote executable program

After making the remote executable program, you should register it. For registration, we use the "omrpc-register" program.

         % omrpc-register -register calc_sin.rex
      

This program creates the ".omrpc_register" directory in the user's home directory and database, which consists of the module name, function name and path to the remote executable program.

After completing the above procedure, setup on the remote host side is finished. We move next to the setup on the client host side.

Client program

We rewrite the client program with OmniRpcCall.

#include <OmniRpc.h>;
#include <stdio.h>;
#include <math.h>;

int main(int argc,char *argv[]){
   int i;
   double x, res[10];

   OmniRpcInit(&argc,&argv);

   x = 0.0;
   for(i = 0; i < 10; i++){
       OmniRpcCall("calc_sin",x,; res[i]);
       x += 1.0;
   }
   for(i = 0; i < 10; i++) 
      printf("sin(%d)=%g\n",i,res[i]);

  OmniRpcFinalize();

}
      

First, we call OmniRpcInit() before we use the OmniRPC library. And, at end of this program, we call OmniRpcFinalize(). Prototype definitions of these functions are described in OmniRpc.h, so the program must include the "OmniRpc.h" header file.

We compile this program.

% omrpc-cc -o test.exe test.c
      

You can compile without omrpc-cc when you specify the directory of OmniRPC library.

% cc -I/usr/local/omrpc/include \
-o test.exe test.c -L/usr/local/omrpc/lib -lomrpc_client -lomrpc_io 
      

By default, OmniRPC software is installed at /usr/local/omrpc. If you install the software in an other directory, you should change /usr/loca/omrpc to the correct directory in which OmniRPC is installed.

Create hostfile.xml

In OmniRPC, usually we describe the execution environment in hostfile.xml with XML notation. In this example case, we describe the execution environment as follows.

<?xml version="1.0" ?>;
<OmniRpcConfig>;
   <Host name="prost" />;
</OmniRpcConfig>;
      

We write the above information in the hosts.xml file.

Execution of Cleint Program

The setup is finished. What we have to do next is to execute the program.

% test.exe --hostfile hosts.xml
      

This program searches for a remote function named sin on a remote host which is specified in the hosts.xml file and activates the remote module. If the "--hostfile" option is omitted, "--${HOME}/.omrpc-registry/hosts.xml" file is used.

Summary

Below list is a summary of the general procedure when using OmniRPC system.

1. Install OmniRPC on the client host and remote host.

2. On the remote host, create a remote executable program of the remote function and register it.

   1. Create the IDL file which defines interfaces.

   2. Generate a remote executable module from the IDL file with the "omrpc-cc" program.

   3. Register with omrpc-register.

3. On the client host, create hosts.xml which describes the remote host.

4. Write the client program, and compile with omrpc-cc.

5. Execute the client program, specifying hosts.xml.

Execution Environment

We assume an execution environment like the following.

• The client host's name is jones

• We use three hosts (dennis, alice, jack) as remote hosts.

• rsh can be executed from jones to the remote hosts without password authentication. In other words, jones is registered at /etc/host.equiv or at the user's .rhost on a remote host.

• OmniRPC is installed in the default path(/usr/local/omrpc/) on both hosts.

• And, calc_sin.rex which is the remote executable program of the "calc_sin" function, is registered on each remote host, as in previous example.

Parallel programming with asynchronous call

We show an example of an asynchronous call with OmniRPC as follows.

#include <OmniRpc.h>;
#include <stdio.h>;
#include <math.h>;

int main(int argc,char *argv[]){
   int i;
   double x, res[10];
   OmniRpcRequest req[10];

   OmniRpcInit(&argc,&argv);

   x = 0.0;
   for(i = 0; i < 10; i++){
       req[i] = OmniRpcCallAsync("calc_sin",x,&res[i]);
       x += 1.0;
   }
   OmniRpcWaitAll(10,req);
   for(i = 0; i < 10; i++) 
      printf("sin(%d)=%g\n",i,res[i]);

  OmniRpcFinalize();

}

The OmniRpcCallAsync API activates the remote procedure call and returns without waiting for its call termination. Its API returns an OmniRpcRequest value which corresponds to the remote procedure call as a return value. We store its value in an array, and by using OmniRpcWaitAll API, the program waits for termination of all RPC calls. For more details about this API, see 15. C API index.

Hostfile and Execution

Before program execution, you should prepare a hostfile.

<?xml version="1.0" ?>
<OmniRpcConfig>
   <Host name="alice" />
   <Host name="dennis" />
   <Host name="jack" />
</OmniRpcConfig>

The rest of the procedure is the same the former example. The relationship between the agent and rex is shown below.

Parallel programming with OpenMP

We can do parallel programming by calling OmniRpcCall on a multi-thread program with Omni OpenMP, which is one of OpenMP compiler of an implementation using a thread. We show an example as follows.

#include <OmniRpc.h>
#include <stdio.h>
#include <math.h>

int main(int argc,char *argv[]){
   int i;
   double x, res[10];
   OmniRpcRequest req[10];

   OmniRpcInit(&argc,&argv);

   x = 0.0;
#pragma omp parallel for
   for(i = 0; i <; 10; i++){
       req[i] = OmniRpcCall("calc_sin",x,&res[i]);
       x += 1.0;
   }
   for(i = 0; i < 10; i++) 
      printf("sin(%d)=%g\n",i,res[i]);

  OmniRpcFinalize();

}

When you want to run this program, remember to set the OMP_NUM_THREADS environmental value, which specifies the number of OpenMP threads, to a value more than the number of remote hosts.

Setting for SMP

In this example, we assume the environment below.

• Client host is jones.

• Remote host(apple) is an SMP system and has 4 processors.

• The same network and process is invoked by rsh.

Setting of Maxjob

Multiple remote execution programs can be executed at the same time on each processor in an SMP system. In this case, we can execute a maximum of 4 programs. We describe hostfile for the SMP system as follow.

<?xml version="1.0" ?>
<OmniRpcConfig>
   <Host name="apple" arch="i386" os="linux">
   <JobScheduler maxjob="4" />
   </Host>
</OmniRpcConfig>

In this description, we set the maxjob attribute as "4" for the JobScheduler element. This shows that the maximum number of remote executable programs executed on the remote host at the same time is 4. Certainly, in a 4-way SMP system, we can run more than 4 jobs, but in practice 4 processes are executed in parallel. Therefore effectiveness may not change.

In this case, the agent invoke multiple rex-es. The relationship is as follows.

Execution of client progra

The client program is executed in the same manner as the others, except for using the above host file.

% a.out --hostfile hosts.xml  args
...  
      

Of course, to achieve good performance, we use a client program which uses the OmniRPC call in parallel.

Execution Environment with Globus Toolkit

We assume an execution environment as follows.

• Client host is alice.hpcs.is.tukuba.ac.jp .

• Remote host is dennis.hpcc.jp. The Globus-gate-keeper must be running on the remote host.

• On the remote host, a non special privilege port (the port number is more than 1024) may be opened.

  Warning

  Please refer to the Globus manual for the method to limit the range of Globus ports.

In the explanation below, we assume an environment setting on which a job can be submitted from the client host to the remote host.

Preparation of Globus

First, initialize the proxy certificate when you use Globus Toolkit.

% grid-proxy-init
      

In this phase, you should input the pass phase, and create the proxy certificate. You can check whether Globus can execute normally or not, by the example.

% globusrun -o -r dennis.hpcc.jp '& (executable=/bin/date)'
      

If you cannot see the current time, you have to return to the setting of Globus. On other hand, if you can see the current time, move to the next step.

Hostfile for Globus

To use dennis.hpcc.jp, you should describe the information inn hosts.xml, and you can execute the client program by its hostfile. When you use Globus, you should describe the following information.

<?xml version="1.0" ?>
<OmniRpcConfig>
   <Host name="dennis.hpcc.jp" arch="i386" os="linux">
   <Agent invoker="globus" />
   </Host>
</OmniRpcConfig>

This description is for an agent using Globus to invoke a remote executable program. In this setting, the relationship between the agent and rex is shown in the following.

Execution of client program

You can execute the client program in the same manner without using the hosts.xml file.

% a.out --hostfile hosts.xml  args ... 
      

Preparing for SSH

In this example, the environment is described below.

• Client host is alice.hpcs.is.tukuba.ac.jp .

• Remote host is dennis.hpcc.jp.

• There is not a firewall between the remote host and client host. In other words, programs can communicate without limitation on the non special privilege port.

We assume that we can use SSH here. That is, we assume that we can access with SSH from alice.hpcs.is.tsukuba.ac.jp to dennis.hpcc.jp.

When using SSH, you should set up auto authentication with an ssh-agent. If you do not, you have to type in the password at each remote host. If you want to know more detail, see man of "ssh-agent", In this example, we outline the usage below.

1. Activate ssh-agent, and set the environment variable on your terminal.

   % eval `ssh-agent`
             

2. Register the pass phrase with ssh-add.

   % ssh-add
   type your pass phrase here.[passwd]
   Identity added: /home/foo/.ssh/id_rsa (/home/foo/.ssh/id_rsa)
   % ssh-add
             

With the above procedure, you should confirm whether auto authentication is running or not.

%  ssh dennis.hpcc.jp
      

With the above command, you can login to remote hosts without typing the password.

Hostfile for SSH

To use dennis.hpcc.jp, you should describe hosts.xml, and execute the client program with this hostfile. Write the hostfile for using ssh, as follows.

<?xml version="1.0" ?>
<OmniRpcConfig>
   <Host name="dennis.hpcc.jp" arch="i386" os="linux">
   <Agent invoker="ssh" />
   </Host>
</OmniRpcConfig>
 

This description is for activating the agent with ssh. In this specification, the relationship between the agent and rex is shown below.

Setting for firewall: Using SSH's port forwarding and MXIO option.

There will be some firewalls between remote hosts and the client host when the client host and the remote hosts extend across administrations. When we use ssh for agent activation, programs communicate with the client and agent, using ssh port forwarding . Using the ssh port forwarding function, we can communicate between remote executable modules on the remote host and client program. This is OmniRPC's multiplex communication function. If you want to use this feature, specify the "mxio" attribution in the agent.

<?xml version="1.0" ?>
<OmniRpcConfig>
   <Host name="dennis.hpcc.jp" arch="i386" os="linux">
   <Agent invoker="ssh"  mxio="on"  />
   </Host>
</OmniRpcConfig>
 

The relationship of the agent and rex with this option is as follows. In this case, communication with rex and the client program occurs by way of rex.

Execution of client program

You can execute in same manner without using hosts.xml.

% a.out --hostfile hosts.xml  args...  
      

Of course, to achieve good performance, we use a client program which uses the OmniRPC call in parallel.

Setting of Login name

In addition, if the user name is different on client host and remote host, describe hostfile.xml as follows.

<?xml version="1.0" ?>
<OmniRpcConfig>
   <Host name="dennis.hpcc.jp" user="foo" arch="i386" os="linux">
   <Agent invoker="ssh" mxio="on"  />
   </Host>
</OmniRpcConfig>
 

Setting of cluster environment

When using clusters, we can access at least one computer in a cluster form the client host. We label this computer as the cluster server host and label the computers in the cluster without a cluster server host as the cluster node host.

• Client host is jones.tsukuba.ac.jp .

• Cluster server host is hpc-serv.hpcc.jp. So hpc1, hpc2 and hpc3 are connected to the cluster server host as cluster node hosts.

• Both the cluster server host and cluster node hosts share the same file system.

• Client host can connect directly to the cluster server host and all of the cluster nodes. All port are not limited.

The last item in the list assumes that the client host and cluster are in same network.

Selection of job scheduler

In OmniRPC, the omrpc-agent is invoked first on the cluster server host, and this agent activates remote executable module on each cluster node host with the appropriate scheduler. We can use one of the scheduler described below.

• Built-in round-robin scheduler (rr)

• Portable Batch System (PBS)

• SunGridEngine (SGE)

Use of built-in round-robin scheduler

OmniRPC's built-in round-robin scheduler is a simple scheduler which is implemented in the agent. This scheduler activates remote executable modules on cluster node hosts usign rsh.

To use this scheduler, we create a nodes file which specifies cluster node hosts on the registry ("$HOME/.omrpc-register") of the cluster server host. Below is the setting for this example.

hpc1
hpc2
hpc3

If you want to use ssh instead of rsh inside of a cluster, please add ssh to next of hostname like below file.

hpc1 ssh
hpc2 ssh
hpc3 ssh

On the client host side, we create this hostfile.

<?xml version="1.0" ?>
<OmniRpcConfig>
   <Host name="hpc-serv.hpcc.jp" arch="i386" os="linux">
   <JobScheduler type="rr" maxjob="4" /> 
   </Host>
</OmniRpcConfig>

Set the type attribute in the job scheduler element to the round-robin scheduler "rr." The default value for this attribute is "fork," which just creates the process on the same host. Our example applies to an SMP system. The number of cluster node hosts is 4, so you should set maxjob equal to 4.

The relationship between the agent and rex with this option is as follows.

Cluster in private network

In the above example, the client host and cluster hosts are in same network. Also, the remote executable programs which execute on the cluster node hosts are activated directly for the client host. In the case in which the cluster and client host are in different networks, programs can communicate to the client host from a cluster node host.

But, as the number of node hosts increases, so do the clusters connected to the local-address network. In this situation, only the server host has a global IP address; the node hosts have local IP addresses. For OmniRPC, the cluster node host must communicate with the client host, but in this situation the cluster node host cannot communicate directly with the client host outside the cluster's network.

In this situation, there are 2 ways to use the cluster.

1. Set NAT to communicate with outside networks from the cluster node hosts. Programs can connect to anonymous ports on the client node from each cluster node. For the setting of NAT, please refer to NAT documents.

2. By using the agent function of multiplex communication, the agent relays communications between remote executable programs, which are executed on the cluster node host and client host.

We show an example hostile.xml which is based on the second way to use the cluster.

<?xml version="1.0" ?>
<OmniRpcConfig>
   <Host name="hpc-serv.hpcc.jp" arch="i386" os="linux">
   <Agent invoker="rsh" mxio="on" />
   <JobScheduler type="rr" maxjob="4" />
   </Host>
</OmniRpcConfig>

You should set the mxio attribute on the agent element with "on." In this case, because we assume that the cluster server host and client host are in same network, we use "rsh." If you want to invoke the agent with SSH, set "ssh". If you want to do this with the Globus gate keeper, use "globus."

Using this option, the relationship is shown in the figure below.

The agent relays communications between every rex which is executed on the remote node host and the client.

Cluster outside firewall

You don't have to prepare for this situation.

Cluster inside firewall

We now explain the case of using clusters from outside of firewalls. When there are firewall(s), it is necessary at least to access with ssh to the cluster server host. If you can not use anonymous ports without a port (#22) of ssh, you can use the function of multiplex communication with the agent. We show the hostfile for this example.

<?xml version="1.0" ?>
<OmniRpcConfig>
   <Host name="hpc-serv.hpcc.jp" arch="i386" os="linux">
   <Agent invoker="ssh" mxio="on" />
   <JobScheduler type="rr" maxjob="4" />
   </Host>
</OmniRpcConfig>

Set the mxio attribute "on" in the agent element and use the function of multiplex communication.

In environments which use Globus Toolkit, usually there are no firewalls, so you don't have to prepare. But, you have to set the mxio attribute in the same manner if the clusters consist of private IP addresses.

The relationship between the agent and rex is shown by the figure below.

Communications between the client and rex, which are executed on remote node hosts are relayed by the agent. And communications between the agent and clients are relayed by SSH's port forwarding through the firewall.

What's OmniRpcHandle

OmniRpcHandle is a data structure which presents a connection with a specific remote executable program. OmniRpcHandle is created by activating the remote executable program which corresponds to the module with the OmniRpcCreateHandle API. Once the remote executable program is executing, remote executable programs can accept the requests of another RPC call which is inside of same module, and the program does not need to exit after finishing calculation using a function in the module.

Using this feature, you can do the following.

• Keep the status on the remote executable program side. For instance, in function B you can use data which are set by function A in the same module. In other words, you can reuse the data sent to modules.

• Specify a remote node on which remote executable modules are activated.

With OmniRpcCall API, you specify the remote function only, and call the RPCs. From function name, the client program searches for the modules which contain it. Also, on adequate remote hosts remote program can run the remote executable program which corresponds to it, and assign. However if the client program uses OmniRpcHandle API, you should program the host executable module on which the programs are allocated. The host executable module should be allocated on a remote host. However, problems may arise when remote executable modules fail or are unavailable due to a timeout.

Programming with OmniRpcHandle

For an easy example, we show a program which adds values which set the in setAB function and get its value, and use the IDL file as is.

Define Sample;

Globals {
 int a,b;
}

Define setAB(int in a0, int in b0)
{
   a = a0; b = b0;
}

Define plusAB(int out ab[])
{
     *ab = a + b;
}

In the Globals directive, we define, in a style similar to C language, the variables which are used in the whole module. In the Globals scope, we write the module definitions in the style of C language.

You should compile this module with omrpc-cc, and register Sample.rex on the remote host (alice.is.tsukuba.ac.jp)) with omrpc-register.

#include <OmniRpc.h>
#include <stdio.h>

int main(int argc,char *argv[]){
   int ab;
   OmniRpcHandle handle;

   OmniRpcInit(&argc,&argv);
   handle = OmniRpcCreateHandle("alice.is.tsukuba.ac.jp","Sample");

   OmniRpcCallbyHandle(handle,"setAB",10,20);
   OmniRpcCallbyHandle(handle,"plusAB",&ab);
   printf("a+b=%d\n",ab);
  
   OmniRpcHandleDestroy(handle);

   OmniRpcFinalize();
   exit(0);
}

After the modules are initialized, the client program first activates the remote executable programs with OmniRpcCreateHandle API, and gets the OmniRpcHandle corresponding to it. Otherwise, by using OmniRpcCallbyHandle, the client program can call functions in the modules. Finally, the client program can stop the remote executable program with OmniRpcDestroyHandle API.

Also available is OmniRpcCallAsyncByHandle API, which call RPCs asynchronously.

Acquisition of host information.

With OmniRpcCreateHandle API, you specify the remote host name on which the remote executable modules run. However, if you don't, programs allocate the appropriate remote host on modules which are registered.

What's auto module initialization

Auto module initialization is a function which calls the initialization method automatically when modules of a remote executable program are invoked.

By using OmniRpcHandle API, you can write efficient programs which keep data on a remote executable program, but you should write the program which is used by the remote executable program. We only specify the remote functions which are specified by OmniRpcCall at the start, the OmniRPC system executes and runs appropriate remote executable programs for requests. But we cannot know which remote executable programs are allocated by the OmniRPC system, so it is impossible to set the data beforehand.

In some OmniRPC master/worker programs, the master and worker share common data. Workers calculate its data with different parameters which are taken from the master. Auto module initialization is convenient in cases like this. In OmniRPC, if there is a function named "Initialize" in a module, "Initialize" is called automatically when a new remote executable program is activated for an other function's RPC call. Common data are set in initialization, and it can be efficient to reuse this data when function calls are called by real varied parameters. The bigger the costs of setup, the bigger are the effects.

Programming with OmniRpcModuleInit API

For example, we will use a program to calculate the appearance of 10 sorts of strings (at a maximum of 10 characters) from a string which is 10000 characters in size. The sequential version may be written as follows.

#include <stdio.h>
#include <string.h>

char data[10000];  /* data to be searched */
char str[10][10];  /* string to be compared */
int occurrence[10];  /* array to record occurrence */

/* prototype */
void count_occurrence(char *data,char *str, int *r);

int main(int argc, char *argv[])
{
    FILE *fp;
    int i;

    if((fp = fopen("data","r")) == NULL){
	fprintf(stderr,"cannot open data file\n");
	exit(1);
    }
    fread(data,10000,1,fp);
    fclose(fp);
    
    if((fp = fopen("strings","r")) == NULL){
	fprintf(stderr,"cannot open strings file\n");
	exit(1);
    }
    for(i = 0; i < 10; i++)
	fscanf(fp,"%s",str[i]);
    fclose(fp);

    for(i = 0; i < 10; i++)
	count_occurrence(data,str[i],&occurrence[i]);
    
    for(i = 0; i < 10; i++)
	printf("string(%i,'%s') occurrence=%d\n",i,
	       str[i],occurrence[i]);
    
    exit(0);
}

void count_occurrence(char *data,char *str, int *r)
{
    int i,len,count;
    len = strlen(str);
    count = 0;
    for(i = 0; i < 10000-len; i++){
	if(strncmp(&data[i],str,len) == 0) count++;
    }
    *r = count;
}

To parallize this program with OmniRpcCallAsync, we calculate count_occurrence in the remote executable program. The IDL file may be like the example shown below.

Module count_occurrence;

Define count_occurrence(char IN data[10000],char IN str[10], int OUT
r[]) Calls "C" count_occurrence(data,str,r);

Link this with the count_occurrence function. and register it. In the client program, we change a call of count_occurrence to a call of OmniRpcCallAsync API.

int main(int argc, char *argv[])
{
    FILE *fp;
    int i;
    OmniRpcReqeust reqs[10];

    OmniRpcInit(&argc,&argv);

    ... /* input data */

    for(i = 0; i < 10; i++)
    	reqs[i] = OmniRpcCallAsync("count_occurrence",
	                      data,str[i],&occurrence[i]);
   OmniRpcWaitAll(10,reqs);
			          
    ... 
   OmniRpcFinalize();
   exit(0);
}

In this case, there is the problem that data are sent for each RPC call on the remote executable module. This data is unchanged, so it is efficient to reuse the data sent on the remote executable program side.

By using OmniRpcModuleInit API, the client program sends the data in the initialization of the remote executable programs, and sends only the strings which are searched with OmniRpcCall. We define the IDL file as follows.

Module count_occurrence;

Globals {
#include <string.h>
char data[10000];
}

Define Initialize(char IN input_data[10000])
{
     memcpy(data,input_data,10000);
}

Define count_occurrence_each(char IN str[10], int OUT r[]){
    count_occurrence(data,str,r);
}

In the client program, initialization is described.

int main(int argc, char *argv[])
{
    FILE *fp;
    int i;
    OmniRpcReqeust reqs[10];

    OmniRpcInit(&argc,&argv);

    ... /* input data */

    OmniRpcModuleInit("count_occurrence",data);

    for(i = 0; i < 10; i++)
    	reqs[i] = OmniRpcCallAsync("count_occurrence_each",
	                      str[i],&occurrence[i]);
   OmniRpcWaitAll(10,reqs);
    ... 
   OmniRpcFinalize();
   exit(0);
}

We specify the necessary data in the initialization with OmniRpcModuleInit. The initializations are indeed taken when the necessary remote executable modules are activated. So, it is important not to write in the area addressed by the pointer variables in initialization.

The case of direct activation of remote executable program

Usually in OmniRPC, the client program executes a remote executable program via the agent. Here are some advantages.

• By registering to the registry, it is possible to activate remote executable modules on the remote host without knowing its path. Furthermore, in the case of the same module's name, it is possible to allocate the job to the appropriate remote host.

• When the remote host is a cluster, it is possible to allocate jobs to the cluster node host.

In this explanation, without the agent, we introduce a method to execute directly the remote executable module and to call RPCs. If there are remote executable programs for which you know the path in a certain remote host, you can omit registering remote executable programs, and so on. The advantages as are listed below.

• Without describing a communication protocol, programs call function by using OmniRPC protocol.

• By activating using Globus and ssh, the function of port forwarding is available in authentication.

Setting environment for remote nodes.

• Client host is jones.tsukuba.ac.jp.

• Remote host is dennis.hpcc.jp.

• Set calc_sin.rex, which we introduced already as an example program, to "/usr/local/tmp/" .

• Globus Toolkit's gate-keeper is running on dennis.hpcc.jp, and it is possible to execute programs with GRAM.

If you use this function, it executes rex directly without the agent.

API for direct execution of remote program

Because, we don't use the agent for direct execution, initialization of the library is taken by OmniRpcExecInit() API.

APIs which activate remote executable programs on the remote host, are like APIs which use OmniRpcHandle. OmniRpcExecOnHost() API enables the execution of a remote executable program, which is specified by its path, on the specified remote host, and it returns OmniRpcExecHandle, which presents its connection. By using OmniRpcExecCall it is possible to call a function which is inside a module.

We show this example below.

#include <OmniRpc.h>
#include <stdio.h>

int main(int argc,char *argv[]){
   double r;
   OmniRpcExecHandle handle;

   OmniRpcExecInit(&argc,&argv);
   handle = OmniRpcExecOnHost("dennis.hpcc.jp","/usr/local/tmp/calc_sin.rex");

   OmniRpcExecCall(handle,"calc_sin",10,&r);
   printf("sin(10)=%g\n",r);
  
   OmniRpcExecTerminate(handle);

   OmniRpcExecFinalize();
   exit(0);
}

OmniRpcExecTerminate API enables the termination of the remote executable program which responds to the handle. Finally, you should use OmniExecFinalize() at the end of the program.

Program execution

You should specify the path of an executable program to activate the remote executable program. For example, the case of an execution using Globus is as follows.

% a.out --globus args...
      

In the case of SSH activation, you specify "--ssh."; if don't specify this, rsh is used to activate.

The limitation of direct activation of remote executable programs.

At this time, there are some limitations with this method.

• No support for a multi-thread environment (no thread-safe)

• No support for asynchronous calls.

• No support for activating with multiple methods of activation.

We hope to improve these issues in the future.

File Transfer using Filename

OmniRPC supports file transfer between client program and remote executable program. In file transfer mode, you should spcify filenames which you want to transfer files.

Module testfile;

Globals { 
#include <math.h>
}

Define testfile(IN filename infile[2], OUT filename outfile[])
{
    FILE *infp, *outfp;
    char tmp[128];
    int i;

    fprintf(stderr, "STUB : infile[0] : %s\n", infile[0]);
    fprintf(stderr, "STUB : infile[1] : %s\n", infile[1]);


    if((outfp = fopen(*outfile, "w+")) == NULL){
        perror("fopen");
        exit(1);
    }
    for(i = 0; i < 2; i++){
        if((infp = fopen(infile[i], "r+")) == NULL){
            perror("fopen");
            exit(1);
        }

        while(fgets(tmp, sizeof(tmp), infp) != NULL){
            fprintf(outfp, "%s", tmp);
        }
        fclose(infp);
    }

    fclose(outfp);
    fprintf(stderr, "STUB : END\n");
    fflush(stderr);
}

         % omrpc-cc testfile.idl
        

#include <OmniRpc.h>
#include <stdio.h>

int main(int argc,char *argv[])
{

   char *infile[2] = {"a.txt", "b.txt"};
   char *outfile = "a.out";
   char tmp[128];
   FILE *fp;
   int i,j;
   int c = 0;

   OmniRpcInit(&argc,&argv);

   for(i = 0 ;i < 2; i++){
       if((fp = fopen(infile[i], "w+")) == NULL){
           perror("cannot open file");
           exit(1);
       }
       for(j =0; j < 128; j++)
           fprintf(fp, "%i\n",c++);
       fflush(fp);
       if(fclose(fp) != 0){
           perror("cannot close file");
       }
   }

   OmniRpcCall("testfile", infile, &outfile);

   fprintf(stderr, "TRANSFER FIN\n");
   fflush(stderr);

   if((fp = fopen(outfile, "r+")) == NULL){
       perror("cannot open file");
       exit(1);
   }
   while(fgets(tmp, sizeof(tmp), fp) != NULL){
       fprintf(stdout, "OUTPUT:%s", tmp);
   }

  OmniRpcFinalize();
}

         % omrpc-cc -o testfile testfile.c 
        

% testfile --hostfile hosts.xml
        

<OmniRpcConfig>
   <Host name="alice.hpcc.jp" user="foo" arch="i386" os="linux">
        <Agent invoker="globus" mxio="on" path="/usr/local/omrpc"/>
        <JobScheduler type="rr" maxjob="6" />
        <Registry path="/home/foo/app/stubs" />
        <WorkingPath path="/home/foo/tmp" />
   </Host>
</OmniRpcConfig>

A simple example in FORTRAN

Let's think about this Fortran program. This program calculates the inner product of a matrix. The main program is called main.f, and ip.f is the subroutine that calculates the inner product.

main.f

      double precision a(10),b(10),r
      do i = 1,10
         a(i) = i
         b(i) = i+10
      end do
      call innerprod(10,a,b,r)
      write(*,*) 'result=',r
      end

ip.f

      subroutine innerprod(n,a,b,r)
      integer n
      double precision a(*),b(*),r
      integer i
      r = 0.0
      do i = 1,n
        r = r+a(i)*b(i)
      end do
      return
      end

Let's set this program to call the innerprod subroutine with OmniRPC.

Create remote executable program

As in a C program, we create a program which executes a subroutine on remote hosts. So, we define an interface to innerprod. We set the module name as f_innerprod, and the function name as innerprod.

Module f_innerprod;

Define innerprod(IN int n, IN double a[n], IN double b[n],
        OUT double result[1])
Calls "Fortran" innerprod_(n,a,b,result);

The argument which specifies the array size must be a scalar variable. The double precision type in FORTRAN is "double", the real type is "float". If you directly call a FORTRAN function with calls, you specify "Fortran". Calling a function can mangle the function name in FORTRAN. Usually, in the case of the FORTRAN compiler, a mangled name is a name to which "_" has been added. Please pay attention to whether a function name contains "_"; if it is mangled, add "__" in g77 (gcc).

This definition is the same as the above definition. As scalar variable is taken as the address pointer.

Module f_innerprod;

Define innerprod(IN int n, IN double a[n], IN double b[n],
        OUT double result[1])
{
    innerprod_(&n,a,b,result);
}

Generate OmniRPC's remote executable modules from this IDL file. We name this IDL file f_ip.idl.

%  omrpc-fc f_ip.idl ip.f
      

When you run this command, the remote executable program f_ip.rex is generated. To register with omrpc-register, the method is the same as in C language.

%  omrpc-register -register f_ip.rex
      

The command omrpc-fc, it compiles FORTRAN program with f77. If you want to use another compiler, use the "-fc" option. For example, if you want to use the intel Fortran compiler of ifc, run the following command.

%  omrpc-fc -fc ifc f_ip.idl ip.f
      

Client program in Fortran

And now, change the client program main.f to call the remote executable program with OmniRPC.

      double precision a(10),b(10),r
      call OMINRPC_INIT
      do i = 1,10
         a(i) = i
         b(i) = i+10
      end do
      call OMNIRPC_CALL("f_innerprod*",10,a,b,r)
      write(*,*) 'result=',r
      call OMNIRPC_FINALIZE
      end

This program initializes with OMNIRPC_INIT and calls by OMNIRPC_CALL. Please keep in mind that the entry name which is specified by OMNIRPC_CALL should have "*" at the end of the name which is defined on the interface. You should add "*" to the end of strings which are obtained with OmniRPC, such as the module name and host name. OmniRPC considers "*" to be a string terminator in OmniRPC's Fortran API.

Using omrpc-fc to compile this file.

%  omrpc-fc main.f
      

An asynchronous call in FORTRAN

Finally, we show an example which is written in FORTRAN. this example is also shown in the section Parallel programming with OmniRPC .

      double precision res(10)
      double precision x
      integer ireqs(10)
      call omnirpc_init
      x = 0.0
      do i = 1, 10
         call omnirpc_call_async(ireqs(i),'calc_sin*',x,res(i))
         x = x + 1.0
      end do
      call omnirpc_wait_all(10,ireqs)
      do i = 1, 10
         write(*,*) 'res(',i, ')=',res(i)
      end do
      call omnirpc_finalize
      end

For API details, see Fortran API