[Hedge] starting out with mpi

[Derek Steinmoeller]

Hi Andreas,

Thanks for the reply and the wiki update. It seems that using mpirun as
you described works for the Maxwell equations, but I get Python errors
when trying the Euler codes. These drivers seem to run fine in serial
(without using mpirun), though.

The last line of the traceback is:
  File "/home/dsteinmo/src/hedge/hedge/discretization/__init__.py", line
1070, in boundarize_volume_field
    result = self.boundary_empty(tag, shape=ls, dtype=field[0].dtype)
AttributeError: 'int' object has no attribute 'dtype'

Any thoughts on how to fix this problem? I also attached a text-file of
the full traceback with the call to mpirun, if that helps.

Thanks,
Derek.

On Thu, 2011-05-05 at 14:29 -0400, Andreas Kloeckner wrote:
> Hi Derek,
> 
> On Thu, 05 May 2011 13:41:23 -0400, Derek Steinmoeller <dsteinmo at math.uwaterloo.ca> wrote:
> > I'm working on solving some dispersive water wave equations in lake
> > geometries, and have already had a good deal of success solving these
> > equations by modifying/combining some of the Matlab codes provided with
> > the NUDG book by Hesthaven and Warburton. The next step seems to be
> > getting a solver that will run in a compiled language and in parallel.
> > 
> > I've ran "test_parallel.py" on an ubuntu machine, which seems to solve
> > the advection equation in parallel by randomly distributing elements to
> > processors. I guess my main question is, are there any other example
> > codes that perform the decomposition in a smarter way to give
> > efficiency?
> 
> (By default, partitions are generated using (py)metis.)
> 
> > I haven't seen much discussion in the wiki about solving
> > problems in parallel; is there anywhere else I can find more
> > documentation on the steps needed for an efficient parallel solve?
> 
> Added:
> http://wiki.tiker.net/Hedge/HowTo/RunningInParallel
> 
> Andreas
-------------- next part --------------
dsteinmo at belize:~/src/hedge/examples/gas_dynamics/euler$ mpirun -np 2 python sod-2d.py -f mpi
/home/dsteinmo/src/hedge/hedge/partition.py:177: UserWarning: make_conformal_mesh is deprecated. Use make_conformal_mesh_ext instead.
  is_partbdry_face)
send rank 1
end send 1
sod-2d.py:80: DeprecationWarning: argument gamma is deprecated in favor of equation_of_state
  source=None)
sod-2d.py:80: DeprecationWarning: argument gamma is deprecated in favor of equation_of_state
  source=None)
/home/dsteinmo/src/hedge/hedge/optemplate/mappers/__init__.py:912: UserWarning: No minimum degree for quadrature tag 'gasdyn_vol' specified--falling back to nodal evaluation
  "falling back to nodal evaluation" % expr.op.quadrature_tag)
/home/dsteinmo/src/hedge/hedge/optemplate/mappers/__init__.py:912: UserWarning: No minimum degree for quadrature tag 'gasdyn_face' specified--falling back to nodal evaluation
  "falling back to nodal evaluation" % expr.op.quadrature_tag)
/home/dsteinmo/src/hedge/hedge/optemplate/mappers/__init__.py:912: UserWarning: No minimum degree for quadrature tag 'gasdyn_vol' specified--falling back to nodal evaluation
  "falling back to nodal evaluation" % expr.op.quadrature_tag)
/home/dsteinmo/src/hedge/hedge/optemplate/mappers/__init__.py:912: UserWarning: No minimum degree for quadrature tag 'gasdyn_face' specified--falling back to nodal evaluation
  "falling back to nodal evaluation" % expr.op.quadrature_tag)
Traceback (most recent call last):
  File "sod-2d.py", line 183, in <module>
    main()
  File "sod-2d.py", line 89, in main
    rhs(0, fields)
  File "sod-2d.py", line 86, in rhs
    ode_rhs, speed = euler_ex(t, q)
  File "/home/dsteinmo/src/hedge/hedge/models/gas_dynamics/__init__.py", line 841, in rhs
    **extra_kwargs
  File "/home/dsteinmo/src/hedge/hedge/backends/jit/__init__.py", line 485, in __call__
    self.discr.exec_mapper_class(context, self))
  File "/home/dsteinmo/src/hedge/hedge/compiler.py", line 492, in execute
    return self.execute_dynamic(exec_mapper, pre_assign_check)
  File "/home/dsteinmo/src/hedge/hedge/compiler.py", line 447, in execute_dynamic
    insn.get_executor_method(exec_mapper)(insn)
  File "/home/dsteinmo/src/hedge/hedge/backends/mpi/__init__.py", line 270, in exec_flux_exchange_batch_assign
    for rank in pdiscr.neighbor_ranks]
  File "/home/dsteinmo/src/hedge/hedge/backends/mpi/__init__.py", line 150, in __init__
    field, TAG_RANK_BOUNDARY(rank), kind="numpy")
  File "/home/dsteinmo/src/hedge/hedge/discretization/__init__.py", line 1083, in boundarize_volume_field_async
    self.boundarize_volume_field(field, tag, kind))
  File "/home/dsteinmo/src/hedge/hedge/discretization/__init__.py", line 1070, in boundarize_volume_field
    result = self.boundary_empty(tag, shape=ls, dtype=field[0].dtype)
AttributeError: 'int' object has no attribute 'dtype'
Traceback (most recent call last):
  File "sod-2d.py", line 183, in <module>
    main()
  File "sod-2d.py", line 89, in main
    rhs(0, fields)
  File "sod-2d.py", line 86, in rhs
    ode_rhs, speed = euler_ex(t, q)
  File "/home/dsteinmo/src/hedge/hedge/models/gas_dynamics/__init__.py", line 841, in rhs
    **extra_kwargs
  File "/home/dsteinmo/src/hedge/hedge/backends/jit/__init__.py", line 485, in __call__
    self.discr.exec_mapper_class(context, self))
  File "/home/dsteinmo/src/hedge/hedge/compiler.py", line 492, in execute
    return self.execute_dynamic(exec_mapper, pre_assign_check)
  File "/home/dsteinmo/src/hedge/hedge/compiler.py", line 447, in execute_dynamic
    insn.get_executor_method(exec_mapper)(insn)
  File "/home/dsteinmo/src/hedge/hedge/backends/mpi/__init__.py", line 270, in exec_flux_exchange_batch_assign
    for rank in pdiscr.neighbor_ranks]
  File "/home/dsteinmo/src/hedge/hedge/backends/mpi/__init__.py", line 150, in __init__
    field, TAG_RANK_BOUNDARY(rank), kind="numpy")
  File "/home/dsteinmo/src/hedge/hedge/discretization/__init__.py", line 1083, in boundarize_volume_field_async
    self.boundarize_volume_field(field, tag, kind))
  File "/home/dsteinmo/src/hedge/hedge/discretization/__init__.py", line 1070, in boundarize_volume_field
    result = self.boundary_empty(tag, shape=ls, dtype=field[0].dtype)
AttributeError: 'int' object has no attribute 'dtype'