Running large jobs in Matlab

Running large numbers of Matlab jobs is complicated, because each concurrent job ordinarily requires a separate licence, and only a limited number of licenses are available locally. To get around this, you can compile your Matlab job, as described below, and the compiled code will then not use up the licenses. Alternatively, you can try running your code under Octave; most Matlab code that doesn't use toolboxes should run as-is. Or you can use other numerical packages, such as R or SciPy (for Python).

Using Matlab on ECDF

A full version of matlab will be running on ECDF from the end of November onwards. According to the latest information there will be 2 licenses of matlab available to use interactively (or in batch mode) in the frontend, two licenses of DCT and two licenses of matlab's DCE. Please see below for some explanation on what that means in terms of number of users and concurrent jobs.

  • General information on how to submit a job can be found on: Matlab on EDDIE
  • If you need to run a job that requires more than 2GB of memory, use @sgeMemSubmitFcn rather than @sgeSubmitFcn e.g memory_slots = '2'; set(sched, 'SubmitFcn', {@sgeMemSubmitFcn, time_limit, memory_slots});
  • If you want to check whether your job finished with errors look under the directory where you store your log files. Find the job directory and load the out.mat file for a given task. This will load the errorstruct structure, which holds the relevant information. e.g. in matlab load $logdir/Job1/Task1.out.mat
  • If you have many functions to run within a job, specify 'PathDependencies' rather than 'FileDependencies'. e.g. file_dep ={'dir1' 'dir2'}; createJob(sched,'PathDependencies',file_dep);
  • UPDATE: Eddie is now accessible for NFS mount to DICE machines. Please contact dice support for access.
  • UPDATE on matlab licenses:
    1. If using the ECDF frontend:
      1. submitting a job through matlab's environment: there are only two licenses available, i.e only two users can be logged into matlab at any one time.
      2. submitting a job through UNIX (calling "matlab -r"): there are only two licenses available, i.e. your job will either wait until a license is available or it will quit giving you a no-license-available error (this information will need to cross-checked, please edit accordingly).
    2. If submitting from your desktop (remote client), you are using your own licenses, i.e. you are not affected by the number of matlab core or DTC licenses available. To submit a job remotely, you will also need to have personal matlab DCT licenses.
    3. Once a job is submitted (either from the frontend or the remote client), the number of DCE licenses controls how many tasks can run concurrently. At the moment there are licenses for 8 workers only. E.g. at any one point a maximum of 8 distributed tasks can run concurrently (the rest will wait for the next available slot). For parallel jobs it means that a job can be maximally distributed among 8 workers only.

Using Matlab with Condor

Clare Giacomantonio has prepared some slides on how to use compiled Matlab code under Condor.

Topic revision: r7 - 19 Nov 2007 - 16:18:33 - Main.s0234516
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies