PySequencer

Introduction

Midas contains a Sequencer that can be used to automate data-taking (e.g. starting and stopping many runs with different settings). The original Seqeuncer uses the custom Midas Script Language (MSL), which has a limited syntax. For more demanding tasks requiring complex calculations, a Python-based sequencer has been developed in 2025. The idea is very similar to the MSL sequencer and the user interface is exactly the same, but instead of MSL commands the PySequencer accepts python programs. The interaction with the midas system is done via a special "seq" object. Following example illustrates a simple script asking for a number of runs and then starting/stopping these runs, each lasting 60 seconds:

# This is PySequencer test file

def define_params(seq):
    seq.register_param("runs", "Number of runs", 3)

def sequence(seq):
    runs = seq.get_param("runs")

    for r in seq.range(runs):
        seq.start_run()
        seq.wait_seconds(60)
        seq.stop_run()

The parameters are optional and can be committed. If defined, a dialog box opens when the user starts the sequence and prompts the user to input the parameter values to use.

The user can track the state of the sequence via a webpage that shows:

• the current line in the script, highlighted in yellow
• the value of any parameters/variables, shown in a table
• the progress of any loops or waits, shown as progress bars

Installation

Before the PySequencer can be used, python must be installed and enabled for midas. See Python Installation for details.

Next, the PySequencer menu must be enabled in the ODB with /Experiment/Menu/PySequencer = 1. Now one can click on the "PySequencer" menu item and use the GUI in a similar way than the MSL sequencer. Syntax highlighting happens according to python, and all python variables are shown at the right side of the page for debugging purposes. If the PySequencer is not running, the GUI asks to start python3 $MIDASSYS/python/midas/sequencer.py -D. If this command failed, the user should try to execute it manually to see if there are any errors.

Command-line arguments

usage: sequencer.py [-c name] [-h host_name] [-e expt_name] [-D] [-O] [--verbose] [--log-to-midas] [--help]

options:
  -c name         Name of additional sequencer. E.g. if 'Test', ODB location will be /PySequencerTest
  -h host_name
  -e expt_name
  -D              Become a daemon
  -O              Become a daemon but retain stdout
  --verbose
  --log-to-midas  Write to midas message log as well as stdout/stderr (only for logging.info() etc, not regular print() statements)
  --help          Show this help message and exit

Commands

PySequencer is generally designed to be controlled via the web interface. However it can also be controlled via settings in the ODB. Most settings are in /PySequencer/Command, but a couple are also in /PySequencer/State

• In /PySequencer/Command
  • Load new file - load the file specified in /PySequencer/State/Filename
  • Start script - start the script that has been loaded
  • Pause script - pause the script by preventing the python interpreter from executing the next line of code
  • Resume script - resume the script after pausing it
  • Stop immediately - stop the sequence() function as soon as possible
  • Stop after run - stop the sequence() function when the next run ends
  • Cancel stop after run - cancel the "Stop after run" request
  • Debug script - start script in debug mode where we won't move to the next line until "Step over" is set (usually via button on webpage)
  • Step over - move to next line when running in debug mode
• In /PySequencer/State
  • Path - specify a subdirectory beneath <experiment_dir>/userfiles/sequencer where your files are located
  • Filename - filename to load (relative to <experiment_dir>/userfiles/sequencer or <experiment_dir>/userfiles/sequencer/<path>)
  • Message - if "Message wait" is true, the sequence will not continue until "Message" has been reset to an empty string - see the messages section for more details

User script content

All scripts must be located in the userfiles/sequencer subdirectory of your experiment directory. PySequencer will only load files with a .py extension.

PySequencer will look for three specific functions in your file:

• define_params(seq) - optional
• sequence(seq) - required
• at_exit(seq) - optional

These are described in more detail below. All three should accept one argument, which is a midas.sequencer.SequenceClient object. This object is also available as a global variable within your script (e.g. if you want to define helper functions and don't want to pass the seq object to each of them).

Your file may include any other modules, and you can define/call your own functions. If you edit the content of your file, you can reload it using the web interface or by setting the ODB parameter "/PySequencer/Command/Load new file" to true. When we reload a file, we also reload all modules that it uses (i.e. if you make changes to a custom module that contains helper functions, you can trigger loading the new definitions by reloading your main script).

define_params(seq) function

This function is optional, and is used to create "parameters" that the user will be prompted for when they start the sequence. It is expected to be a set of seq.register_param() calls. For each parameter, you specify a name, a comment that will be shown in the prompt dialog, a default value, and optionally a list of acceptable values.

The "default value" you provide determines the type of the variable in the ODB. If you specify a string it will be a TID_STRING, an integer will become TID_INT32 etc.

def define_params(seq):
    # Will be stored as an integer
    seq.register_param("runs", "Number of runs", 3)

    # Will be stored as a float
    seq.register_param("voltage", "PMT voltage", 1342.7)

    # Will be stored as a string, and user will see a dropdown to choose from
    seq.register_param("run type", "Purpose of this run", "normal", ["normal", "calibration", "testing"])

The main sequence() function can then retrieve the values of these parameters by calling seq.get_param("voltage") etc.

sequence(seq) function

This function MUST be defined, and is the main script that will be executed. Full documentation of the functions available is in the seq object reference section, but you may call/define your own functions as well, and also call functions from other modules.

There are a few "best practices" to consider to maximise integration with the webpage:

• Use seq.wait_seconds() instead of time.sleep() so you get a progress bar on the webpage. seq.sleep() is an alias for seq.wait_seconds() if you prefer.
• In a normal "for" loop use seq.range() instead of range() so you get a progress bar on the webpage. We don't have a way to show progress through any other type of for/while loop
• Avoid any very long-lasting calls (e.g. blocking on a socket) as these may prevent us from being able to stop the sequence when the user calls the "Stop immediately" command. Better to poll in a loop until a condition is met.

at_exit(seq) function

This function is optional, and is used to "tidy up" when the sequence stops (either due to the sequence() function exiting normally, the user requesting "Stop immediately" or "Stop after run", or the sequence() function throwing an exception). For example, you may wish to move a motor back to a default location, or ramp down PMT voltages etc.

Note that this is not bulletproof and should not be relied on for safety-critical items! Examples of situations where at_exit() will NOT be called include:

• sequencer.py being killed by a user (Ctrl-C etc) or the OS (out-of-memory killer, rebooting etc)
• the user's sequence() function corrupting the python interpreter so badly that it affects the main thread
• the user's sequence() function entering an uninterruptible state where "Stop immediately" can't actually stop anything (e.g. blocking wait on a socket)
• unidentified bugs in sequencer.py

Messages

There are two different types of messages in the sequencer:

• regular Midas messages that get written to the message log
• sequencer-specific messages that show as a dialog box on the sequencer webpage

You can emit regular midas messages using the seq.msg(message, is_error=False, facility="midas") function. Error messages get highlighted in red on the message page. The "facility" routes the message to different log files / different buttons on the message page.

You can emit sequencer-specific messages using the seq.sequencer_msg(text, wait=False) function. If "wait" is True, the sequence will not continue until the user has clicked the "Close" button on the webpage to acknowledge the message.

Logging

PySequencer is integrated with the python logging module. You may use the global logger variable to log your own info/debug messages. You can configure what happens to these logger messages using either command-line arguments or functions in the seq object.

To print logger.debug() messages, you can pass the --verbose command-line flag and/or call seq.set_py_logger_debug(True) in your script. The default is to only print info() messages or higher.

You can also copy messages to the regular midas message log (so you can call logger.info() instead of seq.msg(), and still have messages appear in the midas message log). The "facility" in this case is "pysequencer". You can enable this functionality using the --log-to-midas command-line flag and/or by calling seq.set_py_logger_to_midas(True) in your script. This is most useful for experiments where terminal access is very limited, and they wish to debug/monitor everything via log files that are visible through webpages.

seq object reference

Users have access to the seq object in their scripts, which is a midas.sequencer.SequenceClient object. This class inherits from the standard midas.client.MidasClient object, and adds extra sequencer-specific tools.

Full documentation of these classes can be found in docstrings in the source code:

• MidasClient - https://bitbucket.org/tmidas/midas/src/develop/python/midas/client.py
• SequenceClient - https://bitbucket.org/tmidas/midas/src/develop/python/midas/sequencer.py

A summary of the available commands is:

• Sequencer-specific:
  • seq.register_param(name, comment, default_value, options=[])
  • seq.get_param(name)
  • seq.sequencer_msg(text, wait=False)
  • seq.range(n)
  • seq.wait_seconds(n)
  • seq.wait_odb(path, op, target, between_upper_target=None, stable_for_n_secs=None)
  • seq.wait_func(func, check_period_secs=0.1)
  • seq.wait_clients_running(client_names, timeout_secs=10)
  • seq.wait_events(target)
  • seq.set_run_description(desc)
• Run control
  • seq.start_run()
  • seq.stop_run()
  • seq.pause_run()
  • seq.resume_run()
• ODB access
  • seq.odb_get(path, recurse_dir=True, include_key_metadata=False)
  • seq.odb_set(path, contents, .....) - many optional parameters!
  • seq.odb_exists(path)
  • seq.odb_delete(path, follow_links=False)
  • seq.odb_link(link_path, destination_path)
  • seq.odb_get_link_destination(link_path)
  • seq.odb_rename(current_path, new_name)
  • seq.odb_last_update_time(path)
• Event buffers
  • seq.open_event_buffer(buffer_name, buf_size=None, max_event_size=None)
  • seq.register_event_request(buffer_handle, event_id=-1, trigger_mask=-1, sampling_type=midas.GET_ALL)
  • seq.receive_event(buffer_handle, async_flag=True, use_numpy=False)
  • seq.deregister_event_request(buffer_handle, request_id)
  • seq.send_event(buffer_handle, event)
• Controlling other clients
  • seq.start_client()
  • seq.stop_client()
  • seq.get_all_required_program_names()
  • seq.start_all_required_program()
  • seq.client_exists()
  • seq.clients_exist()
  • seq.connect_to_other_client()
  • seq.disconnect_from_other_client()
  • seq.jrpc_client_call()
  • seq.brpc_client_call()
• Alarms
  • seq.get_triggered_alarms()
  • seq.reset_alarm(alarm_name)
  • seq.create_alarm_class(class_name, execute_command="", execute_interval_secs=0, stop_run=False)
  • seq.create_evaluated_alarm(alarm_name, odb_condition, message=None, alarm_class="Alarm", activate_immediately=True)
  • seq.trigger_internal_alarm(alarm_name, message, default_alarm_class="Alarm")
• History
  • seq.hist_create_plot(group_name, panel_name, variables, labels=[])
  • seq.hist_get_data(start_time, end_time, interval_secs, event_name, tag_name, index=None, timestamps_as_datetime=False)
  • seq.hist_get_events()
  • seq.hist_get_recent_data(num_hours, interval_secs, event_name, tag_name, index=None, timestamps_as_datetime=False)
  • seq.hist_get_tags(event_name)
• Messages
  • seq.msg(message, is_error=False, facility="midas")
  • seq.sequencer_msg(text, wait=False)
  • seq.get_recent_messages(min_messages=1, before=None, facility="midas")
  • seq.get_message_facilities()
• Logging
  • seq.set_py_logger(debug, to_midas_msg_log)
  • seq.set_py_logger_debug(debug)
  • seq.set_py_logger_to_midas(to_midas_msg_log)
• Misc
  • seq.get_midas_version()
  • seq.get_experiment_dir()
• Functions that are in MidasClient, but may NOT be used in a sequencer script
  • seq.odb_watch()
  • seq.register_jrpc_callback()
  • seq.register_brpc_callback()
  • seq.register_disconnect_callback()
  • seq.register_transition_callback()
  • seq.communicate()

Limitations

Variables shown on webpage

The "variables" table on the webpage will display only local/global variables of the types: int, float, string, bool, numpy.number and numpy.bool. It will also display lists and numpy.ndarrays of those types. It does not display any other types (e.g. datetime.datetime, custom objects, dicts etc). We also don't show any variables where the name starts with a double underscore.

No callback functions

Your script cannot use any of the callback-based functions present in MidasClient. This includes odb_watch(), register_jrpc_callback(), register_brpc_callback(), register_message_callback(), register_transition_callback(), register_disconnect_callback() and communicate(). This is because your script runs in a separate thread, but the main thread is the one responsible for talking to midas; any callbacks get executed in the context of the main thread, not your script's thread.

If you want to check whether an ODB value has changed, you'll need to implement a loop and call odb_get() repeatedly.

You MAY use register_event_request() if you want to look at data as part of your script, as you then call receive_event() to explicitly look at the events, rather than relying on a callback function.

No tracing in other modules

It is expected that experiments will build a library/module of helper functions to automate certain tasks (e.g. move_calibration_source(x, y)). Your sequencer script can import such a module and call the functions, but the webpage will not show what is happening within that module - it would just show that we're in move_calibration_source().

It is technically possible to enhance the PySequencer logic to trace within these modules, but it is not yet clear whether users want such a feature, and what the best user interface would be.