All you can do (directly) is turn quick mask mode on and off.
This will turn it on:
Code:
app.activeDocument.quickMaskMode = true;
and this will turn it off:
Code:
app.activeDocument.quickMaskMode = false;
If there is any additional quick-mask functionality, it is not documented.
-X