If you have installed Maya 2011 Hotfix 3 and see errors in the terminal or a black Hotbox in the viewport, we advise you to disable the CIOverlays option in your xorg.conf file, located in /etc/X11/xorg.com.
With the move to Qt in 2011 we no longer have a need for colour indexing. Instead, as of Hotfix 3 we have introduced RGB overlays and this option should be used instead.
- Open your xorg.conf file
- Remove following line:
Option "CIOverlay" "ON"
- Add following line:
Option "Overlay" "ON"
The caveat here is that you cannot set both CIOverlay and Overlay at the same time, and users of Maya 2010 will be aware of the instabilities that can arise when working without CIOverlay.
So what do you do if you use both Maya 2010 and Maya 2011 on a daily basis ?
Enable CIOverlay - for Maya 2010 - set the following Maya environment variable, in the Maya.env file located in: ~/maya/2011:
MAYA_HOTBOX_USE_SOFTWARE = 1
This variable forces Maya Hotfix 3 to fallback on an internal software composition mode to achieve the effect of the transparent hotbox , of the kind used by default in Maya 2011 release 1, and Hotfixes 1 and 2.
If you set this variable, it doesn’t change the setting set in the xorg.conf file. Maya will no longer check for RGB overlay and use its internal software composition mode
This variable has no effect on versions of Maya prior to 2011.
What does it do exactly ?
When we create the hotbox Window, setting MAYA_HOTBOX_USE_SOFTWARE=1 will force Maya to use an internal software composition mode to achieve the effect of the transparent hotbox. We grab an image of the desktop and draw the hotbox on top of that image.
This software composition mode was the mode used if desktop composition was not available in Maya 2011 release 1, HF1 and HF2. The down side of this mode is that on Linux, we are drawing over the entire screen, all the applications on the screen get X damage messages and are forced to redraw themselves. This results in the flickering of all the windows on the desktop
Maya 2011 Hotfix 3 adds support for using the system RGB hardware overlay (previous versions of Maya used Colour Index CI overlays).
The RGB overlay allows for drawing into the overlay without affecting other applications: no flickering.
Why would you use it ?
The expected use of the variable was to enable a possible override for customers to not use the RGB overlay mode or desktop composition mode. If the RGB overlays were not available then Maya would default automatically to using the original software composition mode.
Will this variable have any effect in Maya 2010 ?
This has no effect on version of Maya before 2011
On a related note, the Maya 2011 use of Qt has had an effect on the recommended Linux configuration:
- Pre-Maya 2011, KDE was the recommended Window Manager, and we didn’t care/use compositing desktop.
- In Maya 2011, users may also consider the Gnome Window Manager, and can use the compositing desktop if available. Using Metacity as the compositing desktop manager has worked the best for our internal testing.
Lastly, here is the link to the certified hardware page for Maya 2011 on all platforms.
Thanks to Wayne Arnold for his contributions to this post.
You may also want to read this post : Qt patch for Autodesk Maya 2011 Linux tablet users