Maya 2011 on Linux and the Hotbox

If you have installed Maya 2011 Hotfix 3 and see errors in the terminal or a black Hotbox in the viewport, we advise you to disable the CIOverlays option in your xorg.conf file, located in /etc/X11/xorg.com.

With the move to Qt in 2011 we no longer have a need for colour indexing.  Instead, as of Hotfix 3 we have introduced RGB overlays and this option should be used instead.

• Open your xorg.conf file
• Remove following line:
  Option      "CIOverlay" "ON"
• Add following line:
  Option      "Overlay" "ON"

The caveat here is that you cannot set both CIOverlay and Overlay at the same time, and users of Maya 2010 will be aware of the instabilities that can arise when working without CIOverlay.

So what do you do if you use both Maya 2010 and Maya 2011 on a daily basis ?

Enable CIOverlay - for Maya 2010 - set the following Maya environment variable, in the Maya.env file located in: ~/maya/2011:

MAYA_HOTBOX_USE_SOFTWARE = 1

This variable forces Maya Hotfix 3 to fallback on an internal software composition mode to achieve the effect of the transparent hotbox , of the kind used by default in Maya 2011 release 1, and Hotfixes 1 and 2.

If you set this variable, it doesn’t change the setting set in the xorg.conf file.   Maya will no longer check for RGB overlay and use its internal software composition mode

This variable has no effect on versions of Maya prior to 2011.

What does it do exactly ?

When we create the hotbox Window,  setting MAYA_HOTBOX_USE_SOFTWARE=1 will force Maya to use an internal software composition mode to achieve the effect of the transparent hotbox.   We grab an image of the desktop and draw the hotbox on top of that image.

This software composition mode was the mode used if desktop composition was not available in Maya 2011 release 1, HF1 and HF2. The down side of this mode is that on Linux, we are drawing over the entire screen, all the applications on the screen get X damage messages and are forced to redraw themselves. This results in the flickering of all the windows on the desktop

Maya 2011 Hotfix 3 adds support for using the system RGB hardware overlay (previous versions of Maya used Colour Index CI overlays).

The RGB overlay allows for drawing into the overlay without affecting other applications: no flickering.

Why would you use it ?

The expected use of the variable was to enable a possible override for customers to not use the RGB overlay mode or desktop composition mode.  If the RGB overlays were not available then Maya would default automatically to using the original software composition mode.

Will this variable have any effect in Maya 2010 ?

This has no effect on version of Maya before 2011

On a related note, the Maya 2011 use of Qt has had an effect on the recommended Linux configuration: 

• Pre-Maya 2011, KDE was the recommended Window Manager, and we didn’t care/use compositing desktop.
• In Maya 2011, users may also consider the Gnome Window Manager, and can use the compositing desktop if available. Using Metacity as the compositing desktop manager has worked the best for our internal testing.

Lastly, here is the link to the certified hardware page for Maya 2011 on all platforms.

Thanks to Wayne Arnold for his contributions to this post.

You may also want to read this post : Qt patch for Autodesk Maya 2011 Linux tablet users

Owen

Comments

[Chayan Bhadury]

when i try to install it with msi file it just quits and says 'the wizard was interrupted before autodesk maya 2011 32-bit could be completely installed. your system has not been modified. to complete installation at another time, please run setup again.'
how can i install it?

[Thomas]

We are running Maya2011 Subscription Advantage Pack in TwinView with Option "Overlay" "ON" in the Screen section of xorg.conf. We still get the old screen shot/flicker problem. Any help to get rid of this would be greatly appreciated.

Thomas

[Kamil]

We have huge problem, with using Maya 2011 on Linux. We cannot change our current system setup (CentOS 5.5) so we need to have fully working Maya without compiz enabled. The main problem is that Maya UI is flickering while using marking menus or hotbox. Do you have any ideas how to resolve this problem?

Thanks in advance!

[Owen]

Try setting the MAYA_SET_PARENT_HINT=1 environment variable in Maya.env. This forces 'child' windows to be parented to the main Maya window.

Be warned tho' that KDE is not the Autodesk qualified windows manager, and that 'child' windows on KDE may be missing the minimise/maximise buttons.

Owen

[Nick]

On a hotbox related note, would you guys have any idea why on KDE when you open a window, say outliner, from the hotbox it always opens under the main Maya window.
Whereas when you open outliner from the menu it opens above the main Maya window.

Ideally we always want it to open above the main window as it's a pain to keep minimising the main window to find it. No amount of KDE 'Focus Stealing Prevention' workaround tinkering seemd to be able to fix it.

(Using CentOS 5.5 - which comes with KDE 3.5)

Nick

[Recepti]

Marketing is used to identify the customer, to keep the customer, and to satisfy the customer.