The reasons given for graphics drivers not being open are quite reasonable and I will exclude that quality from that discussion.

Many other devices are so basic that the driver is little more than a variation of the example code in the SDK (I'm considering Windows drivers here). Yet the manufacturers guard the source code for their version of the driver as they would if there was something really valuable in there.

In some software the driver is just a simple layer between the application and the hardware, most of the implementation is in the application and the driver could be opened and still maintain compatibility with the application.

At the same time users that are interested could be utilities or applications that improve what the user could do with the hardware.

As background, a typical graphics stack looks like this, from bottom to top. All components are designed, implemented, and maintained by the GPU manufacturer:

1. Graphics card is presented to the kernel through some mechanism. On Android/ARM, this was as a device memory mapped into a hardware-specified set of addresses. This layer provides the computational component.

2. A kernel driver whose function is to accept GPU code and funnel it down to the device through the appropriate mechanism. This layer provides the input/output mechanisms necessary to communicate with the device, and contains knowledge about the communication protocol the card follows.

3. An OpenGL driver. This is basically a compiler than accepts sequences of OpenGL calls, and translates them down to the hardware-level implementation. Most knowledge of the computational details of the GPU's architecture lives here.

Of these layers, only the kernel driver, number 2, was open source when I worked with it. This is because it links into the Linux kernel, and so needs to be GPL (Not certain about this, please feel free to correct me.) In any case, this component does not contain much or any "secret sauce," so it's fairly safe to share.

The rest of the components, however, are closed source. The reasons for this are:

1. The GPU hardware's architectural design is a trade secret. Even the instruction set architecture is closed source. The reason for this is that much of the competitive advantage a GPU manufacturer might enjoy stems from the speed (and power efficiency) of its GPU architecture. Publishing the architecture would basically provide the GPU maker's competitors access to valuable intellectual property: a lot of research goes into designing a good ISA, and you could probably tell a lot about how the device functions if you had access to the ISA spec.

2. Following this rationale, keeping the ISA secret allows the manufacturer a great deal of flexibility. They have no reason to worry about backwards compatibility because they own the product, so they can do whatever they want with it: add instructions, remove instructions, change calling conventions, whatever.

For comparison, one of Intel's (many) strengths is its robust backward compatibility for its ISA. The basic outline of Intel's x86 ISA have been unchanged for decades, and when they do roll out changes, they're additive rather than subtractive, and they take years to fully roll out. Secrecy presumably allows GPU makers to turn things around as quickly as they feel necessary by freeing them of this backward compatibility constraint.

3. I'm less confident about this point, but I presume there are some tricks they do in implementing the OpenGL library driver. Wherever there are multiple competing implementations of something, you're guaranteed to have tricks the various implementors are proud of.

tl;dr: Keeping drivers secret allow manufacturers to obfuscate their architectural implementation, protecting their intellectual property and affording them a measure of flexibility when they want to make changes to it.

IIRC, certain "Intel Integrated Graphics" variants suffered from this problem, and to this day lack good 3D F/OSS drivers.