In the November / December 2008 edition of DEVELOP3D, we reported on a hidden technology inside Catia V5R18 which can transform the 3D performance of Dassault Systèmes’ flagship CAD/CAM/CAE application. The technology is called VBOs (Vertex Buffer Objects) and works by moving raw geometry processing away from the CPU (Central Processing Unit) and onto the GPU (Graphics Processing Unit) (see box out for more info).
The article was written following a technical meeting with AMD’s professional graphics division, who played a key role in the implementation of VBOs inside Catia. However, soon after the article appeared in DEVELOP3D, AMD’s key graphics technology rival, Nvidia contacted us wanting to know more about the testing procedure used as it didn’t match up with its own internal benchmarks and how it believed its Quadro FX graphics cards stacked up against AMD’s ATI FirePro technology.
Real world testing inside Catia V5R18 using a Lenovo S10 workstation
After a number of discussions with Nvidia and AMD, followed by claims and counter claims, we decided the only fair way to get to the bottom of whose graphics cards work better with Catia was to get both parties in a room together – not so they could go ten rounds with each other (at least we hoped not), but by taking part in a dedicated testing day at one of the UK’s leading Catia resellers, Desktop Engineering.
The test plans
As there was a lot to pack into a single day all parties agreed on a test procedure beforehand. CATbench, the well known Catia benchmark, would be run and, time permitting, Catia’s built-in rotation performance test would also be used.
Both companies’ current entry-level, mid-range and high-end cards would be benchmarked – Nvidia’s Quadro FX 570 (256MB), FX 1700 (512MB) and FX 3700 (512MB), and AMD’s ATI FirePro V3750 (256MB), V5700 (512MB) and V7700 (512MB). Only Catia certified graphics drivers would be used – Nvidia’s 169.55 and AMD’s 8.543.
We had planned to run the benchmarks on two identical Lenovo S10 workstations – one for Nvidia and one for AMD – but these were ‘lost’ in transit by a courier company two days before the event. Lenovo did well to find two replacement machines, but unfortunately the specifications were not identical. As a result, we were left to use a single Lenovo S10 workstation for all testing which involved a lot of swapping in and out of graphics cards and drivers. The specification of the machine was an Intel Core 2 Extreme X9650 (3.0GHz) (Quad Core), 3GB RAM, and Windows XP SP3, Catia V5R18 SP4.
Nvidia Quadro
FX 1700 graphics card
CATbench
The CATbench benchmark uses a number of different models ranging in size from an engine block to an entire nuclear submarine assembly. For each model it measures the time taken to perform a set number of pans, zooms and rotations. CATbench runs in three different graphics modes – edges (wireframe), shaded, and shaded plus edges. However, as most Catia users work in shaded plus edges mode, it was agreed that we would focus on these results.
We tested twice for each card and the results were clearly in favour of AMD with the Peugeot 807 and Holland nuclear submarine, two of the largest models in the test, completed three to four times faster than Nvidia’s cards. The difference in the other models wasn’t as big.
As an aside, while for Nvidia’s cards the higher-end boards were faster than the lower end, there was less difference between AMD’s cards, with high-end the V7700 actually going slower than the entry-level V3750 and mid-range V5700 under some models. Unfortunately, this anomaly was not noticed on the day so this couldn’t be investigated.
Rotation performance test
This simple test is built into Catia and enables users to take any model and perform a set number of rotations in any axis. We tested with the Peugeot 807 model from CATbench and, in a quest to find the largest and most complex dataset we could get our hands on to increase the load on the system, a model which combined the Holland nuclear submarine with the two car models in a single assembly.
User experience should always be the acid test when assessing new hardware
Time constraints meant we could only run this test with a Quadro FX 3700 and ATI FirePro V7700 but Nvidia came out a clear winner with two to three times the performance of the ATI card. Despite using the same datasets from the CATbench test the rotation benchmark showed the exact opposite to the CATbench test in which AMD won, which was more than a little puzzling.
Real world testing
With seemingly contradictory results, we decided to resort to real world tests and asked experienced Catia application engineers from Desktop Engineering to get a feel for both the Quadro FX 3700 and ATI FirePro V7700 by performing a series of random pans, zooms and rotates on the large combined assembly. While there is no exact science in this methodology, both application engineers voted in favour of Nvidia. “Both cards performed very well considering the large assembly files we were using, but the Nvidia Quadro FX 3700 seemed more responsive and had no discernable lag when manipulating and browsing the model,” said Charlie Maguire, Technical Consultant, Desktop Engineering.
Conclusions
When going into this head to head we had hoped to get a definitive answer to the question of which graphics cards are best for Catia. The problem is benchmarking is never an exact science. Every user has a different 3D experience, depending on the size, nature and complexity of their datasets. Trying to define that in any benchmark is not a simple task and one not only has to decide which models to use, but also how to use them, ideally mimicking real world manipulation patterns as best as possible. And as if things weren’t complex enough, it is common practice for graphics card developers to optimise their drivers so they perform better in specific benchmarks, while not necessarily improving real world performance.
AMD ATI FirePro V5700 graphics card
Some of the models used in CATbench are sizeable, but are by no means anywhere near as big or complex as those used by many of Dassault’s leading customers in the automotive and aerospace sectors. Unfortunately we didn’t have such models at our disposal. It would also have been interesting to see what would happen when the graphics card frame buffer memory became overloaded, a common problem for those working with giant datasets, but one that can be overcome by investing in a higher-end card with more memory.
From our observations, CATbench does a better job of mimicking real world use, simply because it rotates, pans and zooms concurrently, whereas the rotation test simply rotates a model a set number of times in a specific plane. However, when two benchmarks contradict each other so dramatically, particularly when using the same data, it would be naïve for us to side with one benchmark over the other unless further investigations were carried out.
With all this benchmarking technology at our disposal, it seems crazy that the net result of a long day of testing boiled down to an empirical test of which graphics card felt more responsive when zooming, panning and rotating around a single model. However, one mustn’t forget that user experience is the most important consideration and should always be the acid test when assessing new hardware. Luckily as most Catia users buy workstations and graphics cards in the tens, hundreds or even thousands, it shouldn’t be too hard getting hold of AMD and Nvidia evaluation units to try out with their own datasets. Most of the time benchmarks can help, but never take the numbers as gospel.
www.3ds.com / www.dte.co.uk / www.lenovo.com
www.ati.amd.com / www.nvidia.com
Vertex buffer objects explained
With VBOs, 3D geometry is loaded up, stored and processed on the graphics card. When changes to the geometry are made, data doesn’t have to be moved back and forth over the PCI Express bus, as has traditionally been the case. This helps minimise the instances when the GPU has to wait for the CPU (as it is often tied up with other tasks) and as a result boosts real-time 3D performance.
VBOs can also free up some of the workstation’s core system resources as data that previously resided in the CPU’s memory, now resides on the graphics card’s frame buffer memory. When working with large assemblies this not only gives you a higher ceiling before you run out of system memory and have to page to hard disk, but when using 32-bit Operating Systems, such as Windows XP, users may find they are now able to load up models that they couldn’t load up before simply because they couldn’t address any more memory.
VBOs is an extension inside OpenGL (the 3D graphics library used to drive Catia) and is supported by professional graphics cards that are OpenGL 2.1 (and above) compliant. VBOs also need to be enabled inside Catia (V5R18 or later), because Dassault Systèmes has chosen to turn them off by default. This is done by making the following change in the Environment Variables settings: CAT_VBO_ALLOWED.
The vendors respond
AMD
Performance testing is never easy, especially when you have limited time. Operating Systems, when they first start, have extra processes running which look at file structure, anti-virus updates etc. These all take resources and explain why workstations are slower if you haven’t used them for a while. Of course, once completed the performance returns. This could account for some anomalies on the FirePro V7700 under CATbench as it was the very first card tested on the day.
CATBench is a good test as it not only performs rotations, but zooms and pans as well, which gives users a real example of how the software is used and it is consistent.
First use rotation, as tested by the rotation performance test, is speed that most users experience. By this we mean grip, rotate and then edit. If you grip a model and rotate it backward and forward the card can cache this info supposedly improving frame rate using cache algorithms. That gives transient frame rate performance but it might not necessarily be true user experience. This is different to VBOs where operations can be performed on the data without a full re-fetch of data.
Unfortunately we had to leave before the interactive tests were carried out so we couldn’t comment on their validity. However, we hope that they didn’t fall into the cache algorithm catch area.
Overall, we are pleased that the initial CATbench testing of the ATI FirePro cards performance has been proved again.
NVIDIA
We are glad to see the great results with a venerable driver. It is a testament to Nvidia’s quality and performance that a driver from April 2008 showed excellent performance in the real-world test and in the test by end users. To be recommended by daily Catia users is testament to the hard work the Nvidia Quadro team do to ensure the best efficiency and work flow improvement for customers.
There are many exciting new technologies on the horizon and we are looking forward to another challenge with new platform and graphic introductions in 2009.
