If it's locking up only in 3D applications, it's more than likely hardware. Your video card is in the toaster, and the heat is rising...
Is the card overclocked? Have you unlocked any pipes or shaders? If you o/c'ed or unlocked the card, first return it to factory settings and see if this helps. If not, move on...
Troubleshooting hardware failure requires you to run the computer with the case open, so do that. Check the GPU fan on the video card. Is it running? If not, then it needs to be replaced because the card is overheating. Also make sure there's not a ton of dust in the fan that could obstruct airflow.
If the fan is okay, then look at the RAM chips on the video card. Did they have heatsinks on them when you got the card? Has one fallen off by any chance?
Okay, if the fan is in place and running properly, and the VRAM seems fine, then you have a hardware defect. The GPU itself could be going bad, or one of the VRAM chips. There's no real way for you to test where the problem is, unfortunately. If your card is under warranty, get it replaced. If not, then well, it still will need to be replaced, but out of your wallet instead of someone elses.

As a rule of thumb I never buy twice from a manufacturer who sold me defective goods, but that's just me...