I have a 2005 SX TX, which has a few TS options. 85,000km on the clock and giving me no problems at all after having had a few of the known issues fixed over the last few years, almost all at Ford's expense.
They are a very good vehicle and the problems with them are known and well documented in the Territory section of Australian Ford Forums:
Australian Ford Forums - Territory
The rust is one of these known issues, with a fix - as all Territorys have a 5 year corrosion warranty you can phone Ford and get a job number, get two quotes from paintshops, get the go-ahead from Ford for one of them and have it fixed. Ours cost $3k or so, and the paint was taken back to metal around the tailgate, fuel tank, rear bar crumple zone etc. and refinished. Three years later and it's still looking great. The problem stems from the introduction of water-based paint with BA and Territory, took them quite some time to get it all sorted.
They also do the afore-mentioned front ball joints (Pedders makes a stout replacement but Ford will usually replace them free with their redesigned job), snap the front electric window cable-to-window clips (a dealer bulletin says replace these during service if they haven't already been done). There's also dealer bulletins to re-do fuel line routing and replace a brake circuitry component. The bonnet cables snap but the $30-40 redesigned version is easy to fit and works much better than the original.
The latches on the rear rubbish bin door and the top-of-dash storage bin break easily.
You can fit an MP3 input jack to the stereo fairly easily, the plug and play wiring harness is available on ebay (buy the amplified version, not the non-powered version).
If you need roof racks, make sure that any factory racks come with the keys to unlock them, otherwise you can't even move them. Note that the racks look like Thule and Rola but are a different profile and no Rola etc. accessories will fit to them.
Any issues with them, you'll usually find an answer with a search of that Ford forum.