Patrol, Prado, Pajero so get in them all and drive them.
I am biased as I own a Diesel D4D Prado.

But I was looking at GU patrols at the time. I was looking for a long distance tourer and like you do 80/20 on road/off road. These things did it for me in my price range:
You will get most places you want to off road in all of those in stock form.
The GU has a 4 speed Auto, the Prados I was looking at were all 5 speed.
The 90 and 120 series Prado have 180 litre fuel tanks - The diesel guys can get close to 1500km on a tank when not towing.
The Patrols are tough as nails, but driving the Patrol is like driving a truck. The live axles are great for off road, but driving long distance in a Patrol vs a Prado is chalk and cheese. Prado's win for comfort in my eyes.
We've got 3 kids and the rear middle seat in the Patrol felt uncomfortable.
Pajero was discounted early to me because we wanted an 8 seater (And I've never known a Mitsi that didn't leak oil. Irrational I know but it was enough) BUT they are underrated off road vehicles. Ground clearance can be a bit low though and I wouldn't be mounting a second battery in the engine bay, but they're not bad trucks.
So that leaves Petrol vs Diesel Prado - The V6 Prado is a hoot to drive and the engine is pretty near indestructible but fuel consumption if you tow and do a lot of low range is up around the mid 20's. But if you own a diesel factor in changing the injectors when they wear out, higher servicing costs, and higher purchase price and you probably come out evens.
Pre-D4D diesel is a great unit, some feel it is underpowered but plenty use it to tow. The early D4D diesel's had a problem with their injector seals/seats causing blow by - can make the engine go boom! This should now be fixed as there is a recall out to get them done but if you hear a knocking from the engine when inspecting, walk away (I've heard the same noise from Diesel Pajero's too btw). 2008 onwards should be ok as they changed the design but might be out your price range. You'll need to swap out injectors on any diesel truck eventually as they're a consumable but the Injector swap on a D4D Toyota is around $1500-$2000. Much less for a non D4D (1KZ-TE).
The early GU 1-3 diesels also had an issue - google Nissan Grenade. Most people I spoke to recommend buying an '06 GU IV and above to avoid it.
Downside of driving a Prado - paying the Toyota Tax for parts and servicing.