ok what mods are you trying to install?
the grand prawn island mod isnt too hard to install... you just have to add some stuff in maps, data, then replace / add some files in the img files..
il give you a quick example how to put a car mod with imgtool v2
say u download a car mod with a ford mustang. it replaces the car "admiral". so what you do is you open up imgtool, click file > open, then find your gta3.img (for vice city, it will be in c:\program files\rockstar games\grand theft auto vice city\models\gta3.img unless you changed it)
ok now, press f2 (find), and then type admiral. it will find admiral.txd, then right click, and rename to admiral1. then press f3 (find next), now it finds admiral.dff . rename this to admiral1.dff.
now u unzip / extract the files from the mod. it will probably have a readme.txt, admiral.txd and admiral.dff. so then, in imgtool click commands > add, then go find admiral.dff and admiral.txd where you unzipped your car mod. now your done.
you would also have to replace lines in handling.cfg (which will control handling characteristics like speed, turning n stuff) but you dont have to do this. then the car will look like a ford mustang but still drive like it did originally.
if you still dont get it, read this topic, then post there.