I've sailed to the Isle about half a dozen times. A couple of times I set off but turned back when I didn't like the look of the weather. The important thing is the current and wind direction, and the absolute thing is the tidal gate through Hurst Narrows. Don't even pretend that you can get through in anything below a power boat against that current, and it needs to be a powerful one. It isn't that you can't make progress, you can, very, very slowly. But if you make the slightest mistake or misjudgement, you are back where you were an hour before in seconds. I have forced my way through once, but realised afterwards that it would have taken me no longer if I had just anchored and waited for the tide to change.
Plan the trip using a tidal atlas. Figure out when the favourable tide ends at Hurst, and plan to be there an hour before if you can (I did it by 10 minutes once, which was a bit squeaky as it was blowing F6 straight into the Solent, so there was no way I could go back). I work on the basis that if there is a reasonable breeze, I can average 3kts, which makes it about four hours or so, Poole entrance to Hurst. Then add on the benefit of a favourable current, which may save you up to an hour of that. I always go via the Looe channel out of Poole and the North Channel into Hurst.
The final thing you need to plan for is that you are going to have to go back the next day. I usually stay at least one night in the Solent. The next day I have to get back whatever the weather. You still have to allow for the Hurst current, but after that there is no problem. It is possible to get into Poole against the ebb if you have to, but you will need more than 2HP. It helps if you know where the back eddies are, Follow a local who seems to be going in a bizarre direction. I have only sailed back once, on a single, three hour reach which left me aching down one side. All other times I have had to motor against a head wind, so make sure you have enough fuel and power.
Its a lovely trip. Yarmouth is an excellent destination, one f my favourites. The harbour is an excellent berth, but very expensive. You get what you pay for. Lymington is nice too, but I like to be on the island if I can.