I'm wrecking my car this weekend as I've downgraded to a 3.0L V6, just wondering if there is a difference in the inlet manifolds if the 3.5 would provide better performance?

Upper intake manifold is same, lower is different and unique to each engine.

would it be worth keeping and boring it out?

Std manifold is fine for any n/ a application