I am certain an 8-HP auxiliary engine will be completely sufficient for a 1998 17 OUTRAGE II boat. An 8-HP engine was used as an auxiliary engine on an OUTRAGE 22 and was completely sufficient. See the REFERENCE section article on that installation:
Mounting a Yamaha T8 Four-Stroke Auxiliary
On a 1992 OUTRAGE 22 with Evinrude 225-HP Main Enginehttp://continuouswave.com/whaler/refere ... icker.htmlI expect that even a 6-HP auxiliary would be sufficient for a 17-foot boat.
Without a transom to mount the auxiliary engine, you may have to fabricate or buy an engine mounting bracket. You should research the location of embedded wood reinforcement in the transom to be certain the auxiliary engine mount will be fastened to a reinforced area.
For examples of auxiliary engine installation that did not use a transom mount, see two articles in REFERENCE:
Auxiliary Engine Bracket for 15-foot Boston Whaler Boatshttp://continuouswave.com/whaler/refere ... Motor.htmland
170 MONTAUK: Auxiliary Motor Installation Directly To Transomhttp://continuouswave.com/whaler/refere ... liary.htmlBefore undertaking this project, I recommend you test the effect of the addition of a mounting bracket and another outboard engine hanging on the stern of the boat on the boat's static trim. The 17 OUTRAGE II is somewhat known for having limited freeboard at the stern. Adding another engine and a bracket could have a detrimental effect on the boat trim. You should simulate the added weight by adding at least the same weight inside the boat at the transom, and perhaps even more than the anticipated engine and bracket weight because the actual engine and bracket will be located aft of the transom. You may find that the effect of the added weight is not particularly favorable.