Really tight...175 ft/lbs. It isn't a castle nut. It is a bit odd-shaped though for a reason and that is to prevent backing off when properly torqued. They can lose their effectiveness in that regard after numerous torque cycles. Also replace the gasket between the t-case and the adapter housing if it is compromised in any way. Resist the urge (if you have one) to use silicone based rtv in place of or in addition to these gaskets. Rtv has its place and purpose, though it escapes me at the moment. The gasket between the t-case and housing serves as a very thin spacer shim, which cannot be duplicated with any smeared on goop.
While you have the trans out and the pan off, it would really behoove you to adjust the bands. The low/reverse, or rear band adjustment screw can only be accessed while the pan is removed. You need an inch/pound torque wrench, or a ft/lbs wrench that can be adjusted down to 6 ft/lbs, which is 72 in/lbs., the initial torque setting for both band screws. Then each is backed off the appropriate number of turns from there and locked down, which for an unmodified 727 behind an sv8 is 2.25 turns out for both. An 8-point 5/16" socket is also very helpful for accomplishing this task, as the adjustment screws have square heads.