Tacoma,
H-D transmissions, at least the conventional 4 and 5 speeds, are notorious for the clunk. If you were to look at the engagement dogs of the lower 3 gears, you would notice a lot of slop in the fit. This is done to make sure the dogs engage rather than hit and bounce off (missed shift). There is a large speed difference between the engaging parts, and unlike an automotive manual trans there are no synchronizer rings to help match the gear speeds during a shift. In other words, that's pretty much the nature of the beast.
As for the effort, that can be caused by tightness in the trans itself (three of the gears slide on splines on the shaft to actually shift gears), and by clutch drag. If the clutch is still trying to drive the trans while you are trying to shift, the force against the shift dogs and the splines adds significantly to the effort required to slide the gear to the next position. That's why trying to shift without pulling the clutch or chopping the throttle is pretty much impossible. Also, when it does finally release and slide, it adds to the clunk issue.
My bike was extremely difficult to shift into 3rd gear when I bought it. After bleeding the hydraulic clutch, it got slightly better. Changing from SYN3 to Redline at 1000 miles made another improvement. After a few thousand miles the trans itself loosened up and it got a little better still. Still not as smooth or quiet as I'd prefer, but much better than when new.
Jerry