It would appear that the raycast function for Points/Line evaluates all vertices in the geometry, even when only a subset is visible (i.e. backed by BufferGeometry with a drawRange set) - resulting in erroneous hits against non-visible geometry.
Solution would be to clamp the raycast position iterators to the drawRange if there's one specified.
Would you like me to do a PR?
Three.js version
Browser
OS
Hardware Requirements (graphics card, VR Device, ...)