I tried learning Japanese and struggled for exactly the reasons you describe so clearly. I felt I might manage to learn if I were living in Japan and picking up the contextual clues but I could not learn Japanese effectively from a textbook. Visiting Japan is high on my wish list, maybe one day.