"Experimental" refers to a procedure, not a genre. In other words, when an artist takes an experimental approach, he is changing parameters of the artwork and noticing the results. The meaning of "experimental" is the same in all the arts and sciences. An artist who uses an "experimental approach" does not know what the final product will be.

How does this apply to silent film scores? Predictably a composer would fill the score with all the sounds, tempos and emotional progressions which the audience expects. What's the point other than showing off his craft and getting paid? An experimental approach would not deprive the audience of unexpected and delicious juxtapositions of sight and sound. Instead of cliched horror music behind Nosferatu, I put a dissonant burlesque which shows the vampyr's power and the desire which feeds his ancient soul.