FOSS gamedev and creative worlds

I want to make a true radio mod, so i need a way to start and control an instance o mplayer (or vlc, or whatever) from a lua mod running on the server...

First of all, the program you want to control must support some protocol (sockets, pipes, command line, ...), then you need to use that protocol.
For example, most media players open the file/URL you pass it on the command line in an already running instance, and some also have a --stop command line switch of some sort.
EDIT: Again, this has portability problems, and will only work in singleplayer (how do you expect to control what program runs in the client machine in the case of multiplayer?).

Might be more practical to make a music player right in lua. That would actually be necessary if you want to play sounds from the server on clients.