Over the years (say a decade), my music collection has grown immensely. At this point, it is about six thousand tracks and ~40GB in size. Most of it was accumulated through the purchase of audio CDs over the years, ripped as MP3s and then tagged and organized.
In this period, due to the sheer size of the collection, I have had to evolve my own method of managing this collection and choosing the tracks to play. With the arrival of iPod, I've again had to adapt the approach a little, but I should say the iPod has made things much more easy.
Though the collection is to listen, the order has to be reversed. The first priority is to organize the music and then to listen. The better a collection is stored, easier it is to find and hear music. I use multiple tools for this. I originally used MediaCenter, but it became quite expensive. It is excellent and AFAIK, it came as close at it could to be a one stop shop for all organizing and playing needs. Its absence is not a show stopper - I was able to accomplish the same effect with multiple but free tools.
Getting the digital version of music
Now a days, this is pretty no brainer stuff. A lot of MP3s are available on the Internet. I personally get my own audio cds and rip them for my collection for quality reasons. I use only MP3 (stay clear of DRM crap!) and encode them at 192 kbps fixed bit rate if it is music. For audio talks, 64 kbps is good enough. A lot of built in tools in the OS itself. For example, in Windows, Media Player does a good enough job of encoding music. iTunes, which I now use also is good. This is free (and excellently usable even if you don't have an iPod). Media Player cleverly defaults to WMA which I don't recommend. MP3 is here to stay - so stick to it!
Organizing music
There are two components to organizing music.
Tagging - is vitally underrated and is the most important step. All audio players and most audio management software relies on MP3 tags - not file names to recognize tracks. I've seen a lot of music files which use their own way of storing information in tags which is not at all good. There are four important tags that need to be filled to keep the music collection sane.
Artist, Album, Track, Genre. Others are optional as far as I've noticed. Track No is normally filled by the encoding software if from an audio CD. Keep a consistent naming convention for naming the artist, albums and track names. Especially when it comes to regional names stored in English, inconsistency can be a pain. Genre is useful when you want to do automated song selection from thousands of tracks - as in my case.

In the above example, the three fields as shown by iTunes are the track name, artist and album name.
The second step is to store them in a consistent way in folders. Normally, you would want to accomplish the following:
- Put all the tracks categorized properly under folders and sub folders
- Name the physical file consistent with the tag names.
Most software follows the Root Folder -> Artist Name -> Album Name -> Tracks folder structure. Some allow you to customize them. I use this option - as you can see below.

Many music softwares allow you to rename files in a way that is consistent with the tag names. I use the following format.
[Track No]...[Track Name]...[Album Name]...[Artist Name].mp3
This good if the music library in the software is lost - the library can be rebuilt from the files. The MP3 themselves have the tags as well - so it is a two way backup.
I use a software called MusicBrainz Picard to accomplish both steps. It is good because it allows to do both steps in one go. For more popular tracks, it does a database lookup as well and fills in the tags (I rarely get the luxury for Tamil tracks!). The really cool part about this is that it allows to customize the file name to no end using formulae. If you're the finicky fussy kind, this is your dream software!

Above is the format I use to rename my files.
%artist%/%album%/$num(%tracknumber%,2)...%title%...[%album%]...[%artist%] is the formula. It roughly translates to the naming convention I listed above. You can also see that the Folder structure is also specified here.
Upto now, whatever we've done accomplishes the following:
- Getting your digital media
- Tagging them
- Putting them properly in folders and files
- Naming the files neatly
If you already have a collection that is not in any proper order, I recommend taking the 30-40 hours it requires to organize them now! Once you have the root folder under which your entire collection sits, you can load them into iTunes in one swoop and voila - automated music library.
Volume Levelling
I do this additional step to ensure all tracks have uniform volume. I use a software called MP3Gain for this. It adjusts the gain of the MP3 tracks after analysis but doesn't change the MP3 file contents itself, which is good!

Above, you can see a sample interface. My default is set to 92 dB.
How do I keep track of which MP3 files have completed this normalization exercise? I use the Comment field of the MP3 file and mark it "Volume Levelling Done". A little lame - because the comment field is a great place to store lyrics. I don't use it that way so this works for me. Another good thing is that this comment travels with the MP3. You can also use a custom field in your music library software to store this, but that gets lost if the library is damaged.

You can see this comment above.
Once all your tracks are volume levelled and you have this comment in the comment field, you can write a smart play list on iTunes to filter out any track that is pending volume levelling.

Choosing tracks and listening to music
What you will need to do is to rate your songs as you listen to them. Over a period of time, you will have a huge collection of songs you've listened to and rated. If you have an iPod, this process is easy. iPod and iTunes are like peas and carrots. Every time you sync, iTunes knows what you've listened so you have to just go and rate them from your "Recently Played" list. Otherwise, do this manually. This is a self updating list. The more you listen to and rate to songs, the better the future picks will be.
After this, picking new songs from the library is a matter of writing a smart play list (very powerful and customizable iTunes feature!). An example below from my library:

The above smart play list gives me 25 random songs that are not Spiritual in nature but are outstanding (as per my rating). If you've rated 5000 songs, the pick will have a lot of variety and quality.
Backing up
For me, my collection is a decade long effort. So having it all in one place, while ultra convenient is also dangerous. For me, my iPod itself is a backup. In the event of a disaster, I can reverse extract from my iPod. Additionally, I also backup the HDD once a month to another.
Finally...
Anything I've missed? Anything that you do is simpler or better?
Technorati : Music iTunes MP3Gain Picard iPod
Del.icio.us : Music iTunes MP3Gain Picard iPod
Zooomr : Music iTunes MP3Gain Picard iPod
Flickr : Music iTunes MP3Gain Picard iPod