Comments

1. #1 Mike Haubrich

   February 14, 2011

   Your keys are in the laundry basket because they fell out of the pants pocket of the jeans you were wearing yesterday.

   At least, that’s where I usually find mine.

2. #2 Greg Laden

   February 14, 2011

   Your keys are in the laundry basket

   They were next to the laundry basket.

3. #3 wockrassa

   February 14, 2011

   Professionally,* I work** in graphics. It’s part of my job to take and use a lot of digital photographs, assemble them into visual media such as brochures or ads, and keep track of the entire bloody mess. My photo directory is about 54 GB in size and contains nearly 19K items. My ads directory has about 2400 items at 9+ GB. Needless to say, organization is absolutely crucial.

   I keep folders organized by year (2006, 2007, etc.) with subfolders dated and named, the dates leading with the year to keep list views sensible. So I’ll have something like “2011-01-12 Fred and Wilma Flintstone, happy customers” in the photos, with the ads given somewhat similar names.

   With a reasonably fast machine, folders opened in icon view – particularly the large icon view available under OSX – provide all the visual indexing I need. Fortunately Mac can read the NEF format (Nikon’s variant of “raw”), so I don’t have to give descriptive filenames to images unless I want to be able to find them for regular, constant use.

   I really do not care for iPhoto. It’s a good example of a program trying to be helpful, but actually getting in the way.

   ==

   * To the extent you can call me a “professional”.
   ** To the extent you can call it “work”.

4. #4 Greg Laden

   February 14, 2011

   wockrassa: Great minds think alike.

   I use almost exactly the same system for my photos. Directories have names like “2003-July” and the photos therein are either part of one of three dsc series provided by the camera as default or something like “1985-Ituri-002″ for the second role from the Ituri Forest in 1985.

   BTW, my PDF files are named with the author(s)_[EtAl]_[year].pdf so I have files like:

   smith_jones_etal_2003.pdf

5. #5 Greg Laden

   February 14, 2011

   Correction: I don’t know why I said that wrong: the folder names do not use month words, but month numbers, so there are folders like this:

   2003-01 for jan 03
   2003-11 for nov 03

   Sometimes it goes down to day (2003-06-21) and using this system it’s easy to add words at the end that help identify the folder but don’t interfere with sorting like

   2009-11-Julias_Camera

6. #6 Warren

   February 14, 2011

   Greg @5:

   Yup on the dates-names conventions. It’s also worth mentioning the importance of file extensions. Mac and Win both “help” you by hiding those by default, which to me is totally ridiculous.

   The extension tells you at a glance what kind of file you’re looking at, and particularly with OSX.6 changing its binding behavior,* that can make a difference when you’re deciding to double-click to open the image in the default editor, or want to right-click it to send it to another one instead.

   Hence my recommendation to all computer users: Always have your file extensions showing.

   ==

   * Usedta could change the binding on file extensions so files that end in “.nef”, for instance, would open in Photoshop by default. Not any more. OSX.6 did away with that, which means that unless you want your raw files opening in Preview, you have to right click and choose “Open with…” instead.

   I don’t know why Apple changed this, and I wish to hell they hadn’t.

   Also, apparently Google has decided to log me in under a different, technically-related but not desired identity. Talk about things trying to help, and doing the opposite…

7. #7 Doug K

   February 14, 2011

   on Windows, Irfanview has a batch mode option which is very handy for resizing all the photos in a folder to a single web-friendly size.

   I toyed with the idea of using Subversion to have version-controlled image management, but it was too much like work. So, dated folders it is. I like the idea of links for topics, thank you.

8. #8 Greg Laden

   February 14, 2011

   File extensions have become de facto needed even in Linux for images, as some software now uses them, but in Linux file extensions are usually optional. Linux has other ways of figuring out what a file is. Of course, emacs breaks that by being programmable to use file extensions and thus, they have gotten used even though they are not necessary there either. (you can put a “hint” in the beginning of a text file to tell emacs if it is a org file, html file, etc.)

9. #9 Warren

   February 14, 2011

   About ten years ago I got an email attachment (on Mac) that I knew was a virus, and a Windows virus, without having to open it. Its name was something like “Kittens.jpg.vbs”.

   Now if I’d been on Windows, and naïve, and had my file extensions hidden, I might well have thought the file was called “Kittens.jpg”, and opened it – as many other people clearly had done already, thus allowing it to propagate via the magic of VisualBasic.

   Showing file extensions can actually help novice users avoid problems. As for emacs – heh. I thought you did it all with ed.

10. #10 Mr. Fright

    February 14, 2011

    Great idea with the links as extra tags to get to certain photos. I did the same thing, name my thousands of photos with numeric dates and description, putting them in directories by year or something. But then I’ve been left with searching several years directories looking for, say, roller derby photos. Now I’ll be sure to put them in by year and make links in additional directories.

    Thanks!

11. #11 Jason Thibeault

    February 14, 2011

    The option to hide extensions optionally is a bane on security. It was a decision I opposed in Windows when it was first introduced, and it’s a decision I still believe was wrong and wrong-headed from the outset.

    Greg: I assume imagemagick and some bash shenanigans? I’ve wanted to do something like what Picasa does, with folders and symlinks, the way you describe, but the trick would be in managing that implementation. I would hate to have to manually pick out each image that has my cat Molly in it, and symlink them to the Molly folder, but if I were to find a GUI interface to do it, the task would be a lot easier. Imagine — if the tagging was done via symlinks as well as a proprietary database for the program’s use, or even better, via a MySQL database, how much power we could harness over our image management thereafter. All programs could be coded to handle such a naming/linking scheme in a standardized way, and you wouldn’t have to reinvent the wheel every time some new version of ImageCheckerOuter 2.359 Beta 4 comes out.

12. #12 TheBrummell

    February 14, 2011

    I use Windows XP at work, and Windows 7 at home. I long ago figured out how to force XP to always show extensions, for exactly the reasons described above. I haven’t figured it out for 7, yet, it’s far too bubbly and cutesy for my tastes.

    I have been naming files and folders by date for years, using the format
    YYYY MM DD (and HH MM SS if required)
    which conveniently keeps everything in chronological order when sorted by ascending order (or the simple alphabetical order that Windows’ file management defaults to). When I’m writing a big document (papers, thesis, etc) or manipulating big data files, I save a new version of the file every day with the date at the end of the filename (MyThesis 110214.docx). This allows tracking progress by simply looking at the list of files; big gaps in dates are signs that I’ve been neglecting that project! For photographs I set the view in folders containing all or mostly photos to large icons, or extra large if I’m going through and renaming photos based on what they show (as you suggest). The built-in picture viewer in Windows 7 is a bit weird and frustrating, but gets the job done for a first-pass approach to a pile of photos (i.e. go through and delete the really bad ones).

    My main camera, a Pentax DSLR, by default saves files in a directory structure based on date not unlike mine; it’s a simple matter to convert those directories to my own format once I pull them off of the memory card. The file names themselves are generic (IMGP1234), so there’s some renaming to be done there.

    I process images in Adobe Lightroom (yes yes commercial software not open source evil capitalist bastards yadda yadda yadda), which allows for batch-renaming (and batch resizing, and other processes) when exporting pictures. Usually I leave it as “filename” and go through and rename only those pictures that I actually edited, rather than the 10x as many that I didn’t consider good enough for anyone else to see. I’ve got Irfanview as well, but I’m not happy with how it handles batch operations, in my experience it seems to be optimizing for speed or minimum file size when I’d like it preserve *some* picture quality and take a few minutes longer. Plus, if you don’t turn off Irfanview’s default setting of “all pictures belong to Irfanview!” Windows Explorer no longer shows the picture in icon view, instead showing that obnoxious Irfanview logo-icon.

    I’m still learning Lightroom, it’s mostly fairly intuitive but how it sets up the workflow by default takes some tweaking to fit it to me. I used to use GIMP, and I still use it for more detailed photo manipulations, but it’s horribly inefficient and slow for working with more than a handful of pictures at a time.

    As for PDFs, the majority of which are scientific papers, I am the only person I know to rename them (or save-as when pulling from a journal’s website) in the format “Author Year” to mimic the most common in-text citation style of the journals I most often read (i.e. “FirstAuthor & SecondAuthor Year”; “LeadAuthor et al Year”). When other people send me PDF papers, they often show up as “fulltext.pdf”, the default filename for several publishers, or as “Authorname-I-met-once-at-a-conference_JournalAbbreviation_keywords_year”.

    Having said all that, I tend to find organization in general tedious and boring, and I am easily distracted. This leads to misplacement of files all the time, and other inefficiencies. Any tips for improving one’s self-discipline?

13. #13 Susan

    February 14, 2011

    Another organization protip: fslint is an invaluable application for dealing with the instances in which files get duplicated (or downloaded twice…) or are misnamed.

14. #14 Maxwell Despard

    February 14, 2011

    I have similar complaints about iPhoto, but I have yet to get around to doing anything about it, partly ’cause of how in the dark I am about Mac options; in the joy of being an end-user for the first time ever, a luxury never afforded me by windows from 3.11 up to Vista, has allowed me to get pretty lazy.

    I should probably remedy that. If nothing else, it’ll help clear some space on my laptop. Thanks for the small kick in the ass.

15. #15 P_Smith

    February 15, 2011

    If you’re going to name files with dates, consistent naming helps as does using year-month-day:

    2002-10-11 for October 11, 2002

    Windoze (and I’m sure other OSes as well) sorts file names alphabetically when sorting by filename. The dating order I mention above will order them alphabetically as well both as folders and as files.

    Then it becomes a matter of how you want to organize. If you wanted to organize by where you took the photos and in order by animal:

    d:\zoo\bunnies_2010_05_04.jpg

    Or if you wanted to organize by animal and date:

    d:\bunnies\2010_05_04_zoo.jpg

    With a well thought out and consistent system, you could keep every photo in a single folder and still be able to find things quickly.

    The most important things, however are to be consistent and be disciplined. If you keep changing your mind on systems by which day of the week it is, or you get lazy and think, “I’ll take care of this later,” you’re going to end up with a mess no matter how good or intuitive your system design is.

    .

16. #16 GrayGaffer

    February 15, 2011

    I have yet to trust to aliases. In particular, DropBox does the wrong thing with them (they show up on the mirrored machine as files not links or folder contents). Images, I use similar conventions and have a date organised staging area where the camera images first land, but I use Graphics Converter Pro for managing and The Gimp for editing. GCP has excellent batch conversion tools, knows all the RAW formats (well, OS X knows them for it), has god proof sheet management, and has file navigation that implements all the Finder drag and drop etc paradigms. And it does not move anything into mysterious depyhs like iPhoto or iTunes do.

    The rest of my stuff is just chaos. I have a deep seated conviction that the standard tree structure is somehow missing the point, and scattering alises around makes it worse, not better.Tools like Yojimbo and Evernote organize by tags rather than hierarchies, but the tags are not structured at all, and one rapidly gets a list that is too long to use. Plus what I might think of as a good tag when entering a new thing into the system is as much a function of my state of mind at the time as it is a function of the subject matter, so my tag list gets polluted with multiple spellings for essentially the same concepts. If I remember to tag at the time in the first place. Then there are Bookmarks. Kind of a superposition of both methodologies. Yet less not more amenable to finding stuff in it again.

    I guess the point is that one way is not enough. But the systems we have don’t help.

17. #17 Lassi Hippeläinen

    February 15, 2011

    Using the folder structure works OK, as long as you don’t have any disk problems, and you have enough neurons in your head to remember the general rules. But some day you might get into trouble with one or the other. Or your dear ones, when you are gone.

    In the long run all metainfo about pictures should be in the pictures themselves. That’s why JPEG and TIFF files have the EXIF extension, where you can put comments about the image, location, copyrights, etc.

    I must admit that in this case I’m preaching what I don’t practice systematically, because I still haven’t found a tool that can tag en masse pictures, e.g. from my visit to the Dominican Republic, which lasted two weeks, and is nicely split to 2010 and 2011.

    Another point about future: I’d recommend using 1280 pixels in stead of 1200, because it is a standard video frame size. You don’t want to resize videos.

    And the “dumb-ass thing” for Windows users is called Irfanview, and it isn’t dumb-ass.

18. #18 Jean-Denis

    February 15, 2011

    So it seems I’ll be the dissenting voice here. To summarize what you are doing: you hijack one piece of information about files (the filename) to hold a bunch of unrelated metadata about the file: date, type, tags.

    I am sorry but I cannot agree with this. It amounts to accept only to use a screwdriver, even for nails.

    It’s already bad enough that we are forced to cope with file extensions in the first place. We inherited that stupid design decision from the CP/M era. They decided to put the information about the file type (metadata) at the same place as the file name.

    I still haven’t managed to make my father understand why he doesn’t get a word file when he renames his document.pdf into document.doc.

    So please don’t use the file system as a database, unless your file system can act as a database (e.g. BFS). And if you do, don’t use one column of the database (the file name) to store the data from many columns (date taken, place, people, topic, weather, time of day, speed, aperture, focal length, you name it).

    Better use a real database. Some of them even have specialized features for managing photographs. Use them, whether Aperture, Lightroom or whatever. Your original photos are still in the file system, they let you organize backups the way you want. They can execute queries such as “show me all pictures with Uncle John”, and much much more.

19. #19 jake

    February 15, 2011

    This is an example of my simple folder-naming technique (using Windows):

    C:/PHOTOS/2011_2m_15d_comment

    I don’t bother renaming photos unless I have many that are from a particular trip or something. Then I do a batch rename like this: 2011_2m_15d_MyName_TripLocation_0001.

    Ideally, I’d like to add notes to the EXIF data in my photos, but I have yet to find a program I like that will do that easily and quickly when you have tons of photos. A great example of how that should work is Picasa. In Picasa, when you are viewing an individual file, you always see a comment field; you can then add text to that field and you do not need to click anything to save it – it saves on the fly and you are free to scroll to the next photo and do the same. Unfortunately, only Picasa can read the comments so it is not a good long-term solution. A program that does the same thing, but saves to EXIF would be my dream program.

20. #20 Greg Laden

    February 15, 2011

    To summarize what you are doing: you hijack one piece of information about files (the filename) to hold a bunch of unrelated metadata about the file: date, type, tags.

    Hijack? What is a filename for but to serve as a label for a file. I’m not sure that sensible file naming is hijacking. Was the filename doing something else?

    I’m actually suggesting metadata like tags goes in links or other places. Also, with photos, metadata can go inside the photo file, and that is where the use of existing well developed tools comes in.

    I have mixed feelings about extensions, but actually, I find myself using them all the time to scan file lists as well as in searches, so maybe they are really a good thing.

    So please don’t use the file system as a database

    Well, there you’ve hit on a key point that I should have made more explicit in the post. The file system IS a database. I’m suggesting that we can use that fact to our advantage… use the database taht is already implemented, get a bit more out of it, but yes, I agree with you, don’t make it do too much.

    Better use a real database.

    YES!!!!!!! If I use F-spot, it actually uses an sql-lite database. What one should really do is make one’s own sql (or other) databse .. but again, use tools that are well estabalished, that don’t hide the data you are using (the files, etc.), that you can manipulate and use in conjunction with other tools, etc.

    I don’t think we disagree at all. You are merely suggesting using a standard database like mysql as one of the tools. Such a database can easily be integrated with the approach I lay out here.

21. #21 Greg Laden

    February 15, 2011

    I have yet to find a program I like that will do that easily and quickly when you have tons of photos.

    You might look at Imagemagick. I’m not sure, but it MUST do that. Also, a tiny bit of perl scripting would probably give you what you want.

22. #22 Eric Lund

    February 15, 2011

    As for PDFs, the majority of which are scientific papers, I am the only person I know to rename them (or save-as when pulling from a journal’s website) in the format “Author Year” to mimic the most common in-text citation style of the journals I most often read

    I have a somewhat different approach. I have a bunch of folders with the first authors’ names (with subfolders in cases where more than one author has the same surname), and I rename the papers to the format Year_Journal_Vol_Page[_Identifier]. So if I need to look up the A. B. Jones et al. (2008) paper in Journal of Recent Results, I go to my bibliography folder, then Jones, then (if needed) AB, then click on 2008_JRecentRes_999_999.

23. #23 Greg Laden

    February 15, 2011

    Eric,, why did you chose to do that instead of the opposite? Journal then within, author? To me, the journal info does not tell me what the paper is until I really, really know that paper. But I know that “Marks_J_1992.pdf” is going to be something complaining abut “Ruvolo_1988.pdf” because, well, that’s what Marks was doing in that year.

    It is hard to decide between subfolders for PDFs vs not. But still, either way, one has access to the data because they are filenames sorted in a database (the file system) that are, essentially, pointers to your file!

    (Also for PDF’s there is the possibility of indexiing/searching them, and something like Menedlay can be overlaid as a secondary organizing system without messing up the original files).

24. #24 Eric Lund

    February 15, 2011

    Greg, in my field any given paper has at least 3-4 journals in which it might appear, and some have much more. (I’m in physical sciences, where some journals specialize in short papers while longer papers on the same topics get steered to other journals, in addition to competition from journals with different societies/publishers.) Since the papers are already sorted by author, I know by the date roughly what the author was doing at the time, as you do in your example, and related papers by the same author which may have appeared in one of the other journals will be in the same folder. The sort-by-journal approach would work if your field is dominated by journals whose specialties don’t significantly overlap, but that simply isn’t the case for me.

25. #25 Warren

    February 15, 2011

    TheBrummell @12:

    “Having said all that, I tend to find organization in general tedious and boring, and I am easily distracted. This leads to misplacement of files all the time, and other inefficiencies. Any tips for improving one’s self-discipline?”

    Basically, if your organization system is consistent from the start, all you need to do (in theory) is add appropriate naming conventions to your workflow when you copy a given file to your drive.

    That is, if you’re pulling shots off an SD card, the first thing you’d do is create a folder to drop them into – and name the folder with the correct label for your particular file name convention. Doesn’t take that long.

    For bulk downloads – such as porn – it makes more sense to have a default download location, then using a script or dedicated program to bulk rename them.

    Also, make sure to clear off your desktop from time to time.

    GrayGaffer @16:

    “DropBox does the wrong thing with [aliases/links] (they show up on the mirrored machine as files not links or folder contents).”

    That’s not the wrong thing; it’s the only correct thing. DropBox stores whatever you’ve put into its folder on a remote server. That server then hosts copies out to whatever machines you have connected to your DB folder.

    It has no other way to present aliases to remote systems – they must be copied to the server so you can see them. Otherwise, the alias is meaningless. It’s a pointer to a file that doesn’t exist on the remote system.

    Jean-Denis @18:

    “So please don’t use the file system as a database…”

    But the file system is the database. That’s all a tree structure is. In using some sort of sensible convention, you’re actually working optimally with the design of that database, without needing to strap on additional software in the name of enforcing a different organization protocol than the one that already exists.

    This requires learning a little about the machine, but not very much, and the payoff is that you have a very good, very secure knowledge of where your files are located, and can always tell at a glance what kind of files they are. I’m not sure what part of that is objectionable, unless you’re arguing about aesthetics.

    Using a third-party database/file-manager program is akin to creating a robot to drive your car under remote control. It’s never going to be optimal, it’s always going to be clunky, and you can’t really trust it not to FUBAR the whole deal. Much more simple and sensible to drive the car yourself.

26. #26 Greg Laden

    February 15, 2011

    By the way, if you use the existing tools, you can have a file name like

    Journal_Year_Author_Topic.pdf

    and sort by journal, year, auhor, or topic, or search thusly

    sed/grep/awk/sort.

    In awk you can define _ and . as the field separators, for instance.

    A tool for quicker renaming of files would be nice. (Individual files).

    By the way, I’m not sure about current Windows, but in older verions, you can’t rename a file you are looking inside of. In Linux, of course, you can.

27. #27 Greg Laden

    February 15, 2011

    Gray: First, you are using aliases, but I’m using links, and they are slightly different. But, I’m not sure what Dropbox does with links.

    In windows one might consider using URL’s.

    Anyway, I agree taht one system is not enough, which is the underlying motive for my suggestions. Some app concieved of by some team somewhere is not going to do what I want (remember the example I gave? No one has suggested app that would do that yet). But, many apps take away your control of basic stuff like where the file is located, etc.

    There are a lot of issues being brought up here that could be solved with well made little tools.

28. #28 Lowell Gilbert

    February 15, 2011

    The filesystem absolute is a database. But it’s a database of limited capability (essentially, the filesystem tree is equivalent to a flat-file database). It’s always possible to represent more types of relationships by adding new tables for each, but it gets awkward to handle; there’s a reason relational databases were invented.

    Personally, my overwhelming preference for my photo albums is chronological, so a relational database would be overkill. But if you really want to organize large numbers of pictures by more than one characteristic, you should at least consider a more heavyweight approach than putting all the metadata in the directory and file names.

    Incidentally, some of the people in this thread who were looking for ways to mass-tag files might want to look at jhead, which is a command-line program that parses information out of Exif headers and can do various simple things with them, including renaming them into a data-based format as suggested in this post, and modifying the image comment — and a bunch of other simple manipulations.

29. #29 TheBrummell

    February 16, 2011

    Greg: By the way, I’m not sure about current Windows, but in older verions, you can’t rename a file you are looking inside of. In Linux, of course, you can.

    It’s still true for Windows 7 (you get an error message if you try), and it’s still annoying. Most annoying is when I get that error message because a file (that is not obviously part of the operating system or other ongoing, high-importance functions) is in use by a program I did not personally, specifically run. Whaddaya mean I can’t move that file because it’s in use? Who’s using it? Argh.

    Warren, I do most of the things you suggest. The main it’s-boring-I’m-distracted problem I have is with the large backlog of photos and other files that are in an unorganized jumble because they predate my current organizational system.

    Also, I keep running into a more insidious file organisation problem. When analyzing my data and writing manuscripts, versions of spreadsheets and documents get passed around among co-authors. Everybody has their own file naming conventions, but usually it’s easy enough to convert filenames as they come in as email attachments, sometimes by appending the sender’s name to the end of the filename just before the extension or the name and date. But the other problem I’ve encountered is a tendency for myself and my coauthors to each refer to the same project by a different nickname prior to somebody coming up with a stable title for the nascent manuscript; sometimes the title is highly unstable, and the manuscript goes through several name changes, often as revisions are suggested by peer-reviewers following submission to a journal. It all gets very confusing, very quickly, and I’m pretty sure there are a couple of “\MANUSCRIPT – name\” folders on my harddrive that contain different versions of the same project. This is less a computer-management problem than it is a personal-communication problem. *sigh*

    Finally, people keep mentioning scripts and small programs for bulk-renaming files – does anyone have any good suggestions for do so either in Windows 7 or Windows XP?

30. #30 Jean-Denis

    February 17, 2011

    Quote: “But the file system *is* a database”

    Yep, a database with one column. So you cram everything in its lone column, using string concatenation.

    That’s not my idea of doing the right thing, nor doing things right.

31. #31 Greg Laden

    February 18, 2011

    Jean-Denis: Not really. You don’t “cram everything” into one string. You use the strings that are there whether you want them to be there or not (there are no files without file names) sensibly. The files are “physically” stored somewhere, and where they are stored is important (you will find that out if you lose track of that) and existing album software stores data about them i a way that you don’t have easy access to … doesn’t even tell you where their metadata is, so how do you back it up?

    I’m not suggesting cramming everything into filenames. I am suggesting using the existing OS resources as your primary means of organizing things, using the filename and directories and the tools for manipulating them, rather than some software that hides all that for you.

    Then, to that, add tools to use the OS file system to further organize/backup/manipulate your stuff while still being able to directly “see” your files, rather than giving up control.

    I think maybe you only skimmed the post.

32. #32 Duane

    February 24, 2011

    I like this. Obvious in its simplicity to use the database that is already there and works without error.