I absolutely keep track of time. The PC's just finished a story arc where they traveled after a group of slavers to rescue some of their countrymen. They thought they would be back in a week, and told that to the ship captain. It took them 3 weeks to catch up, and by that point the slaves had been sold to a noble family. They then went on a two-week excursion to earn the slaves back. They just got back to the coastal town, seven weeks after they thought they'd be back.
Obviously, I also track distances between places, and have that prepared beforehand; winging it has gotten me into spacial problems before.
Right now my campaign is just heading into an adaptation of Red Hand of Doom, but the big "Battle of Brindol" will occur at a specific spot, during the Winter Solstice. There will also be specific "planar conjuctions" that occur in timing with the lunar phase that will impact this adventure. Add to that the fact that my campaign is set in the equivalent of scandinavia, and the shortening days will begin having quite an impact (by the time they head to "Rhest" it will already be within the polar night). So, I've got a big spreadsheet that tracks the days (which I can check off), the lunar cycle, the temperature high and low for where the PC's are likely to be at that time, dawn, dusk, sunrise, and sunset of where they are likely to be, wind direction and speed, and movements/happenings of the horde.
Finding real world data for the weather of various points around Scandinavia from several years (I'm basing my weather pattern on 1986), and sunrise/sunset data through the fall (I've been collecting it real-time every few days this fall) has been a great help in constructing the above.