FlickrMetadataSynchr Forum Rss Feed

New Post: Problem with large sets

frisno — Sat, 14 Nov 2009 12:27:12 GMT

I was a afraid of that :(

I'm going to test it at another ISP when I get a chance. Thanks for the help so far. No other program is even close to get the job done.

New Post: Problem with large sets

erwyn — Fri, 13 Nov 2009 23:33:08 GMT

You are apparently suffering from a seriously broken Internet connection to Flickr. As there are no other reports for these kinds of connection problems I have no idea what is causing this on your end.

In such circumstances there is just no reliable way to continue syncing. That's why the program deliberately stops when it runs into these kinds of problems.

New Post: Problem with large sets

frisno — Fri, 13 Nov 2009 23:23:23 GMT

It works :)

I have now set the time-out to 600 and the thread to 1 and it allows me to download larger sets, the network error still shows up, but less frequently.

Other errors show up now. In one case I had to do a split, but for the most part it have worked with a few retries. Its a bit sad that the program run "error-fail-stop" instead of error-retry-log and continue. I will paste a few just for info..

This error seemed a bit weird. I tried to put all images in a new temporary group, with no luck. Splitting them worked, giving me 2 images less, so I guess there are duplicates in the set, but I'm not sure.

FlickrMetadataSynchr.exe Information: 0 : Flickr photoset has 500 pictures. Will fetch info in 3 batches.

DateTime=2009-11-13T18:59:14.7436041Z

FlickrMetadataSynchr.exe Information: 0 : Fetched 250 pictures in batch #1.

DateTime=2009-11-13T18:59:16.0636041Z

FlickrMetadataSynchr.exe Information: 0 : Fetched 250 pictures in batch #2.

DateTime=2009-11-13T18:59:17.4116041Z

FlickrMetadataSynchr.exe Error: 0 : Could not fetch photos from Flickr. Exception details follow:

FlickrNet.FlickrApiException: Photoset not found (1)

at FlickrNet.Flickr.PhotosetsGetPhotos(String photosetId, PhotoSearchExtras extras, PrivacyFilter privacyFilter, Int32 page, Int32 perPage) in C:\Sources\FlickrNet\FlickrNet\Flickr.cs:line 4375

at Yorrick.FlickrMetadataSynchr.Flickr.FlickrHelper.GetPhotos(Photoset photoset) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\FlickrHelper.cs:line 445

DateTime=2009-11-13T18:59:17.8426041Z

FlickrMetadataSynchr.exe Error: 0 : Yorrick.FlickrMetadataSynchr.Flickr.FlickrHelperException: Could not fetch photos from Flickr. ---> FlickrNet.FlickrApiException: Photoset not found (1)

at FlickrNet.Flickr.PhotosetsGetPhotos(String photosetId, PhotoSearchExtras extras, PrivacyFilter privacyFilter, Int32 page, Int32 perPage) in C:\Sources\FlickrNet\FlickrNet\Flickr.cs:line 4375

at Yorrick.FlickrMetadataSynchr.Flickr.FlickrHelper.GetPhotos(Photoset photoset) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\FlickrHelper.cs:line 445

--- End of inner exception stack trace ---

at Yorrick.FlickrMetadataSynchr.Flickr.FlickrHelper.GetPhotos(Photoset photoset) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\FlickrHelper.cs:line 467

at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.FetchPictureMetadata(Int32 currentProgress, Int32 targetProgress, IList`1& flickrPictureSyncMetadataList, IList`1& localPictureSyncMetadataList) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 563

at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.DoSync() in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 401

at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.DoSync(Object sender, DoWorkEventArgs e) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 938

DateTime=2009-11-13T18:59:17.8476041Z

FlickrMetadataSynchr.exe Information: 0 : Status #8: 0% - The sync process failed: Could not fetch photos from Flickr.

DateTime=2009-11-13T18:59:17.8626041Z

Loading cache from disk

This is just an example-row of what I get on all images when I resynchronise a newly synced/downloaded set. All geo is adjusted by 0,000001.

Third synch do not show this behaviour.

FlickrMetadataSynchr.exe Information: 0 : Updated geolocation for Flickr picture with id 3039398948 from (59,593027 , 17,88719) to (59.593026 , 17.88719).

DateTime=2009-11-13T20:13:22.7656041Z

Not sure about this one, it worked after a few retries without changing anything.

FlickrMetadataSynchr.exe Information: 0 : Status #4164: 48% - Reading metadata from Flickr pictures. Progress: 543 of 560.

DateTime=2009-11-13T20:40:24.6476041Z

FlickrMetadataSynchr.exe Error: 0 : Could not fetch photo info from Flickr. Exception details follow:

FlickrNet.FlickrApiException: Sorry, the Flickr API service is not currently available. (0)

at FlickrNet.Flickr.PhotosGetInfo(String photoId, String secret) in C:\Sources\FlickrNet\FlickrNet\Flickr.cs:line 2669

at Yorrick.FlickrMetadataSynchr.Flickr.AsyncPhotoInfoFetcher.ProcessItem(Photo metadata) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\AsyncPhotoInfoFetcher.cs:line 43

at Yorrick.FlickrMetadataSynchr.Flickr.AsyncFlickrWorker`2.StartWork() in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\AsyncFlickrWorker.cs:line 148

DateTime=2009-11-13T20:40:25.0186041Z

FlickrMetadataSynchr.exe Error: 0 : Yorrick.FlickrMetadataSynchr.Flickr.FlickrHelperException: Could not fetch photo info from Flickr. ---> FlickrNet.FlickrApiException: Sorry, the Flickr API service is not currently available. (0)

at FlickrNet.Flickr.PhotosGetInfo(String photoId, String secret) in C:\Sources\FlickrNet\FlickrNet\Flickr.cs:line 2669

at Yorrick.FlickrMetadataSynchr.Flickr.AsyncPhotoInfoFetcher.ProcessItem(Photo metadata) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\AsyncPhotoInfoFetcher.cs:line 43

at Yorrick.FlickrMetadataSynchr.Flickr.AsyncFlickrWorker`2.StartWork() in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\AsyncFlickrWorker.cs:line 148

--- End of inner exception stack trace ---

at Yorrick.FlickrMetadataSynchr.Flickr.FlickrHelper.ProcessMetadataWithMultipleWorkers[TIn,TOut](IList`1 metadataInList, EventHandler`1 progressCallback, CreateAsyncFlickrWorker`2 workerFactoryMethod) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\FlickrHelper.cs:line 1005

at Yorrick.FlickrMetadataSynchr.Flickr.FlickrHelper.GetPhotoInfo(IList`1 photoList, EventHandler`1 progressCallback) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\FlickrHelper.cs:line 523

at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.GetFlickrPictureSyncMetadata(IList`1 flickrPhotoList, Int32 startProgress, Int32 targetProgress) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 821

at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.DoSync() in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 401

at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.DoSync(Object sender, DoWorkEventArgs e) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 938

DateTime=2009-11-13T20:40:25.0246041Z

FlickrMetadataSynchr.exe Information: 0 : Status #4165: 48% - Reading metadata from Flickr pictures. Progress: 544 of 560.

DateTime=2009-11-13T20:40:25.0286041Z

FlickrMetadataSynchr.exe Information: 0 : Status #4166: 0% - The sync process failed: Could not fetch photo info from Flickr.

DateTime=2009-11-13T20:40:25.0326041Z

New Post: Problem with large sets

erwyn — Thu, 12 Nov 2009 10:50:26 GMT

This seems to be a issue in your network infrastructure. The app establishes up to 10 simultaneous connections to the Flickr web services with a timeout of 30 seconds. And apparently some of those connections fail to respond within the timeout period for you. Your bandwith seems to be plenty. Could your ISP be throttling connections? Do you have a proxy server configured in Internet Explorer?

If you open the FlickrMetadataSynchr.exe.config file in Notepad you can try changing some settings. There is

Try lowering the number. And there is:

<setting name="FlickrNetHttpTimeoutInSeconds" serializeAs="String">
<value>30</value>
</setting>

Try increasing that number. Normally 30 seconds is plenty of time. For me the Flickr web services respond in a matter of seconds.

New Post: Problem with large sets

frisno — Thu, 12 Nov 2009 00:02:28 GMT

Yes I have the latest version, and I've tried on another puter (vista, but on the same internet connection) with the same error.

I downloaded a copy just now and got this in my log:

---------------

FlickrMetadataSynchr.exe Information: 0 : Status #332: 31% - Reading metadata from Flickr pictures. Progress: 319 of 632.

DateTime=2009-11-11T23:40:58.8010000Z

FlickrMetadataSynchr.exe Error: 0 : Yorrick.FlickrMetadataSynchr.Flickr.FlickrHelperException: Failed to connect to Flickr. Do you have a working Internet connection? Make sure this application isn't blocked by your firewall. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 68.142.214.24:80

at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)

at System.Net.Sockets.Socket.InternalConnect(EndPoint remoteEP)

at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)

--- End of inner exception stack trace ---

at System.Net.HttpWebRequest.GetRequestStream()

at FlickrNet.Flickr.DoGetResponse(String url, String variables) in C:\Sources\FlickrNet\FlickrNet\Flickr.cs:line 446

at FlickrNet.Flickr.GetResponse(Hashtable parameters, TimeSpan cacheTimeout) in C:\Sources\FlickrNet\FlickrNet\Flickr.cs:line 611

at FlickrNet.Flickr.GetResponseCache(Hashtable parameters) in C:\Sources\FlickrNet\FlickrNet\Flickr.cs:line 561

at FlickrNet.Flickr.PhotosGetInfo(String photoId, String secret) in C:\Sources\FlickrNet\FlickrNet\Flickr.cs:line 2661

at Yorrick.FlickrMetadataSynchr.Flickr.AsyncPhotoInfoFetcher.ProcessItem(Photo metadata) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\AsyncPhotoInfoFetcher.cs:line 43

at Yorrick.FlickrMetadataSynchr.Flickr.AsyncFlickrWorker`2.StartWork() in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\AsyncFlickrWorker.cs:line 148

--- End of inner exception stack trace ---

at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.DoSync() in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 401

at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.DoSync(Object sender, DoWorkEventArgs e) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 938

DateTime=2009-11-11T23:40:58.9670000Z

FlickrMetadataSynchr.exe Information: 0 : Status #333: 0% - The sync process failed: Failed to connect to Flickr. Do you have a working Internet connection? Make sure this application isn't blocked by your firewall.

DateTime=2009-11-11T23:40:58.9680000Z

Loading cache from disk

FlickrMetadataSynchr.exe Information: 0 : Status #332: 31% - Reading metadata from Flickr pictures. Progress: 319 of 632.

DateTime=2009-11-11T23:40:58.8010000Z

at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)

at System.Net.Sockets.Socket.InternalConnect(EndPoint remoteEP)

--- End of inner exception stack trace ---

at System.Net.HttpWebRequest.GetRequestStream()

at FlickrNet.Flickr.DoGetResponse(String url, String variables) in C:\Sources\FlickrNet\FlickrNet\Flickr.cs:line 446

at FlickrNet.Flickr.GetResponse(Hashtable parameters, TimeSpan cacheTimeout) in C:\Sources\FlickrNet\FlickrNet\Flickr.cs:line 611

at FlickrNet.Flickr.GetResponseCache(Hashtable parameters) in C:\Sources\FlickrNet\FlickrNet\Flickr.cs:line 561

at FlickrNet.Flickr.PhotosGetInfo(String photoId, String secret) in C:\Sources\FlickrNet\FlickrNet\Flickr.cs:line 2661

at Yorrick.FlickrMetadataSynchr.Flickr.AsyncPhotoInfoFetcher.ProcessItem(Photo metadata) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\AsyncPhotoInfoFetcher.cs:line 43

at Yorrick.FlickrMetadataSynchr.Flickr.AsyncFlickrWorker`2.StartWork() in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\AsyncFlickrWorker.cs:line 148

--- End of inner exception stack trace ---

at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.DoSync() in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 401

at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.DoSync(Object sender, DoWorkEventArgs e) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 938

DateTime=2009-11-11T23:40:58.9670000Z

DateTime=2009-11-11T23:40:58.9680000Z

Loading cache from disk

----------------------------------

When I first got the error I where on a mobile connection and simply filed it under "things to do when I get a real connection" now I'm on a cable modem 10/24mbps

New Post: Problem with large sets

erwyn — Wed, 11 Nov 2009 23:07:52 GMT

I've successfully synced sets with up to 1200 pictures. As a test I just ran the tool on my largest set. It fetches the metadata for 1200 pictures from Flickr within a minute. I've no reason to suspect it cannot handle much higher numbers of pictures.

Are you running the latest version of the app?

New Post: Problem with large sets

frisno — Wed, 11 Nov 2009 18:05:02 GMT

I have problem with sets larger then 100 images.

To be fair I do have manages to synch larger sets but with a bit of struggle.

Is it just my set-up, or is it not possible to synch sets with 1000-4000 images.

The most common error shows up after 100 images or so, it starts to slow down and then asks I I have an internet connection.

If I break down the set into smaller temporary sets it works fine.

New Post: Microsoft Photo Metadata Policy

erwyn — Wed, 26 Aug 2009 17:41:33 GMT

Microsoft has implemented the Microsoft Photo Metadata Policy as a Windows platform component, called the Windows Imaging Component or WIC for short. My app uses this component (through .NET Framework 3.0) to read and write metadata to image files.

New Post: Microsoft Photo Metadata Policy

rctaubert — Wed, 26 Aug 2009 17:33:50 GMT

You made the comment "My app uses the Microsoft Photo Metadata Policy." By that, do you mean you use the Microsoft Photo Metadata Policy as a guide to your coding or does MS provide code to implement Microsoft Photo Metadata Policy?????

New Post: Not compatible with Picasa / Which EXIF/IPTC fields are used?

erwyn — Sat, 22 Aug 2009 13:19:11 GMT

This discussion has been copied to a work item. Click here to go to the work item and continue the discussion.

New Post: Not compatible with Picasa / Which EXIF/IPTC fields are used?

erwyn — Sat, 22 Aug 2009 13:15:53 GMT

Nate, indeed it doesn't look very difficicult to add writing keywords to IPTC in addition to XMP. I might add it in a next release (but no date planned for that yet).

The Microsoft Photo Metadata Policy for System.Keywords does already read from "/app13/irb/8bimiptc/iptc/keywords" if present, but doesn't write to it if no IPTC keywords are already present in the file. In that case it just writes to XMP (the most modern spot to write this type of metadata to).

All in all, I still don't understand why the issue exists. If Picasa was used to add keywords in IPTC to a file I would think that my tool would update those keywords if changes have occurred on Flickr. I'll create a work item for this and lets continue the discussion there. Can you attach a picture for this happens to that work item?

New Post: Not compatible with Picasa / Which EXIF/IPTC fields are used?

drumroll99 — Sat, 22 Aug 2009 04:55:03 GMT

Erwyn, Picasa is one of the reasons for my side project flickrmetasync. It sucks that Picasa doesn't read XMP, but I like the UI so much that I can't do without it at the moment. If you ever decide to write IPTC keywords (which is something I personally would recommend) feel free to take a look at my source code on google code to see how it is writing the IPTC keywords. All the best, Nate

New Post: Not compatible with Picasa / Which EXIF/IPTC fields are used?

erwyn — Sun, 16 Aug 2009 19:38:37 GMT

I use Windows Live Photo Gallery (WLPG). It is very easy to add keywords/tags using this application and manage a lot of photos and it deals with XMP/IPTC/EXIF just fine. I also use WLPG to stitch photos together, either using the built-in function or using Microsoft Image Composite Editor that integrates as a menu option into WLPG.

WLPG also allows you to easily upload photos to Flickr or Windows Live Photos. WLPG is a bit wonky in that it adds the filename as title on Flickr, and the caption as description on Flickr when uploading. Tags are mapped fine.

After this initial upload hurdle my app syncs the Flickr title with what WLPG displays as caption. WLPG does not allow you to edit the comments field that I map to the Flickr description. But Windows Explorer in Vista and 7 does allow you to edit both the title and comments field.

WLPG doesn't do geocoding. I use the Microsoft Pro Photo Tools for that or I do it on Flickr and then sync. The UI of the Pro Photo Tools is not quite as polished as WLPG and I sometimes encounter some bugs, but all in all it works. It opens up a lot of additional metadata that you can add to a picture file.

New Post: Not compatible with Picasa / Which EXIF/IPTC fields are used?

fjordaan — Sun, 16 Aug 2009 19:00:07 GMT

Hi Erwyn,

Thanks for the explanation. Sounds like you've thought it through. Picasa is a frustrating application: It's excellent in so many respects, but deficient in several. (E.g. not supporting Flickr in any way, seemingly out of competitive spite.) And like many other Google properties, they're pretty much deaf to their user community, so I don't know when/whether they'll ever support XMP. Just out of curiousity, what application do you use for photo management?

Anyway, it's still valuable to be able to back up my Flickr metadata, so thanks again.

francois

New Post: Not compatible with Picasa / Which EXIF/IPTC fields are used?

erwyn — Sat, 15 Aug 2009 23:58:38 GMT

Hi Francois,

My app uses the Microsoft Photo Metadata Policy. The primary leading format for metadata is XMP. But I believe in cases where other metadata formats exists in the file, the corresponding IPTC fields should also be overwritten.

I am not planning on any changes to my app in this respect. Since I don't use Picasa I cannot comment on what it does or doesn't support. XMP is the way forward, so if Picasa doesn't support XMP metadata, hopefully it will be added soon.

Ciao,
Erwyn

New Post: Not compatible with Picasa / Which EXIF/IPTC fields are used?

fjordaan — Sat, 15 Aug 2009 21:22:44 GMT

Hi,

Firstly, thanks for a very useful programme. I heard about it via my request for something similar on Ask Metafilter.

I've tested it, and it seems to work. Local JPGs are updated with the correct title, description and keywords.

However, I was hoping that Picasa, which stores Caption and Keywords in IPTC fields, would recognise the the new metadata from Flickr -- at least the Keywords -- but it doesn't.

Using Irfanview to check image properties, I cannot see any updates to the IPTC fields. The only field that appears to have changed is UserComment, in the EXIF data.

Why did you decide not to use the IPTC fields? Would you consider doing so in a future update?

francois

New Post: Log errors and continue

erwyn — Wed, 29 Jul 2009 08:43:20 GMT

If you still have a "corrupt" image file that will be useful to me as well in order to replicate the error. The app skips over all metadata problems that I am currently aware of (and reports them in the log file indicating the filename), so this must be a new class of problems.

New Post: Log errors and continue

ianjohnson — Tue, 28 Jul 2009 22:42:22 GMT

I'll post the error after my laptop gets back from the shop. It was
something like the metadata couldn't be read because the file was
corrupt. I had several photos that wouldn't upload thru Flickr
Uploadr for the same reason but I didn't keep track of which ones.

The stack trace doesn't help because I was synching a lot of
subfolders and I don't know what order it was doing them in. Using
file names in the log would make tracking down problems a lot easier.

New Post: Log errors and continue

erwyn — Tue, 28 Jul 2009 18:27:11 GMT

Hi Ian. The app does handle some common errors, logs them and continues. Even if an error goes unhandled, the log window and log file should show the "stack trace" for the error that occurs. This should help in pinpointing the picture that causes the issue.

Please attach your log file containing an error that you would like to see handled more gracefully.

New Post: Log errors and continue

ianjohnson — Mon, 20 Jul 2009 01:25:45 GMT

Also I have some corrupted images on the local disk that are halting the sync. Unfortunately I don't have any way to tell which image stopped it. It would be nice to automatically skip such files, and give me a list of problem files at the end.