Problem with large sets

Nov 11, 2009 at 5:05 PM

I have problem with sets larger then 100 images.

To be fair I do have manages to synch larger sets but with a bit of struggle.

Is it just my set-up, or is it not possible to synch sets with 1000-4000 images.

The most common error shows up after 100 images or so, it starts to slow down and then asks I I have an internet connection.

If I break down the set into smaller temporary sets it works fine.

Coordinator
Nov 11, 2009 at 10:07 PM

I've successfully synced sets with up to 1200 pictures. As a test I just ran the tool on my largest set. It fetches the metadata for 1200 pictures from Flickr within a minute. I've no reason to suspect it cannot handle much higher numbers of pictures.

Are you running the latest version of the app?

Nov 11, 2009 at 11:02 PM

Hi

Yes I have the latest version, and I've tried on another puter (vista, but on the same internet connection) with the same error.

I downloaded a copy just now and got this in my log:

---------------

 

FlickrMetadataSynchr.exe Information: 0 : Status #332: 31% - Reading metadata from Flickr pictures. Progress: 319 of 632.
    DateTime=2009-11-11T23:40:58.8010000Z
FlickrMetadataSynchr.exe Error: 0 : Yorrick.FlickrMetadataSynchr.Flickr.FlickrHelperException: Failed to connect to Flickr. Do you have a working Internet connection? Make sure this application isn't blocked by your firewall. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 68.142.214.24:80
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.InternalConnect(EndPoint remoteEP)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.GetRequestStream()
   at FlickrNet.Flickr.DoGetResponse(String url, String variables) in C:\Sources\FlickrNet\FlickrNet\Flickr.cs:line 446
   at FlickrNet.Flickr.GetResponse(Hashtable parameters, TimeSpan cacheTimeout) in C:\Sources\FlickrNet\FlickrNet\Flickr.cs:line 611
   at FlickrNet.Flickr.GetResponseCache(Hashtable parameters) in C:\Sources\FlickrNet\FlickrNet\Flickr.cs:line 561
   at FlickrNet.Flickr.PhotosGetInfo(String photoId, String secret) in C:\Sources\FlickrNet\FlickrNet\Flickr.cs:line 2661
   at Yorrick.FlickrMetadataSynchr.Flickr.AsyncPhotoInfoFetcher.ProcessItem(Photo metadata) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\AsyncPhotoInfoFetcher.cs:line 43
   at Yorrick.FlickrMetadataSynchr.Flickr.AsyncFlickrWorker`2.StartWork() in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\AsyncFlickrWorker.cs:line 148
   --- End of inner exception stack trace ---
   at Yorrick.FlickrMetadataSynchr.Flickr.FlickrHelper.ProcessMetadataWithMultipleWorkers[TIn,TOut](IList`1 metadataInList, EventHandler`1 progressCallback, CreateAsyncFlickrWorker`2 workerFactoryMethod) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\FlickrHelper.cs:line 1005
   at Yorrick.FlickrMetadataSynchr.Flickr.FlickrHelper.GetPhotoInfo(IList`1 photoList, EventHandler`1 progressCallback) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\FlickrHelper.cs:line 523
   at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.GetFlickrPictureSyncMetadata(IList`1 flickrPhotoList, Int32 startProgress, Int32 targetProgress) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 821
   at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.FetchPictureMetadata(Int32 currentProgress, Int32 targetProgress, IList`1& flickrPictureSyncMetadataList, IList`1& localPictureSyncMetadataList) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 575
   at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.DoSync() in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 401
   at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.DoSync(Object sender, DoWorkEventArgs e) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 938
    DateTime=2009-11-11T23:40:58.9670000Z
FlickrMetadataSynchr.exe Information: 0 : Status #333: 0% - The sync process failed: Failed to connect to Flickr. Do you have a working Internet connection? Make sure this application isn't blocked by your firewall.
    DateTime=2009-11-11T23:40:58.9680000Z
Loading cache from disk

FlickrMetadataSynchr.exe Information: 0 : Status #332: 31% - Reading metadata from Flickr pictures. Progress: 319 of 632.

    DateTime=2009-11-11T23:40:58.8010000Z

FlickrMetadataSynchr.exe Error: 0 : Yorrick.FlickrMetadataSynchr.Flickr.FlickrHelperException: Failed to connect to Flickr. Do you have a working Internet connection? Make sure this application isn't blocked by your firewall. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 68.142.214.24:80

   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)

   at System.Net.Sockets.Socket.InternalConnect(EndPoint remoteEP)

   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)

   --- End of inner exception stack trace ---

   at System.Net.HttpWebRequest.GetRequestStream()

   at FlickrNet.Flickr.DoGetResponse(String url, String variables) in C:\Sources\FlickrNet\FlickrNet\Flickr.cs:line 446

   at FlickrNet.Flickr.GetResponse(Hashtable parameters, TimeSpan cacheTimeout) in C:\Sources\FlickrNet\FlickrNet\Flickr.cs:line 611

   at FlickrNet.Flickr.GetResponseCache(Hashtable parameters) in C:\Sources\FlickrNet\FlickrNet\Flickr.cs:line 561

   at FlickrNet.Flickr.PhotosGetInfo(String photoId, String secret) in C:\Sources\FlickrNet\FlickrNet\Flickr.cs:line 2661

   at Yorrick.FlickrMetadataSynchr.Flickr.AsyncPhotoInfoFetcher.ProcessItem(Photo metadata) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\AsyncPhotoInfoFetcher.cs:line 43

   at Yorrick.FlickrMetadataSynchr.Flickr.AsyncFlickrWorker`2.StartWork() in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\AsyncFlickrWorker.cs:line 148

   --- End of inner exception stack trace ---

   at Yorrick.FlickrMetadataSynchr.Flickr.FlickrHelper.ProcessMetadataWithMultipleWorkers[TIn,TOut](IList`1 metadataInList, EventHandler`1 progressCallback, CreateAsyncFlickrWorker`2 workerFactoryMethod) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\FlickrHelper.cs:line 1005

   at Yorrick.FlickrMetadataSynchr.Flickr.FlickrHelper.GetPhotoInfo(IList`1 photoList, EventHandler`1 progressCallback) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\FlickrHelper.cs:line 523

   at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.GetFlickrPictureSyncMetadata(IList`1 flickrPhotoList, Int32 startProgress, Int32 targetProgress) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 821

   at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.FetchPictureMetadata(Int32 currentProgress, Int32 targetProgress, IList`1& flickrPictureSyncMetadataList, IList`1& localPictureSyncMetadataList) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 575

   at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.DoSync() in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 401

   at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.DoSync(Object sender, DoWorkEventArgs e) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 938

    DateTime=2009-11-11T23:40:58.9670000Z

FlickrMetadataSynchr.exe Information: 0 : Status #333: 0% - The sync process failed: Failed to connect to Flickr. Do you have a working Internet connection? Make sure this application isn't blocked by your firewall.

    DateTime=2009-11-11T23:40:58.9680000Z

Loading cache from disk

----------------------------------
When I first got the error I where on a mobile connection and simply filed it under "things to do when I get a real connection" now I'm on a cable modem 10/24mbps

 

Coordinator
Nov 12, 2009 at 9:50 AM

This seems to be a issue in your network infrastructure. The app establishes up to 10 simultaneous connections to the Flickr web services with a timeout of 30 seconds. And apparently some of those connections fail to respond within the timeout period for you. Your bandwith seems to be plenty. Could your ISP be throttling connections? Do you have a proxy server configured in Internet Explorer?

If you open the FlickrMetadataSynchr.exe.config file in Notepad you can try changing some settings. There is

<setting name="MaximumNumberOfAsyncWorkers" serializeAs="String">
        <value>10</value>
</setting>

Try lowering the number. And there is:

<setting name="FlickrNetHttpTimeoutInSeconds" serializeAs="String">
        <value>30</value>
</setting>

Try increasing that number. Normally 30 seconds is plenty of time. For me the Flickr web services respond in a matter of seconds.

Nov 13, 2009 at 10:23 PM

It works :) 

I have now set the time-out to 600 and the thread to 1 and it allows me to download larger sets, the network error still shows up, but less frequently.

Other errors show up now. In one case I had to do a split, but for the most part it have worked with a few retries. Its a bit sad that the program run "error-fail-stop" instead of error-retry-log and continue. I will paste a few just for info..

 

This error seemed a bit weird. I tried to put all images in a new temporary group, with no luck. Splitting them worked, giving me 2 images less, so I guess there are duplicates in the set, but I'm not sure.

 

FlickrMetadataSynchr.exe Information: 0 : Flickr photoset has 500 pictures. Will fetch info in 3 batches.

DateTime=2009-11-13T18:59:14.7436041Z

FlickrMetadataSynchr.exe Information: 0 : Fetched 250 pictures in batch #1.

DateTime=2009-11-13T18:59:16.0636041Z

FlickrMetadataSynchr.exe Information: 0 : Fetched 250 pictures in batch #2.

DateTime=2009-11-13T18:59:17.4116041Z

FlickrMetadataSynchr.exe Error: 0 : Could not fetch photos from Flickr. Exception details follow:

FlickrNet.FlickrApiException: Photoset not found (1)

at FlickrNet.Flickr.PhotosetsGetPhotos(String photosetId, PhotoSearchExtras extras, PrivacyFilter privacyFilter, Int32 page, Int32 perPage) in C:\Sources\FlickrNet\FlickrNet\Flickr.cs:line 4375

at Yorrick.FlickrMetadataSynchr.Flickr.FlickrHelper.GetPhotos(Photoset photoset) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\FlickrHelper.cs:line 445

DateTime=2009-11-13T18:59:17.8426041Z

FlickrMetadataSynchr.exe Error: 0 : Yorrick.FlickrMetadataSynchr.Flickr.FlickrHelperException: Could not fetch photos from Flickr. ---> FlickrNet.FlickrApiException: Photoset not found (1)

at FlickrNet.Flickr.PhotosetsGetPhotos(String photosetId, PhotoSearchExtras extras, PrivacyFilter privacyFilter, Int32 page, Int32 perPage) in C:\Sources\FlickrNet\FlickrNet\Flickr.cs:line 4375

at Yorrick.FlickrMetadataSynchr.Flickr.FlickrHelper.GetPhotos(Photoset photoset) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\FlickrHelper.cs:line 445

--- End of inner exception stack trace ---

at Yorrick.FlickrMetadataSynchr.Flickr.FlickrHelper.GetPhotos(Photoset photoset) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\FlickrHelper.cs:line 467

at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.FetchPictureMetadata(Int32 currentProgress, Int32 targetProgress, IList`1& flickrPictureSyncMetadataList, IList`1& localPictureSyncMetadataList) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 563

at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.DoSync() in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 401

at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.DoSync(Object sender, DoWorkEventArgs e) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 938

DateTime=2009-11-13T18:59:17.8476041Z

FlickrMetadataSynchr.exe Information: 0 : Status #8: 0% - The sync process failed: Could not fetch photos from Flickr.

DateTime=2009-11-13T18:59:17.8626041Z

Loading cache from disk 

 

 

This is just an example-row of what I get on all images when I resynchronise a newly synced/downloaded set. All geo is adjusted by 0,000001.

Third synch do not show this behaviour.

 

FlickrMetadataSynchr.exe Information: 0 : Updated geolocation for Flickr picture with id 3039398948 from (59,593027 , 17,88719) to (59.593026 , 17.88719).

DateTime=2009-11-13T20:13:22.7656041Z

 

 

Not sure about this one, it worked after a few retries without changing anything.

 

FlickrMetadataSynchr.exe Information: 0 : Status #4164: 48% - Reading metadata from Flickr pictures. Progress: 543 of 560.

DateTime=2009-11-13T20:40:24.6476041Z

FlickrMetadataSynchr.exe Error: 0 : Could not fetch photo info from Flickr. Exception details follow:

FlickrNet.FlickrApiException: Sorry, the Flickr API service is not currently available. (0)

at FlickrNet.Flickr.PhotosGetInfo(String photoId, String secret) in C:\Sources\FlickrNet\FlickrNet\Flickr.cs:line 2669

at Yorrick.FlickrMetadataSynchr.Flickr.AsyncPhotoInfoFetcher.ProcessItem(Photo metadata) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\AsyncPhotoInfoFetcher.cs:line 43

at Yorrick.FlickrMetadataSynchr.Flickr.AsyncFlickrWorker`2.StartWork() in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\AsyncFlickrWorker.cs:line 148

DateTime=2009-11-13T20:40:25.0186041Z

FlickrMetadataSynchr.exe Error: 0 : Yorrick.FlickrMetadataSynchr.Flickr.FlickrHelperException: Could not fetch photo info from Flickr. ---> FlickrNet.FlickrApiException: Sorry, the Flickr API service is not currently available. (0)

at FlickrNet.Flickr.PhotosGetInfo(String photoId, String secret) in C:\Sources\FlickrNet\FlickrNet\Flickr.cs:line 2669

at Yorrick.FlickrMetadataSynchr.Flickr.AsyncPhotoInfoFetcher.ProcessItem(Photo metadata) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\AsyncPhotoInfoFetcher.cs:line 43

at Yorrick.FlickrMetadataSynchr.Flickr.AsyncFlickrWorker`2.StartWork() in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\AsyncFlickrWorker.cs:line 148

--- End of inner exception stack trace ---

at Yorrick.FlickrMetadataSynchr.Flickr.FlickrHelper.ProcessMetadataWithMultipleWorkers[TIn,TOut](IList`1 metadataInList, EventHandler`1 progressCallback, CreateAsyncFlickrWorker`2 workerFactoryMethod) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\FlickrHelper.cs:line 1005

at Yorrick.FlickrMetadataSynchr.Flickr.FlickrHelper.GetPhotoInfo(IList`1 photoList, EventHandler`1 progressCallback) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Flickr\FlickrHelper.cs:line 523

at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.GetFlickrPictureSyncMetadata(IList`1 flickrPhotoList, Int32 startProgress, Int32 targetProgress) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 821

at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.FetchPictureMetadata(Int32 currentProgress, Int32 targetProgress, IList`1& flickrPictureSyncMetadataList, IList`1& localPictureSyncMetadataList) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 575

at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.DoSync() in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 401

at Yorrick.FlickrMetadataSynchr.Sync.SynchronizationWorker.DoSync(Object sender, DoWorkEventArgs e) in C:\Sources\FlickrMetadataSynchr\Main\FlickrMetadataSynchr\Sync\SynchronizationWorker.cs:line 938

DateTime=2009-11-13T20:40:25.0246041Z

FlickrMetadataSynchr.exe Information: 0 : Status #4165: 48% - Reading metadata from Flickr pictures. Progress: 544 of 560.

DateTime=2009-11-13T20:40:25.0286041Z

FlickrMetadataSynchr.exe Information: 0 : Status #4166: 0% - The sync process failed: Could not fetch photo info from Flickr.

DateTime=2009-11-13T20:40:25.0326041Z

 

Coordinator
Nov 13, 2009 at 10:33 PM

You are apparently suffering from a seriously broken Internet connection to Flickr. As there are no other reports for these kinds of connection problems I have no idea what is causing this on your end.

In such circumstances there is just no reliable way to continue syncing. That's why the program deliberately stops when it runs into these kinds of problems.

Nov 14, 2009 at 11:27 AM

I was a afraid of that :(

I'm going to test it at another ISP when I get a chance. Thanks for the help so far. No other program is even close to get the job done.