Topic: phpdvdprofiler

[FredLooks]

The difference should be that the messages about how many profiles have been processed should have a time at the front of each line. I'm hoping that those times will indicate whether we're hitting a time limit of some sort.

Incidentally, the theory of operation ( ) of updates is this:

Ordinarily, one would leave the Complete Update checkbox unchecked. This will cause the program to skip profiles that haven't changed since the last update. If for some reason one thinks that there is a problem with the database or for some other reason, the Complete Update checkbox can be checked. When it is checked, the update process completely removes all of the data in the database before starting, resulting in all profiles looking "new".

Now, if one is having a problem such as yours, the idea is to do a Complete Update to start from a known state. After it stops, then run regular updates until the process actually finishes. Each consecutive update should get farther, as it skips more of the collection.

I would be interested in how this expected behaviour compares with your observations (I'm not entirely certain which of the updates you are doing are Complete Updates, which "reset the clock" so to speak...)

[Dragonfire]

Here's what shows up when I try to do an update..not the complete one.

Faster, more memory-intensive updates: On
Importing to database b2_5841265_phpdvdprofiler
Importing 2748 files from directory XML
1.932: DVD profiles processed: 100
2.785: DVD profiles processed: 200
3.829: DVD profiles processed: 300
XML/.xml: Profile IDs must not be blank. Profile #325
4.715: DVD profiles processed: 400
5.855: DVD profiles processed: 500
6.616: DVD profiles processed: 600
7.315: DVD profiles processed: 700
7.852: DVD profiles processed: 800
8.401: DVD profiles processed: 900
9.022: DVD profiles processed: 1000
9.928: DVD profiles processed: 1100
10.783: DVD profiles processed: 1200
11.594: DVD profiles processed: 1300
12.416: DVD profiles processed: 1400
13.224: DVD profiles processed: 1500
13.633: DVD profiles processed: 1600
14.336: DVD profiles processed: 1700
15.209: DVD profiles processed: 1800
16.065: DVD profiles processed: 1900
16.735: DVD profiles processed: 2000


Here's what comes up when I try the complete one.

Faster, more memory-intensive updates: On
Complete Update will be performed
Importing to database b2_5841265_phpdvdprofiler
Importing 2748 files from directory XML

*******************************************************************************
XML/043396503892.xml, the current collection file, is missing some components. When exporting from DVDProfiler
it is important to select ALL of the sections to be exported. There will be no data stored
for the following missing sections: Locks Crew Tags Easter Eggs
*******************************************************************************
14.601: DVD profiles processed: 100
22.830: DVD profiles processed: 200
31.309: DVD profiles processed: 300
XML/.xml: Profile IDs must not be blank. Profile #325
38.829: DVD profiles processed: 400
46.907: DVD profiles processed: 500
55.183: DVD profiles processed: 600

It has stopped at 600...I'll try again.

[Dragonfire]

Ok...Here's what is coming up while I keep trying to get the update to finish.  This is without having the option for complete update checked.

Faster, more memory-intensive updates: On
Importing to database b2_5841265_phpdvdprofiler
Importing 2748 files from directory XML
0.319: DVD profiles processed: 100
0.464: DVD profiles processed: 200
0.615: DVD profiles processed: 300
XML/.xml: Profile IDs must not be blank. Profile #325
0.766: DVD profiles processed: 400
0.921: DVD profiles processed: 500
1.087: DVD profiles processed: 600
8.662: DVD profiles processed: 700
26.896: DVD profiles processed: 800
43.925: DVD profiles processed: 900
62.327: DVD profiles processed: 1000


Faster, more memory-intensive updates: On
Importing to database b2_5841265_phpdvdprofiler
Importing 2748 files from directory XML
0.404: DVD profiles processed: 100
0.578: DVD profiles processed: 200
0.734: DVD profiles processed: 300
XML/.xml: Profile IDs must not be blank. Profile #325
0.873: DVD profiles processed: 400
1.009: DVD profiles processed: 500
1.147: DVD profiles processed: 600
1.246: DVD profiles processed: 700
1.365: DVD profiles processed: 800
1.516: DVD profiles processed: 900
1.694: DVD profiles processed: 1000
11.079: DVD profiles processed: 1100
30.055: DVD profiles processed: 1200
52.799: DVD profiles processed: 1300
72.322: DVD profiles processed: 1400
91.781: DVD profiles processed: 1500


Faster, more memory-intensive updates: On
Importing to database b2_5841265_phpdvdprofiler
Importing 2748 files from directory XML
0.290: DVD profiles processed: 100
0.441: DVD profiles processed: 200
0.574: DVD profiles processed: 300
XML/.xml: Profile IDs must not be blank. Profile #325
0.707: DVD profiles processed: 400
0.899: DVD profiles processed: 500
1.150: DVD profiles processed: 600
1.271: DVD profiles processed: 700
1.383: DVD profiles processed: 800
1.510: DVD profiles processed: 900
1.688: DVD profiles processed: 1000
1.840: DVD profiles processed: 1100
1.973: DVD profiles processed: 1200
2.093: DVD profiles processed: 1300
2.338: DVD profiles processed: 1400
2.547: DVD profiles processed: 1500
5.235: DVD profiles processed: 1600
24.640: DVD profiles processed: 1700

[Dragonfire]

More attempts...

Faster, more memory-intensive updates: On
Importing to database b2_5841265_phpdvdprofiler
Importing 2748 files from directory XML
0.441: DVD profiles processed: 100
0.647: DVD profiles processed: 200
0.861: DVD profiles processed: 300
XML/.xml: Profile IDs must not be blank. Profile #325
1.016: DVD profiles processed: 400
1.154: DVD profiles processed: 500
1.269: DVD profiles processed: 600
1.392: DVD profiles processed: 700
1.521: DVD profiles processed: 800
1.652: DVD profiles processed: 900
1.810: DVD profiles processed: 1000
1.982: DVD profiles processed: 1100
2.141: DVD profiles processed: 1200
2.338: DVD profiles processed: 1300
2.508: DVD profiles processed: 1400
2.705: DVD profiles processed: 1500
2.898: DVD profiles processed: 1600
3.061: DVD profiles processed: 1700
22.566: DVD profiles processed: 1800
41.213: DVD profiles processed: 1900


Faster, more memory-intensive updates: On
Importing to database b2_5841265_phpdvdprofiler
Importing 2748 files from directory XML
0.263: DVD profiles processed: 100
0.348: DVD profiles processed: 200
0.451: DVD profiles processed: 300
XML/.xml: Profile IDs must not be blank. Profile #325
0.587: DVD profiles processed: 400
0.679: DVD profiles processed: 500
0.797: DVD profiles processed: 600
0.942: DVD profiles processed: 700
1.062: DVD profiles processed: 800
1.146: DVD profiles processed: 900
1.247: DVD profiles processed: 1000
1.340: DVD profiles processed: 1100
1.435: DVD profiles processed: 1200
1.683: DVD profiles processed: 1300
1.944: DVD profiles processed: 1400
2.134: DVD profiles processed: 1500
2.350: DVD profiles processed: 1600
2.560: DVD profiles processed: 1700
2.740: DVD profiles processed: 1800
2.920: DVD profiles processed: 1900
22.488: DVD profiles processed: 2000


Faster, more memory-intensive updates: On
Importing to database b2_5841265_phpdvdprofiler
Importing 2748 files from directory XML
0.340: DVD profiles processed: 100
0.473: DVD profiles processed: 200
0.583: DVD profiles processed: 300
XML/.xml: Profile IDs must not be blank. Profile #325
0.676: DVD profiles processed: 400
0.773: DVD profiles processed: 500
0.877: DVD profiles processed: 600
0.976: DVD profiles processed: 700
1.065: DVD profiles processed: 800
1.215: DVD profiles processed: 900
1.397: DVD profiles processed: 1000
1.599: DVD profiles processed: 1100
1.759: DVD profiles processed: 1200
1.891: DVD profiles processed: 1300
2.020: DVD profiles processed: 1400
2.183: DVD profiles processed: 1500
2.321: DVD profiles processed: 1600
2.443: DVD profiles processed: 1700
2.568: DVD profiles processed: 1800
2.690: DVD profiles processed: 1900
2.826: DVD profiles processed: 2000


Faster, more memory-intensive updates: On
Importing to database b2_5841265_phpdvdprofiler
Importing 2748 files from directory XML
0.262: DVD profiles processed: 100
0.348: DVD profiles processed: 200
0.462: DVD profiles processed: 300
XML/.xml: Profile IDs must not be blank. Profile #325
0.553: DVD profiles processed: 400
0.645: DVD profiles processed: 500
0.760: DVD profiles processed: 600
0.895: DVD profiles processed: 700
1.021: DVD profiles processed: 800
1.154: DVD profiles processed: 900
1.316: DVD profiles processed: 1000
1.471: DVD profiles processed: 1100
1.594: DVD profiles processed: 1200
1.710: DVD profiles processed: 1300
1.832: DVD profiles processed: 1400
1.983: DVD profiles processed: 1500
2.112: DVD profiles processed: 1600
2.240: DVD profiles processed: 1700
2.411: DVD profiles processed: 1800
2.606: DVD profiles processed: 1900
2.789: DVD profiles processed: 2000

Faster, more memory-intensive updates: On
Importing to database b2_5841265_phpdvdprofiler
Importing 2748 files from directory XML
0.415: DVD profiles processed: 100
0.602: DVD profiles processed: 200
0.731: DVD profiles processed: 300
XML/.xml: Profile IDs must not be blank. Profile #325
0.863: DVD profiles processed: 400
0.978: DVD profiles processed: 500
1.105: DVD profiles processed: 600
1.248: DVD profiles processed: 700
1.358: DVD profiles processed: 800
1.476: DVD profiles processed: 900
1.628: DVD profiles processed: 1000
1.803: DVD profiles processed: 1100
1.922: DVD profiles processed: 1200
2.097: DVD profiles processed: 1300
2.285: DVD profiles processed: 1400
2.514: DVD profiles processed: 1500
2.718: DVD profiles processed: 1600
2.901: DVD profiles processed: 1700
3.113: DVD profiles processed: 1800
3.328: DVD profiles processed: 1900
3.517: DVD profiles processed: 2000

[Achim]

XML/.xml: Profile IDs must not be blank. Profile #325

This looks like it would be one of those empty profile profiles we discussed earlier. But then, it's not coming to a grinding hold right there or even shortly after. again, this would be something within the PC program already, not directly phpDVDProfiler related. (I offer to have a look; I would need a program backup file, basic profiles only, no cover images.)


I remember one kind of problem I had one, where the error would not cause an immediate stop, but it would take a while to mess up the import. But I think that came from using a plugin (not sure that even saw public release) to export single XMLs each time I changed a profile.

[Dragonfire]

It was saying another profile must not be empty for a while...I could also do the update.  After a while it just stopped giving that message..then this one started up.  This might be about the 3rd or 4th different one it has said that about.

[FredLooks]

Nice.

First about the blank profile - when using the many-files-in-directory approach rather than the single-XML-file approach, profile deletions (including blanks) must be done by hand rather than automatically by the windows program. Bottom line is that you have a file named .xml in your XML directory and if you delete it then the problem will disappear. Since that file name starts with a period, it may be invisible so if you have trouble finding it to delete it, a one-liner php script can do it for you (specifically, <?php unlink("XML/.xml"); ?> ). It isn't hurting anything, only cluttering up the output a little.

Back to the more interesting problem of the update not finishing. We can see some telling details.

If you look at the sequence of updates starting at the CompleteUpdate, you can see that the CompleteUpdate terminates after 600-ish profiles. The next update quickly (looking at the times in the first column of output) gets to 600-ish profiles and then starts slogging on, terminating after about 1000. The next goes quickly to 1000 and then stops after around 1500. This is all consistent with there being enough resources (provisional guess: time) to process about 600 or so profiles. Good, then it should take about 5 updates to complete the <3000 profiles.

But. Notice that after it hits 2000 profiles it doesn't advance. And from the times in the first column we can see that it is taking very little time to get to where it should continue the processing. So my best guess is that the first few runs are being stopped by a time limit of some sort (if I had to guess I'd say that your provider has imposed a 60-sec. time limit on accounts). The final few runs would seem to indicate that there is some other additional resource constraint that you're exceeding.

Can you try setting $ReportOnMemory = true; in your localsiteconfig.php.  Note that this may not work, since it relies on the precise format of the output of some system routines on your provider. If it fails we should be able to debug it ...

It may be worthwhile opening a support ticket with your provider, who can likely tell from inspection of her system logs what the issue may be, and if there is a mechanism to work around it ...

[Dragonfire]

I'll try changing the one thing in the localconfig file later this evening.  I'm going out to run some errands soon.

Also, the update thing was working fine until about 2 or 3 weeks ago.  Sometimes I would have to run the process a couple of times to get it to finish, but it would finish.

[FredLooks]

That would seem to indicate that your provider made some changes 2 or 3 weeks ago (since you hadn't changed the code prior to the problem, it's your provider or a profile that changed then and since you were double-processing the data before without incident, my money is on the provider). That might help them pinpoint the issue ...

[Dragonfire]

I can't find $ReportOnMemory = true; in the locasitelconfig.php

I went through it 3 times and can't see it...  is it near the top?  bottom?

[FredLooks]

Sorry, I was unclear ... just add a line anywhere in localsiteconfig.php that says: $ReportOnMemory = true;
There won't be one there already.

[Dragonfire]

Ok...I'll try that now.
Least I know I wasn't losing my mind and not being able to find it. lol

[Dragonfire]

Ok..I did that. 
Here's the result stuff.

Faster, more memory-intensive updates: On
Importing to database b2_5841265_phpdvdprofiler
Importing 2748 files from directory XML
0.185: Before Processing: Resident Set Size is 0.0MB (% of real memory) currently using 5.9MB
0.333: DVD profiles processed: 100: Resident Set Size is 0.0MB (% of real memory) currently using 5.8MB
0.466: DVD profiles processed: 200: Resident Set Size is 0.0MB (% of real memory) currently using 5.7MB
0.593: DVD profiles processed: 300: Resident Set Size is 0.0MB (% of real memory) currently using 5.6MB
XML/.xml: Profile IDs must not be blank. Profile #325
0.732: DVD profiles processed: 400: Resident Set Size is 0.0MB (% of real memory) currently using 5.5MB
0.829: DVD profiles processed: 500: Resident Set Size is 0.0MB (% of real memory) currently using 5.4MB
0.932: DVD profiles processed: 600: Resident Set Size is 0.0MB (% of real memory) currently using 5.3MB
1.087: DVD profiles processed: 700: Resident Set Size is 0.0MB (% of real memory) currently using 5.3MB
1.195: DVD profiles processed: 800: Resident Set Size is 0.0MB (% of real memory) currently using 5.2MB
1.331: DVD profiles processed: 900: Resident Set Size is 0.0MB (% of real memory) currently using 5.1MB
1.547: DVD profiles processed: 1000: Resident Set Size is 0.0MB (% of real memory) currently using 5.0MB
1.799: DVD profiles processed: 1100: Resident Set Size is 0.0MB (% of real memory) currently using 4.9MB
2.031: DVD profiles processed: 1200: Resident Set Size is 0.0MB (% of real memory) currently using 4.8MB
2.189: DVD profiles processed: 1300: Resident Set Size is 0.0MB (% of real memory) currently using 4.7MB
2.313: DVD profiles processed: 1400: Resident Set Size is 0.0MB (% of real memory) currently using 4.6MB
2.447: DVD profiles processed: 1500: Resident Set Size is 0.0MB (% of real memory) currently using 4.5MB
2.585: DVD profiles processed: 1600: Resident Set Size is 0.0MB (% of real memory) currently using 4.4MB
2.686: DVD profiles processed: 1700: Resident Set Size is 0.0MB (% of real memory) currently using 4.3MB
2.922: DVD profiles processed: 1800: Resident Set Size is 0.0MB (% of real memory) currently using 4.2MB
3.155: DVD profiles processed: 1900: Resident Set Size is 0.0MB (% of real memory) currently using 4.1MB
3.370: DVD profiles processed: 2000: Resident Set Size is 0.0MB (% of real memory) currently using 4.0MB


Faster, more memory-intensive updates: On
Importing to database b2_5841265_phpdvdprofiler
Importing 2748 files from directory XML
0.214: Before Processing: Resident Set Size is 0.0MB (% of real memory) currently using 5.9MB
0.356: DVD profiles processed: 100: Resident Set Size is 0.0MB (% of real memory) currently using 5.8MB
0.494: DVD profiles processed: 200: Resident Set Size is 0.0MB (% of real memory) currently using 5.7MB
0.641: DVD profiles processed: 300: Resident Set Size is 0.0MB (% of real memory) currently using 5.6MB
XML/.xml: Profile IDs must not be blank. Profile #325
0.784: DVD profiles processed: 400: Resident Set Size is 0.0MB (% of real memory) currently using 5.5MB
0.950: DVD profiles processed: 500: Resident Set Size is 0.0MB (% of real memory) currently using 5.4MB
1.074: DVD profiles processed: 600: Resident Set Size is 0.0MB (% of real memory) currently using 5.3MB
1.247: DVD profiles processed: 700: Resident Set Size is 0.0MB (% of real memory) currently using 5.3MB
1.329: DVD profiles processed: 800: Resident Set Size is 0.0MB (% of real memory) currently using 5.2MB
1.468: DVD profiles processed: 900: Resident Set Size is 0.0MB (% of real memory) currently using 5.1MB
1.647: DVD profiles processed: 1000: Resident Set Size is 0.0MB (% of real memory) currently using 5.0MB
1.822: DVD profiles processed: 1100: Resident Set Size is 0.0MB (% of real memory) currently using 4.9MB
1.976: DVD profiles processed: 1200: Resident Set Size is 0.0MB (% of real memory) currently using 4.8MB
2.125: DVD profiles processed: 1300: Resident Set Size is 0.0MB (% of real memory) currently using 4.7MB
2.295: DVD profiles processed: 1400: Resident Set Size is 0.0MB (% of real memory) currently using 4.6MB
2.485: DVD profiles processed: 1500: Resident Set Size is 0.0MB (% of real memory) currently using 4.5MB
2.711: DVD profiles processed: 1600: Resident Set Size is 0.0MB (% of real memory) currently using 4.4MB
2.894: DVD profiles processed: 1700: Resident Set Size is 0.0MB (% of real memory) currently using 4.3MB
3.107: DVD profiles processed: 1800: Resident Set Size is 0.0MB (% of real memory) currently using 4.2MB
3.273: DVD profiles processed: 1900: Resident Set Size is 0.0MB (% of real memory) currently using 4.1MB
3.438: DVD profiles processed: 2000: Resident Set Size is 0.0MB (% of real memory) currently using 4.0MB

[FredLooks]

Hmmm ... it looks like there is no problem with memory. I would be inclined to ask your ISP if they can tell you why the process is failing to complete - the code is instrumented to squawk if it detects any errors, and generally speaking, only dies silently when it has been terminated with prejudice and can't say anything. If there were an SQL error (limits can be put on the number of DB operations performed over a period of time, and an update does rather a lot of them) then the code should be saying something, but sometimes those errors are muted or logged and processes stopped.

I'm happy to help if I can with such a conversation ...

[Dragonfire]

The host for where my collection site is?